Strategy backtest has some wrong dates and prices

[ElMurcielago]

I am running a backtest using the strategy analyzer in NinjaTrader-64 v 7.0.1000.11 bit on Windows 7 home premium, SP1 on a Toshiba laptop (Portege R835). I am using end-of-day data provided through Yahoo (corrected for dividends and splits via the Instrument Manager), and a set of ten ETFs (DIA, EEM, EFA, EPP, ILF, IWM, MDY, QQQ, SMH, SPY ) to test a long-only strategy. I have unlocked the code and written my own backtest logic. I've run the backtest, and then for each security, I'm viewing the trades, which I then export to excel so I have all the trades of the backtest (1489 trades) in a spreadsheet. There I'm analyzing the data and checking for errors. The test seems to be working properly, more or less, but there are a couple of strange things.

1) The trades are showing entries and exits on days the market is closed (Saturdays and holidays). All the trade dates appear to be one day later than the actual execution. So for example, if the trade is supposed to execute on the open of a Friday, it executes at that Friday's opening price (more or less), but the trade date shown is the next day, i.e., Saturday. This doesn't seem to be affecting the results, but it's doesn't build confidence either, and it's a pain when creating formulas in excel that need to refer to the date.

2) A subset of the trades show execution prices that are way out of whack, i.e., more than 3% different (many 6.5% to 11.9% different) than they should be, even assuming built-in slippage. Of the 1489 trades executed in the backtest (purchase + sale = 1 trade), 157 exhibit this problem.
Despite this, the entries and exits appear to be correct with regard to timing, but it means the results of the backtest aren't correct.

So my questions are these. Is problem #1 a known bug? Is there a fix for this? Does it matter? And for question #2, is this also a known bug? Is there something I'm doing wrong? I'll paste the key logic of the backtest code below. (can't paste it all due to length restrictions)

Thanks for looking at this,

Kim G

[ElMurcielago]

Re: Problems in backtest

It appears that the code is too long for the length restrictions. I'm not sure how else to show it to you. I'm going to try to attach a text file with it. Thanks

[NinjaTrader_ChelseaB]

Hi Kim,

To export your script do the following:

1. Click File -> Utilities -> Export NinjaScript
2. Enter a unique name for the file in the value for 'File name:'
3. Select the strategy from the objects list on the left -> click the right facing arrow ">" to add the strategy to the export
4. Click the 'Export' button -> click 'yes' to add any referenced indicators to the export -> click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 7/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.
http://www.ninjatrader.com/support/h...nt7/export.htm


Are you running this script on a daily time frame? If so, the orders appearing on the next bar is expected.

When NinjaTrader is running with Calculate on bar close (COBC) as true, orders are placed after the bar closes using logic calculated from that bar.

Below is a link to the help guide on Calculate on bar close.

http://www.ninjatrader.com/support/helpGuides/nt7/calculateonbarclose.htm

This means that any orders that are triggered from that bar are submitted after the bar closes as the new bar opens. As the order is placed as the new bar opens and the timestamp of the order is within the time of the next bar, the order will show on the next bar.

If the strategy were running with COBC as false, or if the script has intra-bar granularity and submitted orders on a smaller timeframe, this would cause the timestamp of the order to be within the bar it is triggered on as the order may be submitted intra-bar. This order would show on the same bar as the bar that triggered the order submission.


With the order fills that are incorrect, what price was the order submitted at? (Find this by printing the order price in OnOrderUpdate when the order is in an accepted state)

What price did the order fill at? (Find this by printing the execution price in OnExecution)

[ElMurcielago]

Let me clarify

Hi Chelsea,

Thanks for the quick response. But with all due respect (and I'm sure you are VERY busy), I don't think you read my original post carefully since the questions you ask were answered in the original post.

So to restate, I am using daily bars only, provided through Yahoo. I am using COBC=TRUE, and I understand that the logic is for a buy/sell to be triggered on the close of bar 0, and then executed at the open of bar 1. But I'll restate what I said initially. This is happening, but the date stamp on the trade is the day after bar 1. THE DATE STAMP IS INCORRECT. So I'm getting execution dates on SATURDAYS and MARKET HOLIDAYS. But those executions are happening at the price of the open of bar 1, i.e., the day before the timestamp (which is when they are supposed to be executed). Problem #1 is that the TIMESTAMP on the trade is a day behind events. I understand how orders are executed, but this appears to be a bug in NinjaTrader. How else could it be possible to execute an order on a backtest on a Saturday? There's no pricing data for Saturdays.

As for problem #2, the orders are submitted as (I believe) market orders. I'm using EnterLong(Shares, "Pos1"); as the order-entering statement. "Shares" is an integer variable, and "Pos1" is the name of the entry signal. I believe this statement should create an order at the opening price of the next bar. Please correct me if I'm wrong on this. In any case, of the 1489 trades executed in the backtest (purchase + sale = 1 trade), only 157 exhibit the problem that the price was not very close to the open price of the next bar. The rest were executed at or very close to the opening price of bar 1. And I have no idea where the errant prices are coming from. They do not appear to be any of the prices (O,H,L, or C) of any of the days near the trade. So this is a total mystery to me, and I think it's also a bug, but I'm hoping you can prove me wrong on that.

And I'm not sure if you wanted me to export the script and post it or not. But it won't export, (says "Error compiling export sources: c:\Users\Kim G\Documents\NinjaTrader 7\bin\Custom\Strategy\Channeling.cs(63,4) : error CS0103: The name 'Bollinger' does not exist in the current context."). Yet it is compiling fine and running fine in NT. But as I said, in my second post, I have uploaded a text file with the script so you can easily see it, and if necessary paste it into a strategy window and compile it. It does not require any custom indicators. I should add that the dates for my backtest are 1/1/2002 to 12/31/2013. The pricing errors are happening on EFA, EPP, and EEM. I can provide further detail if you wish.

Thanks again for your help and I really am not trying to be snide at all about reading the first post.

Kim G

P.S. though it looks like I'm the newest of "newbies," I have been working hard for the past year learning NT, and I didn't want to ask any questions on this forum until I had exhausted all other possibilities. I've read a TON of forum posts to learn how to use NT, and I've already worked hard to figure out this error on my own, but I sincerely believe it's either an NT bug, or something very strange that would be hard to figure out on my own. Problem #2 is only affecting 10.5% of the trades, but Problem#1 seems to be affecting them all. Thanks again for your attention to this.

[NinjaTrader_ChelseaB]

Hi ElMurcielago,

Thanks for clarifying this for me.

My thoughts were that the trades showing on saturday and holidays are from the close of the day before.

In the backtest results chart tab, do you see data for the days there are Saturdays and holidays?

Are there trades showing on sunday / monday?

What are the specific dates for me to look for in the backtest


The script was not able to compile from the text file but it looks like a few missing characters from the top of the script. I'm not sure if anything else is missing.

I added the following code to help see when the executions were happening.

protected override void OnExecution(IExecution execution)
{
Print (execution.ToString());
Print(Time[0]+" - "+Time[0].DayOfWeek);
}

I'm testing using Yahoo as my connection. I'm finding the daily historical data on yahoo goes back to sometime in September 2013.


The output I have attached. Do I need to change the inputs to see the trades you are seeing?

May I have your output with this execution code added?

[ElMurcielago]

Getting back to you

Hi Chelsea,

Thanks for taking another look at this data. Again, I'd like to emphasize that, for example, the trades that are taking place on a Saturday or holiday are taking place (more or less) at the opening price of the prior day. That prior day is when the trades are supposed to take place, and the execution is fine with regard to price and trigger. Again, to emphasize, my issue is that the time stamp is a day behind where it should be, which is how some of these trades are getting time-stamped on days where the market is closed. As far as I can tell, ALL trades are getting time stamped a day behind actual events.

Today I brought up a chart for EWW, and noticed that it had a bar dated 6 Sep 2014. As you can quickly tell, 6 Sep 2014 is a Saturday. I'm including a screen shot of this chart, with some text I added in Photoshop.

I'm also going to try again to send you my Ninjacode in a new file. I don't understand why it didn't compile for you. The way I have created the second file is to go to the strategy editor window, do "select all" then copy and paste into Microsoft Notepad, and then save the result to a text file. I don't think Notepad inserts unwanted characters, but you may know something about that. My belief is that it creates a clean text file that should compile. You may need to create a strategy called "Channeling," then unlock the code, delete it all, and then paste in my code. But again, you know more about this than I do.

As for output, I can either do as you asked, via the output window, or I can send you a spreadsheet that I get from the output window of the strategy analyzer. For now, since I have it, I'm going to send the spreadsheet. There's one tab for QQQ and one tab for DIA. It has the Saturdays highlighted in pink. There were no trades executed on Mondays. I have not gone through to see if there were executions on holidays, but I think it's basically the same problem. I think this will be much easier for you to read anyway, as my output window has a ton of other stuff besides what the code you sent would put there.

I did go through my entire backtest of the 1,489 trades and not a single entry or exit was processed on a Monday or a Sunday. I also took a quick look at a chart of EWW and also noticed that there are Saturday bars, and no Monday bars. I think this date problem is something more core to NT than just my backtest.

As for the pricing problem that I mentioned in my original post (Problem #2), I'm inclined to think this is bad data where NT is using some kind of "adjusted close" number for the ETFs in question. But any light you can shine on the problem would be much appreciated.

Thanks for your help,

Kim G

P.S. It looks like an Excel file upload is not permitted, so I saved it as a CSV file. Let me know if you still want the output window too. Also looks like I can't upload 3 files, so I skipped the picture of the chart. But pull up your own chart and see if there are any bars on Mondays. I'll bet there aren't.

[ElMurcielago]

Files attached

I'm attaching the output window contents for both DIA and QQQ as requested.

[NinjaTrader_ChelseaB]

Hi Kim,

I would like the output from the output window as this is what I will be comparing to my output. (Specifically, I want to know what bar the OnExecution() is firing on)

You can upload more than two files, simply attach the first two, then attach two more, etc...


What holidays (please use the date) specifically are showing in this data?

[NinjaTrader_ChelseaB]

Hi Kim,

You have data that is not available from the Yahoo connection.

Where is this data coming from so that I can make a comparison?

[ElMurcielago]

Hi Chelsea,

I have been using the connection to Yahoo, though I tried Kinetick (free EOD data) also. I'm not sure how you are finding non-Yahoo data, but those two are the only sources for my data. Yahoo is set up as the default and automatically connects when I start NT.

I'm attaching a CSV file with all the DIA and QQQ trades with holiday executions flagged in column T (entries) and U (exits). I'm also going to put a list of NYSE holidays toward the bottom of that file for you too.

Thanks again,

Kim

P.S. I attached the output window files in a prior post.

[NinjaTrader_ChelseaB]

Hi Kim,

The data may be from the Kinetick End Of Day connection but I need to know who the data is from because the data will be different between providers.

Please temporarily move your historical data, and connect to yahoo only to re-download, run this script, and post the output.

Please use the steps I have provided below to delete your cache data:

• Close NinjaTrader
• Open (My) Documents/NinjaTrader 7/db/cache/
• Delete all files within this folder
• Go up one directory to the /db/ folder
• Rename the Day folder to Day.bkup
• Restart NinjaTrader
• Connect

[ElMurcielago]

Hi Chelsea,

OK, I've followed your instructions and here are the data files for QQQ and DIA.

Thanks,

Kim

[NinjaTrader_ChelseaB]

Hi Kim,

I'm not sure what happened there.

I opened a chart first and loaded the data back and then was able to get that data from Yahoo. Sorry about the confusion there.

Currently, I am using the Default 24/7 template to test this.

I am not getting fills on Saturday...
Attached is my output.

Maybe the timezone difference is causing differences?

I'd like to give you a call so we can find the difference between our computers.

We can also address the issue with exporting as well.

Please send an email to platformsupport [at] ninjatrader [dot] com. In the email please add a link to this forum thread.

[NinjaTrader_ChelseaB]

To All,

We found the issue here to be the Default 24/7 template.

Once changed to US Equities RTH this corrected both trades on Saturday and on holidays.

[GuppyDRV]

What was the issue with the default templates? Has this been corrected?

GuppyDRV