Discrepancy between Strategy Analyzer Backtest and Historical Chart Strategy Perform

[RunnrX]

Hi, sorry about the long title, but it says the most of it.

I've run a strategy backtest from 1/1/2000 to 3/1/2013 with a strategy I've programmed on the IWM that got some astronomical success, and turned $10,000 into $14,500,000 in the backtester, but when putting the exact same strategy in the exact same instrument over 1/1/2000 - 3/2/2013 in a chart window with historical data only (loaded it all on a sunday during market closed time), the exact same strategy over that period returns something like $-5000 in the strategy performance window.

I'd have thought that both performance reports would be the exact same calculations over the exact same price data, but something very different is coming up

The point of this post is to ask the questions: 1) Which performance report do I believe, and 2) Why would they disagree when they have the same price data and strategy parameters? Are there any NT pros that know about this kind of thing?

Thanks!

[NinjaTrader_Lance]

Hello,

If the settings between the two are the same you will need to ensure the same data was used in both tests. First check that session templates are the same between the two and also the days loaded are equal.

Next you need to keep in mind that this is a large amount of data you are testing over. Are these daily bars or lower? Most data providers will not provide accurate trading data this far back. You can check the data you currently have downloaded through the historical data manager (tools -> Historical data manager -> Edit tab -> select the instrument and scroll through ensuring the data is there)

If this is a futures contract you also have to keep in mind your merge policy: http://www.ninjatrader.com/support/h...l?data_tab.htm

If you need more information let me know the instrument and data provider.

[RunnrX]

Hi,

Thank you for helping out. I am using daily bars with the Kinetick Free EOD data for both the strategy analyzer and the "real" chart window backlooking performance of the strategy. I have the session set to mon-fri 9:30am - 4pm for the chart window, and this is on the IWM which is an ETF that trades during stock market hours.

The strategy running on them is running the same version with the same parameters, so I wonder about how ninjatrader processes things differently between the two to get such a different outcome.

[NinjaTrader_Lance]

Hello,

Try this with a smaller test, set the start date of the chart and the Strategy Anazlyer to 1 year and run the stock Strategy "SampleMACrossOver"

Do the results still differ on the same settings?

[RunnrX]

That's a good idea.

I did another thing first to test:

1) I did my own strategy on just 365 days on the chart window and it came up with 21 trades in the strategy performance window, not the same entry dates as in the startegy analyzer -- and it made about $3 using $10,000 initial capital. The strategy analyzer version from 3/1/2012 to 3/1/2013 made $7000 in profits starting from $10,000 over 21 trades with approx the same dates but not quite the same as in the chart window. I'm using session template of mon-fri 9:30am-4pm for the chart window and strategy analyzer is whatever it defaults to.

2) SampleMACrossOver on the strategy tester got $1185.00 over 8 trades from 3/1/2012 to 3/1/2013, and the chart window historical performance on 365 days got $1185 over 8 trades just like the strategy analyzer.

That then identifies my strategy itself as the targeted factor, so what could my strategy have in it that makes it get differing results between the two backtest methods? I'm open to any speculation

Thanks!

[RunnrX]

Extra things that may be of importance: I have my strategy set to make its share size based on a variable of a virtual $10,000 account size and then the share size is scaled based on the cumulative P&L growth of that starting amount. Could that bring up a conflict in how charts connected to a sim account behave versus in a strategy analyzer? Also, I'd really like to know which chart to believe is correct - and I hope it's the strategy analyzer one!

[sledge]

Extra things that may be of importance: I have my strategy set to make its share size based on a variable of a virtual $10,000 account size and then the share size is scaled based on the cumulative P&L growth of that starting amount. Could that bring up a conflict in how charts connected to a sim account behave versus in a strategy analyzer? Also, I'd really like to know which chart to believe is correct - and I hope it's the strategy analyzer one!

Hmmm... Maybe Strategy Analyzer will trade for you with no problem with insufficient funds ...

What if you start with $100??

[RunnrX]

That was a good thought but it doesn't seem to be the issue. I changed it to $2 account value and no trades were executed at all in my strategy

[NinjaTrader_Lance]

Are you able to attach the strategy or simplified version of the strategy which replicates the differences so I can test on my end?

[RunnrX]

Are you able to attach the strategy or simplified version of the strategy which replicates the differences so I can test on my end?

Sorry but unfortunately I won't be able to. It's a proprietary strategy I'm making for someone else and I wouldn't be able to send it. I was hoping for just some ideas from people about further areas to troubleshoot and test like what I've been getting so far. The next thing I would do is simply take the last 1 year period and physically write down each of the 21 trades and see what they really returned in value and then I'll know which backtest summary was adding them correctly. In that, your idea of doing it over a shorter period was the key to making this a manageable solution vs trying to test all 320 trades over 13 years

[NinjaTrader_Lance]

Understandable. Based on what I've seen my assumption is that it has to do with the money management tracking. But I really can't say for sure without going in and adding print statements to test different trades.

[RunnrX]

I wonder if this new information may help:

I've tracked each of the 21 trades it took manually in the chart window and found that they do in fact line up with the strategy performance window Trades grid perfectly. (And their money tracking is correct as well - it's starting with $10,000 and the share sizing / accumulation is consistent and correct)

Then, I went into the strategy analyzer running the exact same start and end dates as were loaded in the chart and I found this oddity: Every single daily candle in the strategy analyzer chart is being given a date that is one day later than it should be. March 5 is being called March 6, etc. and I've double checked this by pulling up a chart from another source to compare - Strategy analyzer is calling them date+1 somehow.

Then I also noticed that while they're running the exact same strategy, they are entering the first trade on the same Candle, which should be the same date, but ignoring the date labeling.. the next issue is that the very first trade has a different exit point in each instance. Trade 1 gets in the IWM on April 19 2012 and exits the open of April 24 in the chart window, but exits the open of April 23rd in the strategy tester (though it says it's April 24).

However, my strategy does not make calculations based on dates at all. It is essentially doing this:

Set a buy stop at a pre-determined level to get long. Once long, if the stock closes below a set level, exit the stock the open of the next daily candle. The strategy is not differing between the chart window and the strategy analyzer window, but the actual candles being exited at the open are differing, with the strategy analyzer looking to be the one at fault (which is too bad because I was liking the astronomical numbers it was giving me for $PL)

It's calling the candles by a later date than they really were in the SampleMACrossOver strategy also. But I don't see a way to edit StrategyAnalzyer's session management etc

[NinjaTrader_Lance]

It sound like you are but just to double check , are you setting the strategy analyzer's session template to the same session template as the chart? You can do this just under the From and To date selection.

The bars will always be labeled with the closing time of the bar. If you're saying the Analyzer and the Chart display different closing times for the same bar could you please provide a screen shot of the difference?
Please provide me with an image of your chart.
To send a screenshot press Alt + PRINT SCREEN to take a screen shot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.
For detailed instructions please visit the following link

* http://take-a-screenshot.org/



If you've already checked this go into file -> historical data manager -> edit -> choose the instrument being tested and delete the hisorical data. Connect back to Kinetick EoD Free and run the two tests again.

Other settings to check are going to be Exit on Close true/false, ensure these are the same.

I know you can't provide me with the script you're working with but if you could create a simplified script that replicates the the error I could take a look at it.

[RunnrX]

I've taken a look at the strategy analyzer and found that while using daily bars I was not able to change the session template. But when I went into the instrument manager and looked at the session for the instrument IWM I saw that it was set to Default 24/7 and when I changed it there to Mon-Fri 9:30 - 4pm, it made the dates on the candles look right.

That made the two trading histories get closer together but not fully reconcile by a longshot. Also, it dropped the strategy analyzer from making $14,500,000 profit from an initial $10,000 over 13 years into a $9000 loss so I'd really like to dissect the glitchy results to see how it was possible.

ExitOnClose is set to false for both, calculate on bar close false, etc.

Thank you for your offer to look at the coding but there are about 360 lines to it which to me is a lot to try to simplify and I think I wouldn't be able to recreate a simplified version for you unless you wanted me to write it out in an english logicflow versus the actual ninjascript syntax. I did notice one other odd discrepancy though..

This strategy is designed to run on two bars objects. Bars[0] is the instrument being traded and bars[1] is the S&P500. It is designed to exit the instrument being traded when the S&P falls through a certain downside level, and I'm seeing that S&P-based exit occuring in the Strategy Analyzer more often than it is happening in the live window.. I'm going to look further into this, but please let me know if you want a basic written logic-flow rendition of the script and I can create that for you (thanks for sticking with me this far by the way)

[NinjaTrader_Lance]

Hello,

With Kinetick EoD data you're only going to get the data for the daily bars. Daily bars do not have a session template but the display of them can be affected by the session template. To change the display template for the strategy analyzer you would need to edit the default instrument in the instrument manager.

Tools -> Instrument manager -> search for the instrument being used -> select it click Edit -> change the default session template to match between SA and Chart.

Let me know if this doesn't resolve the issue.