Backtesting versus RealTime Autotrading

[siroki]

Today I ran my script on a realtime feed and had my orders executed. Then after the market close I ran the strategy analyzer over the same set of stocks and also today's timeframe.

To my surprise the results did not match the actual orders that were triggered today!

Is this a known issue?

I am seeing that on the real feed / live trading, there were 3 more stocks traded that are not triggered on the Strategy Analyzer.

All input parameters have been checked and are exactly the same.

Hope to hear from somebody.

Thanks.

[NinjaTrader_ChelseaB]

Hi siroki,

There can be difference between backtest and live results.

Below is a link to a document that outlines these differences.
http://www.ninjatrader.com/support/h...ime_vs_bac.htm

However, if your strategy is running with Calculate on bar close set to true, there may be other things that are causing differences.

The best way to move forward is to add prints to your code that show values are being used for conditions during the live and the backtest. Sometimes we are able to find differences in the live data and backtest data such as a PriorDayOHL().PriorClose value. Printing all values used in your conditions along with the time of the bar (Time[0]) can help you to determine what is different between the tests.

[siroki]

Thanks for your reply. Yes, I already print all the values to the output window when signals occur and yes I use the CalculateOnBarClose method.

I found differences between the live and backtest results through comparing the output window results of the live versus the backtest scenario. The problem is that I cannot reproduce the live result. Once the live results are exported I can only compare it to the backtest results and pinpoint differences.

One thing I did find out, which in hindsight is obvious, is that in backtesting you always get filled. In live results your order might not get through. So I might need to tweak a bit by offering and bidding a margin below/above the bid ask so I have a higher chance to get filled. And then the backtest results will come closer to the actual results.

But still, I am a bit stuck on how I can re-run a live trading day based on the loaded data versus using the exact data in the backtest results.

Question: the data that is used for backtesting is the exact same data as comes in during the live trading session? So I am looking at the same data set for both scenarios live and backtesting? Otherwise I am comparing apples to oranges.

Thanks so far .

[NinjaTrader_ChelseaB]

Hi siroki,

The issue is that the historical data will only have OHLC points and will not have any of the intrabar data that happened during that bar. The ask and bid data is not there, and also GetCurrentAsk() and GetCurrentBid() doesn't work.

For a more accurate test, I would suggest running the strategy through the Market Replay. The Market Replay is basically a recording of a live market. This will be pretty much exactly what was received during the day it was recorded with all of the market dynamics and intrabar data you may need for an exact test.

Below is a link to the help guide on Market Replay.
http://www.ninjatrader.com/support/h...ket_replay.htm

Also, here is a link to a NinjaTrader YouTube Training video on Market Replay.
https://www.youtube.com/watch?v=cgB1...D7105&index=11

While most of the popular futures are available for download from the NinjaTrader Market Replay servers, most equities are not. Instead, you have the option to record your live data for use with the Market Replay.

To do this:

• Click Tools -> Options... -> select the Data tab
• Check the box labeled 'Record for market replay'
• Click OK

You may need to disconnect and reconnect to start the recording.

[siroki]

Hello NinjaTrader_ChelseaB,

Thanks for your answer. I have recorded the market data last Friday on a stock named YELP. I'm running the backtest now and the stock was not triggered as a short. But when I run the market replay and run the exact same strategy on YELP, the stock is triggered. So there must be a difference between the historical backtest/backfill data and the market replay data.

Can I view the data-sets so I can find possible data-differences between historical and data replay? Because at this point there is absolutely no value in backtesting on historical downloaded data if it doesn't match with the actual live trading data.

Btw: both sets come from the same broker.

[sledge]

So there must be a difference between the historical backtest/backfill data and the market replay data.

That's what she explained. There is a difference in how things are computed.

Backtest is for getting an idea started, optimizing parameters on a general scale, and does it very quickly over a data set.

Market Replay is 2nd close as you can get to live (Running Live in SIM mode is final step). Market Replay takes a lot longer to process.

[koganam]

Hello NinjaTrader_ChelseaB,

Thanks for your answer. I have recorded the market data last Friday on a stock named YELP. I'm running the backtest now and the stock was not triggered as a short. But when I run the market replay and run the exact same strategy on YELP, the stock is triggered. So there must be a difference between the historical backtest/backfill data and the market replay data.

Can I view the data-sets so I can find possible data-differences between historical and data replay? Because at this point there is absolutely no value in backtesting on historical downloaded data if it doesn't match with the actual live trading data.

Btw: both sets come from the same broker.

Such differences are to be expected: the fill engine is different.