Replay & Backtest Differences

[lee612801]

Hello,

I have been trying to verify my backtest results with replay in NT7 to ensure that the strategy I'm using is working properly. One of the big issues I have with this is that I cannot seem to get it to start on the day I choose. If I choose a range from 6/17 to 6/19, it starts on 6/15, then skips to 6/17. I don't know if I have a setting wrong, but it seems like it is off by one day. It is inconsistent IMO and therefore difficult to correlate backtest results with real-time performance when you are looking at exact entry and exits. the only way I have been able to get it to start and run without skipping days is to start on the weekend before.

Also, I have found some inconsistencies with the results when using playback. I have an entry on say 6/23 with backtest. If I am using replay with COBC = false, I get the same entry point. If I set the COBC = true, and run the replay straight through from the day before (set to before market start time) I don't get an entry signal until 6/24. BUT, if I start the replay from the day before with COBC = true, except start it during market hours, I get an entry signal on 6/23 that matches the backtest...

I know that the strategy could have something to do with (I'm using daily time period). I also know the data could be an issue. But, I am using recorded data from my real-time Kinetick feed. So I don't think the data is an issue. With respect to the strategy code, this could be a problem, but with the other inconsistencies I have experienced with market replay, and the fact that I have got the code to execute correctly numerous times with a different time period, I just don't know. I would prefer not to tear into the code right away if the problem is with the replay.

So, are there some known issues with this and is there anything that can be done to fix or help minimize some of these problems I'm experiencing?

Thanks,

Lee

[NinjaTrader_DaveI]

Hello,

When playing downloaded Market Replay data, the playback may sometimes begin on a prior day, but this can be adjusted by changing the Date fields in the Market Replay control window, dragging the slider to the right a bit, or right-clicking within the window and clicking "Go To."

If you find that the data skips entire days, I recommend right-clicking in the Market Replay control window, then clicking Available Data. In the window that appears, it is important to expand the listing for the instrument you are viewing, so that you can drill down and see exactly what data you have (the range listed in the primary line for an instrument can be deceiving, as there can be periods missing when you drill down further). If you find that you are missing days here, you can then go back and download them. This is especially possible if you are working with data that you saved in real-time.

One big thing to be aware of is what your strategy will consider historical vs. real-time data in Market Replay. Whenever you begin playing, the strategy considers everything prior to that point to be historical, and everything after that point to be real-time, until and unless you jump to a different point in time, at which point the strategy will re-evaluate what it considers historical vs. real-time according to the same criteria. Thus, the specific point at which you begin, and the amount of Market Replay data available before that point, can greatly impact your strategy's entries and exits.

[lee612801]

Hi Dave,

Thank you for the info. What you have explained will be valuable when I go to review my current settings and data.

I do have one more question. How does the session template effect replay vs. backtest results? For instance, If I wanted to match backtest results as closely as possible, should I use a certain template (like RTH) to do this?

Thanks again,

Lee

[NinjaTrader_DaveI]

Hello,

You are able to select a specific session template from the Backtest/Optimizer settings in the Strategy Analyzer before running a test, so the best policy will be to make sure that you are using the same template in both Market Replay and the Strategy Analyzer. I personally recommend using whichever template you plan to use in live deployment, for the most accurate results.