Drastically Diffferent Results than Market Replay

[Briansstocks]

Why would I be getting drastically different results in Strategy analyzer than I am using the same settings and time frames as in Market Replay?

[NinjaTrader_MichaelM]

Hello Brianstocks,

Thank you for writing in. Please feel free to read through this document in our help guide for a basic explanation: http://ninjatrader.com/support/helpG...ime_vs_bac.htm

There are many posts concerning this on our forums which you can search through for more information. Here is another: http://ninjatrader.com/support/forum...ad.php?t=64943

In short this behavior is expected due to the differences between backtesting and market replay. If you have any concerns which you believe are not related to this, but are affecting your results, please provide me with a specific example as well as your strategy code and I will be happy to investigate further.

Thank you in advance.

[Briansstocks]

Thanks MichaelIM,

I had realized what was going on after I submitted this thread. My question is now how to get around my problem.

I am stuck with a very poor internet connection for a few days and cant do a lot of searching right now.

Just so we are clear as to my purpose I am trying to run back tests on my strategy that will require tick data.

Market replay is essentially garbage that does not work on 20% of the Instruments I have run tests on. I have already dealt with the forum for months on that issue and the solution provided is wait for Ninja Trader 8 to be released an hire another programmer to re code my strategy then Market Replay will work.

Let me just sneak in a quick question about Replay even though this is the wrong section. How far back can I download data from Market Replay? I have read 3 months and I have read a year.

For the Strategy analyzer there is an option when setting up a test for the data series as Tick's. I am unclear why I still get different results for the same time frame and settings as Market Replay using downloaded data. Is this due to the 150 Value in the data series?

I am at a cost of around 5000$ all together and 6 months of my time so far and can not even use Ninja Trader to run a test on my strategy that I can trust the results from yet. I'm sure you can see where I'm running out of patience with all of this.

I'm aware that no back test can truly account for actual real time fills so there would be a difference in live performance.

I can see no reason why using tick data should give me different results whether I am using Strategy analyzer or Market replay with downloaded data. I'm not a programmer so I don't understand some of the help topic you linked to your reply and I don't think it is quite the same issue I am having.

Since market Replay don't work on 20% of the instruments I want to run tests on how do I run accurate tests on my strategy with Strategy analyzer or any other feature in Ninja Trader?

[NinjaTrader_MichaelM]

Hello Brianstocks,

Do you run your strategy with Calculate On Bar Close set to true or to false?

Also, market replay should work on any instrument you can load tick data for. You simply need to go to Tools -> Options -> Data tab -> Check "Record for market replay" and then leave a chart open with the instrument you wish to be able to use in market replay. Please note: With this method, market replay data for the instrument will only be available for the times you have recorded.

The amount of market replay data available to download from the servers is usually about 90 days back for a supported instrument.

Could you please elaborate on why you believe your tests are inaccurate? I recommend comparing live results to backtest/replay results as opposed to comparing backtest to replay results.

Thank you in advance.

[Briansstocks]

MichaelIM,

I run all tests with bar close set to false. I realized that was part of the problem after I created this thread. Strategy analyzer was using bar close. I then tried with tick selected and still came up with drastically different results from the same time frame and settings as Market Replay.

I am aware of how to use Market replay. I have not recorded my own data. I download the data from Market replay. Obviously I would not want to wait months and months to record enough data to run tests.

To be clear I have run live tests and found the live performance to be inline with Market Replay. The problem is 20% of the instruments I would like to run tests on have huge phantom trades that don't exist.

Obviously when I run a test on Strategy analyzer with tick data and get a result of Positive 7000$ and then run the exact same test with the exact same settings on Market Replay with the data I downloaded from Market replay and it shows a result of Negative 1500$ something is wrong with one of the options available to me to run tests.

[sledge]

MichaelIM,


I am aware of how to use Market replay. I have not recorded my own data. I download the data from Market replay. Obviously I would not want to wait months and months to record enough data to run tests.

To be clear I have run live tests and found the live performance to be inline with Market Replay. The problem is 20% of the instruments I would like to run tests on have huge phantom trades that don't exist.

.

I'm not sure how much time you have left on this planet - but I have some good data after 2 years of recording (although it's got it's NT8 issues)...

Anyways - quit making market order trades when the market is closed. Don't do it. My testing showed at least up to a bunch of trades before market replay was "stable". You should be fine after 30 seconds to 1 minute from phantom trade issue I hope... I don't know what NT7 is doing internally and why it's lagging - I don't work there and haven't had time to fix their code. I can only guess. So maybe these other 20% might need more time because of less trades.

I hope this helps. Good luck.

[NinjaTrader_MichaelM]

Hello Brianstocks,

Thank you for the update. At this point I would need to see your strategy code or screenshots of both your strategy analyzer and market replay results to further investigate.

Please feel free to either post on the forums here or to email platformsupport[AT]ninjatrader[DOT]com with the subject line : "ATTN: Michael M http://www.ninjatrader.com/support/forum/showthread.php?t=78815".


Thank you in advance.

[Briansstocks]

Sledge,

That is disappointing news. I was assured the phantom trade issue would not be in NT8.

I have experimented with creating a time template to test market relay on with no success. In the end my test results wouldn't count for much since unless I have a volume issue my real time trading would cover 24/7.

Regards

[Briansstocks]

MichealIM,

I will send you what you need. it will take me a few days because I am stuck with a really bad internet connection right now but I will get it to you when I can.

Regards
Brian

[NinjaTrader_MichaelM]

Hello Brianstocks,

Thank you for the update. I will investigate this matter further at that time.

In the meantime, please have a great rest of your week!

[bluelou]

Brian,
1) Market replay will be different from strategy analyzer. You can test this by setting up a print statement with last/bid/ask. In strategy analyzer you'll see crossed/locked markets; i.e, bid == ask.

Market replay should be a bit more accurate due to the sequence in which the market events are recorded (I think). But, it will still be wrong! (see below)

2) You really can't test at the tick level with NT. Try writing code with resting limits and see if you get a fill when the bid is hit (market goes ask at your limit price) on your resting buy order.. Lots of times you won't get filled due to issues with NT. I was never able to resolve this and this was part of the reason I upgraded to a different platform. Tho, I still use NT, too, since I have legacy code here.

[DaFish]

2) You really can't test at the tick level with NT. Try writing code with resting limits and see if you get a fill when the bid is hit (market goes ask at your limit price) on your resting buy order.. Lots of times you won't get filled due to issues with NT. I was never able to resolve this and this was part of the reason I upgraded to a different platform. Tho, I still use NT, too, since I have legacy code here.

I am really struggling with getting NT to work, and to be able to back test my stuff. I get COMPLETELY different results. I use multiple time frames and I can't even seem to get the right data when my strategy starts up from older time frames. When it finally loads and runs live, it seems to work.

Running LIVE market data gives me the proper results, but I cannot seem to back test at all with any consistency or trust in the system though.

I am not sure if this is related to my skill level, or just knowledge in general but I have thousands of lines of code trying to get this to work and 2+ YEARS of NT7 work invested.

I have had to stop on NT8, I did port all my code over, but market replay crashes and is unusable, I think some of my code didn't port correctly, and NT8 crashes ... to the point of giving up on NT8 because I HAVE to finish the work debugging my trading system so I can actually start trading it automatically. I will wait for the released version.

That said in NT7 I don't seem to be close to trusting the system enough to use it.... which is exceptionally disappointing given the time and effort put into the systems.

bluelou, what system did you replace NT with please? And why are you still using NT?

I will continue to put in support tickets and work through my issues on NT7 - but I have to tell you I am losing the faith.

[NinjaTrader_MichaelM]

Hello DaFish,

Thank you for your note. It sounds like you are already utilizing the support system to address and investigate specific behavior experienced by your current system but if you have any questions for me, please feel free to let me know.

[bluelou]

DaFish,
All backtesting will vary from live results. Probably the best you can do is in a 'market replay' type environment since each market event is evaluated as it was recorded. However, I still had issues where NT would neither fill nor reject some resting limits.

It's probably not fair for me to discuss competing products or services (you can pm me).

Having said that, I did make decent progress with prototype models in NT given its low cost. It took the help of a NT developer promoted on the website here. I'd highly recommend him. His name is Muly Oved (based in Israel, forgot his firm name- 'financial algorithms' something or other). He's 'expensive' but I tried a few other developers promoted here and they literally took my money and ran. Real scum bags.

I'll still use NT for quick prototyping just b/c I have code here already. Also, it's a perpetual license versus a few thousand $/month for pro platforms. So, I'll always have it.

I am really struggling with getting NT to work, and to be able to back test my stuff. I get COMPLETELY different results. I use multiple time frames and I can't even seem to get the right data when my strategy starts up from older time frames. When it finally loads and runs live, it seems to work.

Running LIVE market data gives me the proper results, but I cannot seem to back test at all with any consistency or trust in the system though.

I am not sure if this is related to my skill level, or just knowledge in general but I have thousands of lines of code trying to get this to work and 2+ YEARS of NT7 work invested.

I have had to stop on NT8, I did port all my code over, but market replay crashes and is unusable, I think some of my code didn't port correctly, and NT8 crashes ... to the point of giving up on NT8 because I HAVE to finish the work debugging my trading system so I can actually start trading it automatically. I will wait for the released version.

That said in NT7 I don't seem to be close to trusting the system enough to use it.... which is exceptionally disappointing given the time and effort put into the systems.

bluelou, what system did you replace NT with please? And why are you still using NT?

I will continue to put in support tickets and work through my issues on NT7 - but I have to tell you I am losing the faith.

[bluelou]

One more thing. You'll have to write your own fill logic for backtesting. The fills you'll get from their default fill code will never match what you'd get from an intraday trading strategy in real-time. You can take the NT default version and modify it.