Go Back   NinjaTrader Support Forum > NinjaTrader 8 > Strategy Development

Strategy Development Support for the development of custom automated trading strategies using NinjaScript.

NinjaTrader
Reply
 
Thread Tools Display Modes
Old 11-12-2017, 03:32 PM   #1
iantg
Senior Member
 
Join Date: Jan 2015
Location: Charlotte, NC
Posts: 108
Thanks: 11
Thanked 13 times in 10 posts
Default Market Replay 7 vs. 8 Discrepancies

I have recently started migrating my NT7 strategies over to 8 to start to compare results between market replay on both platforms. From what I have read I am expecting to have similar results in market replay, though I have heard that significant improvements have been made to make the basic back-testing more accurate in 8. From what I have seen published and promoted by NinjaTrader, Market Replay in 7 was generally considered accurate and a good representation of how how live trading in the market would perform. It was my impression that there would not be a significant deviation in market replay results between these two platforms, but nonetheless I am in the process of going through some validations to check some of my strategies against 8 now.

After seeing some initial differences, (some small and some huge), I think I have found a good example of one that has major enough differences that it should be looked at by NinjaTrader support. I recreated the two strategies in both 7 and 8 reasonably well, so that they can be tested side by side to evaluate what is causing the large discrepancies.

It's a very simple test strategy just to see how limit orders are handled specifically on the exit by way of a profit target. Therefore I am using a 1 tick profit target against a greater stop loss, just to see how and when a 1 tick limit order (Profit Target) will not get hit on an exit. Here are my observations: The NT7 strategy has around a 20-1 win to lose ratio while the NT8 strategy has maybe a 10 to 1 or 15 to 1 win to lose ratio.

So I guess my question is how would you explain these differences and which assumptions are more representative of the real market? Are we assuming in 7 that profit targets are always filled on touch? Are we assuming that in 8 all profit targets are only filled on pass-through? Or is it less black and white?

Either way I would like to know the underlying assumptions being made by both 7 and 8 regarding this, so I can understand this behavior.

For testing purposes I used the ES Jan 04-8 2016 on a 250 tick chart in Market Replay.

Please let me know if you need any further information to test on your end.

Thanks,

Ian
Attached Files
File Type: cs NT7Test3.cs (3.1 KB, 8 views)
File Type: cs NT8Test3.cs (4.0 KB, 4 views)
iantg is offline  
Reply With Quote
Old 11-12-2017, 05:54 PM   #2
NinjaTrader_ChelseaB
NinjaTrader Customer Service
 
NinjaTrader_ChelseaB's Avatar
 
Join Date: Dec 2012
Location: Denver, CO
Posts: 8,639
Thanks: 126
Thanked 1,889 times in 1,761 posts
Default

Hello iantg,

There have been some changes to the real-time simulation engine. I will need to further research specifics about how limit orders fill in real-time.

In the meantime, I am providing a link to a forum post that demonstrates how to find what is causing differences between tests.
https://ninjatrader.com/support/foru...402#post514402

Are you including any historical data with your tests or are you ensuring that no historical data is included?
NinjaTrader_ChelseaB is offline  
Reply With Quote
Old 11-12-2017, 06:15 PM   #3
iantg
Senior Member
 
Join Date: Jan 2015
Location: Charlotte, NC
Posts: 108
Thanks: 11
Thanked 13 times in 10 posts
Default

Hi ChelseaB,

Thank you for following up with me. The test were conducted in Market Replay with no data previously loaded historically that already had trades. In terms of the data itself.... It was replay data download from Kinetic in NT7. In NT8 when I loaded it for the first time it asked me if I wanted to migrate my existing data over, so it converted the original NTM file into a NRD file. But unless there was some degrading of the quality in the conversion process should in theory be the same.

If you run the two scripts I attached in 7 and 8 respectively using the ES 250 tick chart it will produce what I experienced. Around 20 to 1 winner to loser ratio in 7 and around 15 to 1 winner to loser ratio in 8 on the same trades. So this got me thinking that something has dramatically changed in the simulation engine between 7 and 8 though I have never heard anyone mention this anywhere.

I was just hoping to get to the bottom of this and understand the assumptions in 7 and 8. My guess is that this pertains to the topic of touch vs. pass through on exits using profit targets.

Let me know what you find out.

Thanks,

Ian
iantg is offline  
Reply With Quote
Old 11-14-2017, 08:54 PM   #4
iantg
Senior Member
 
Join Date: Jan 2015
Location: Charlotte, NC
Posts: 108
Thanks: 11
Thanked 13 times in 10 posts
Default Any Updates to this question?

Pertaining to Market Replay: What I would specifically like to know is the following:

1. What logic changed in the decision engine for filling entries and exits between 7 and 8 for Market Replay.

2. If it was determined that a major change needed to be made with 8, does this imply that the market replay engine in 7 was flawed? What are the specific areas that were determined to be incorrect, or unrealistic?

3. What would be the steps one would take to reconcile results between 7 and 8. Is this as simple as adding 1 extra tick for entries and exits? What would be needed to walk between 7 and 8?

4. What are the underlying assumptions made by each (7 & 8) with respect to execution speed. Would this be a good proxy for someone in Asia executing local entries / exits to the CME in Chicago, or would this be closer to someone collocating at the CME on a dedicated server? How would you characterize the standard assumptions built into the decision engine on both 7 & 8?

If anyone from NT or any senior members have any insight into this I would greatly appreciate it.

Thanks,

Ian
iantg is offline  
Reply With Quote
Old 11-14-2017, 09:06 PM   #5
sledge
Senior Member
 
Join Date: Aug 2010
Location: Washington, D.C.
Posts: 4,487
Thanks: 1,682
Thanked 1,094 times in 937 posts
Default

I downloaded both your files and ran a compare, and saw 1 exit fill missed by NT8. It was during high volatility. NT8 does further refinement beyond seconds. I don't know if the same order is maintained between NT7 (which is only at seconds) and NT8 converted data.

Well, if you ran both replays together and slowed it down at a discrepancy, you'll probably see what's going on.
sledge is offline  
Reply With Quote
Old 11-14-2017, 09:16 PM   #6
iantg
Senior Member
 
Join Date: Jan 2015
Location: Charlotte, NC
Posts: 108
Thanks: 11
Thanked 13 times in 10 posts
Default

Thank you sledge for taking the time to compare my two strategies. I have found specific orders that filled differently between the two, and it looks like it is mostly on the exits. In NT 7 there were more exits filled. In NT 8 it missed quite a few profit target exits. It also looked like it hit the market order exits easier in NT 8.

While I can observe this behavior and have my own theories about it, I would really like to understand the rational from NT about why the behavior was changed and then to learn more about the specific rule changes.

For anyone running strategies in Market Replay it would be very helpful to know if some of the underlying assumptions in 7 were less than realistic and they have since corrected them in 8, or if perhaps 7 would be more indicative of faster executions that could be achieved through low latency HFT systems and 8 is more representative of someone clicking from a local machine far away from an exchange.

I think it's obvious that the fill engine has dramatically changed from 7 to 8 but I have yet to hear any explanation as to what assumptions changed, why they made the decision to change it, and if one is considered more realistic to the markets vs. the other.

Any input would be appreciated.

Thanks,

Ian Guthrie
iantg is offline  
Reply With Quote
Old 11-15-2017, 12:03 AM   #7
sledge
Senior Member
 
Join Date: Aug 2010
Location: Washington, D.C.
Posts: 4,487
Thanks: 1,682
Thanked 1,094 times in 937 posts
Default

I'm not sure they can explain it.

My strategy takes a couple trades a day depending on conditions - and usually goes through the target without issue as it's way bigger than 1 point target.

I have about 3 trades over 3 years that "miss" in NT8.. considering how I busted their chops through the entire beta and beyond (and some how end up being mostly right), it's something I can deal with.
sledge is offline  
Reply With Quote
Old 12-04-2017, 02:05 PM   #8
NinjaTrader_PatrickH
NinjaTrader Customer Service
 
NinjaTrader_PatrickH's Avatar
 
Join Date: Jul 2011
Location: Denver, CO
Posts: 11,497
Thanks: 779
Thanked 1,977 times in 1,906 posts
Default

Hello iantg,

Thank you for your patience.

This was discussed in the past on the following thread: https://ninjatrader.com/support/foru...ad.php?t=96647

In short, there was logic in place in NinjaTrader 7 that would randomly 'guess' at a starting volume for Market Replay. This guess would make the calculation inefficient in NinjaTrader 7. To improve performance we elected for a more conservative approach. Therefore you will see some orders fill faster in NinjaTrader 7's simulator then you would see in NinjaTrader 8's.

Please let me know if you have any questions.
NinjaTrader_PatrickH is offline  
Reply With Quote
Reply

Tags
market replay

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Unable to Activate Market Replay Systems - Bulk Replay CaptainFill Platform Technical Support 5 10-08-2014 01:52 PM
Need help with finding and importing market replay data from market replay download teenagemutantNT Platform Technical Support 8 02-28-2014 07:23 AM
Live Sim and Replay discrepancies Downhillryder Platform Technical Support 6 03-07-2013 10:34 AM
Market replay - can i replay multiple charts simultaneously ? illumintai Platform Technical Support 1 07-07-2012 03:58 PM
Market Replay Connection Only Replay some hours of each day cbaer General Development 5 01-25-2012 01:23 AM


All times are GMT -6. The time now is 09:58 AM.

Futures, foreign currency and options trading contains substantial risk and is not for every investor. An investor could potentially lose all or more than the initial investment. Risk capital is money that can be lost without jeopardizing ones financial security or lifestyle. Only risk capital should be used for trading and only those with sufficient risk capital should consider trading. Past performance is not necessarily indicative of future results. View Full Risk Disclosure.

CFTC Rules 4.41 - Hypothetical or Simulated performance results have certain limitations, unlike an actual performance record, simulated results do not represent actual trading. Also, since the trades have not been executed, the results may have under-or-over compensated for the impact, if any, of certain market factors, such as lack of liquidity. Simulated trading programs in general are also subject to the fact that they are designed with the benefit of hindsight. No representation is being made that any account will or is likely to achieve profit or losses similar to those shown.

This website is hosted and operated by NinjaTrader, LLC (“NT”), a software development company which owns and supports all proprietary technology relating to and including the NinjaTrader trading platform. NT is an affiliated company to NinjaTrader Brokerage (“NTB”), which is a NFA registered introducing broker (NFA #0339976) providing brokerage services to traders of futures and foreign exchange products. This website is intended for educational and informational purposes only and should not be viewed as a solicitation or recommendation of any product, service or trading strategy. No offer or solicitation to buy or sell securities, securities derivative or futures products of any kind, or any type of trading or investment advice, recommendation or strategy, is made, given, or in any manner endorsed by any NT affiliate and the information made available on this Web site is not an offer or solicitation of any kind. Specific questions related to a brokerage account should be sent to your broker directly. The content and opinions expressed on this website are those of the authors and do not necessarily reflect the official policy or position of NT or any of its affiliates. 

Vendors along with their websites, products and services, collectively referred to as (“Vendor Content”), are independent persons or companies that are in no manner affiliated with NT or any if its affiliates. NT or any of its affiliates are not responsible for, do not approve, recommend or endorse any Vendor Content referenced on this website and it’s your sole responsibility to evaluate Vendor Content. Please be aware that any performance information provided by a vendor should be considered hypothetical and must contain the disclosures required by NFA Rule 2-29(c). If you are interested in learning more about, or investigating the quality of, any such Vendor Content you must contact the vendor, provider or seller of such Vendor Content. No person employed by, or associated with, NT or any of its affiliates is authorized to provide any information about any such Vendor Content.