Tick replay backtest and order fill

[mantak007]

Hello, I'm not sure I understand the order fill concept.

I have a strategy using OnMarketData (this is given, no way around that). To backtest this strategy, I seem to have no other option than using Tick Replay. I'm running backtests on 5 min timeframe and the orders (market orders by EntryLong, EntryShort) are filled only with 5 minute granularity, completely out of the market price at the moment the order has been placed. This of course completely ruins the results.

Can you please advice, am I missing something, or is Ninjatrader not supporting this scenario at the moment? Is it possible to run the backtest with the setup above, and get correct trades? Thank you.

[NinjaTrader_Jim]

Hello mantak007,

Thanks for opening the thread.

Historical fill processing in the Strategy Analyzer will use the primary data series to simulate order fills. To add intrabar granularity, most strategies can use High Order Fill Resolution with a single tick data series to have orders filled against a single tick data series.

High Order Fill Resolution is not compatible with Tick Replay, so a Tick Replay strategy must add a single tick data series on its own and submit orders to that single tick data series for intrabar order fills.

This is demonstrated in the backtesting intrabar granularity example. I'll provide a link and additional public reference for multi series NinjaScripts and Historical Fill Processing.

Backtesting with intrabar granularity - https://ninjatrader.com/support/foru...ead.php?t=6652

Multi Series NinjaScripts - https://ninjatrader.com/support/help...nstruments.htm

Historical Fill Processing - https://ninjatrader.com/support/help...ical_fill_.htm

Please let me know if I can be of further assistance.

[mantak007]

Hello Jim, is the approach described in "Backtesting with intrabar granularity" compatible with the OnMarketData()? My understanding is OnMarketData() is not called without Tick Reply enabled, or is it?

[NinjaTrader_Jim]

Hello mantak007,

Yes it is. Tick Replay must be enabled to playback OnMarketData events historically. The approach in "Backtesting with intrabar granularity" adds an additional data series and submits orders to that data series, thus changing the data series used to fill the orders. OnMarketData() can still be used to call your order entry and exit methods.

Please let us know if you have any additional questions.

[mantak007]

OK, got it, all seems to work. Thank you.

[mantak007]

Hi Jim, seems there is another issue. I'm successfully using the tick data series for entries, however the exits managed by SetStopLoss and SetProfitTarget are still executed on the primary data series, so in my case the exits are executed with 5 minute granularity and at a wrong price. Is there a way how to "attach" the SetStopLoss and SetProfitTarget management to the tick data series? If not, can you please submit a feature request for the development?

[NinjaTrader_Jim]

Hello mantak007,

Yes, the Set methods will still use the primary data series and cannot specify a BarsInProgress to submit the order to. We have a feature request tracking interest in this and I will submit a vote on your behalf. The ticket ID is SFT-689.

As with other feature requests, we cannot present an ETA as they are fulfilled based on the development team's schedule and priorities. Upon implementation the ticket ID can be found publicly on the Release Notes page of the help guide. I'll provide a link below.

Release Notes: https://ninjatrader.com/support/help...ease_notes.htm

Currently, the advised approach is to use your own Exit methods that can specify a BarsInProgress index for intrabar granularity.

If there is anything else we can do to help, please let us know.

[saturntd]

As of NT8 8.0.16.3 has SFT-689 been incorporated into NT8?

I find that if I use Tick Replay on a chart when looking at a historical backtest, I get dramatically better results with tick replay than without, even though I'm not using any indicators that need bid/ask data.
This regardless of whether or not I use OrderFillResolution.High & specify a 1 tic BarsPeriodType.

This is my 1st real strat in NT8 & I haven't done much live testing yet so am wondering if historical backtests results are at all accurate.

[NinjaTrader_Jim]

Hello saturntd,

SFT-689 has not been implemented at this time. Currently there are 3 votes behind the request. If there is more demand it will be more likely to be considered for implementation.

Tick Replay will allow NinjaScripts to Calculate with OnEachTick or with OnPriceChange in addition to allowing historical use of OnMarketData. This could explain a difference in performance that you are seeing as the signals will not be triggered OnBarClose.There are also known differences with order fills which should be noted comparing these features to standard backtests.

Another item to note is that Tick Replay is not compatible with High Order Fill Resolution, so order fills will still be simulated using "virtual bars" the same way they would be simulated with Standard Fill Resolution. If you would like to mix Tick Replay with High Order Fill Resolution, you will have to add an additional data series to submit your orders to. This would involve using Exit methods to specify the BarsInProgress index since the Set methods will still use the primary data series for order fills.

To better explain these discrepancies, I have included links to our documentation on discrepancies between Realtime and Backtesting. Please also see Understanding Historical Fill Processing linked in post #2. An example demonstrating submitting orders to an additional data series for intrabar granularity is also linked in post #2.

Discrepancies between Realtime and Backtesting - https://ninjatrader.com/support/help...ime_vs_bac.htm

Further notes for using Tick Replay are noted in the help guide page linked below.

Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm

Let me know if you have any additional questions.

[saturntd]

Hello Jim,
Thanks for the response.
How can I vote for SFT-689?

[NinjaTrader_Jim]

Hello saturntd,

You can let one of us know that you would like a vote added. I've also added a vote on your behalf.

Let me know if there is anything else I can do to assist.

[saturntd]

Thanks Jim

[KarstenKafl]

Hi saturntd and mantak007 and Jim - did any of you ever get anything near the same results in backtest/realtime - as you had when backtesting with Tick Reaplay?

I have the exact same thing - where i get very good performance - when using the tick replay in backtest.

Im using just single or two timeframes - and my question is - how can the results be so much better in backtesting/chart view with that Tick replay turned on - if this is meant to be bit more precice than testing without it?

= is there a way i can make a strategy perform more like it does with tick replay turned on?

For a simple cross above/below EMA strategy - If i test it with just the order fill resolution set to high - and 1 tick - then the performance is basicly negative - but with the tick replay turned on it is very positive.

Thanks for any replies

[NinjaTrader_Jim]

Hello KarstenKafl,

I never perform backtests with the goal of getting realtime and historical to match exactly. These forms of testing are fundamentally different and we need to understand these differences so we can use those tests reliably.

There is not any intrabar movement in a backtest (historical processing.) With realtime simulations we have intra bar processing, but there is going to be added latency with order submissions and increased slippage.

Tick Replay adds intrabar logical processing but orders will still be filled based off of the data series we submit the order to.

High order fill resolution can be used to change the data series used to fill the order so it is filled with more granular data, but this is not compatible with Tick Replay. You will need to submit orders to a single tick data series to have intrabar order fill simulations along with intrabar actions (Tick Replay.) See post #2.

With all this in mind, we are still filling orders based off of a single tick data series, when realtime simulations will be filled with bid/ask/volume and live orders are filled with actual market dynamics. We can still expect differences, although we would have then taken all steps we can take to minimize these discrepancies.

Testing with the Playback Connection is advised as the next step after backtesting as this will give more true-to-market results.

My colleague Chelsea has put together a thread with in depth videos on the matter, linked below.

Comparing real-time, historical, and replay performance — https://ninjatrader.com/support/foru...mance?t=102504

We look forward to assisting.

[KarstenKafl]

Hi Jim

Thanks for you reply.

So three more questions.

1.
If i use the high order fill resolution in the the strategy - then when watching the results as on the chart - if i have set the
Calculate = Calculate.OnEachTick;

and i have again simple go long on close cross above EMA - will i not be able to see the "correct" order placement on the chart?

2. Will the fill limit order on touch have anything to do with this crossover on historical on chart?

3.
The Playback Connection have interbar ticks right? so it will only differ from real life - because of orders slipage?

THanks.