Backtesting with playback at various speeds

[utilizator]

Hello,
I have the same issue reported at https://ninjatrader.com/support/foru...d.php?t=106818
namely, I get different orders depending on the playback speed.
There is no solution posted there.

To exemplify the issue, attached are results for 1x, 10x and Max speed, respectively.
The strategy uses Calculate.OnEachTick
I am using NT 8.0.14.1 64-bit

What should I do to avoid this discrepancy? Thank you

[NinjaTrader_EricB]

Hello utilizator,

Thank you for your post. Could you possible post simple sample with instructions on how to reproduce this behavior?

I look forward to your reply.

[utilizator]

Hi Eric,
Unfortunately, I cannot provide the code for my custom strategy and indicators. I tried to reproduce the issue by using the NT8 sample strategies, but none produced a single order, I don't know to properly launch them, apparently.

I'm not sure if the below steps are relevant or not without the actual strategy to run, thanks

Control Center > New Chart > Data Series: End Date =May25, 2018; Day to Load=1
Control Center > Connections > Playback >Market Replay; Start=May 25, 2018; End=May26, 2018

Playback: GoTo May25,2018 9:30AM
Playback: set 1x speed
Chart>Strategies>MyCustomStrategy
This will generate the 1st image I attached

Playback: GoTo May25,2018 9:30AM
Playback: set 10x speed
Chart>Strategies>MyCustomStrategy
This will generate the 2nd image I attached

Playback: GoTo May25,2018 4:05PM
Playback: set MAX speed
Chart>Strategies>MyCustomStrategy
This will generate the 3rd image I attached

Playback: GoTo May25,2018 4:05PM
Chart>Strategies>MyCustomStrategy
This will generate the 3rd image as well

[NinjaTrader_Jim]

Hello utilizator,

We will be happy to assist you further, although what you are describing would need to be broken down to a root cause so we could give further input or report the matter further. Without the code, we could not give clear advise on how to avoid what you are encountering.

The detail you provided is helpful, but we would need to have a strategy to run against and we would also need to know which instrument, bar type, and period to use so we can generate the same trades and behavior you are witnessing. If you cannot reproduce with the SampleMACrossover strategy that comes with NinjaTrader, we can move forward with a reduced and modified version of your strategy or a simple test strategy that can reproduce.

You may also reach out to me at platformsupport[at]ninjatrader[dot]com if your reduced sample contains some code that you would like to keep off the public forum. Please include the text "Attn Jim" and a link to this thread if that is the case.

I'm happy to be of any further help with this matter.

[utilizator]

Hi Jim,
I managed to somehow reproduce the issue with the Sample multi-timeframe strategy.
Although there was no difference in 'realtime' orders when Playback at 1x, 10x and MAX speed (as exhibited by my custom strategy),
the orders are different from the 'historical' ones.
By 'historical' I mean that:
* Playback > GoTo 16:00
* Chart > Strategy>Sample multi-timeframe; CalculateOnEachTick
* orders generated up to the date on the Playback controller ( in this case 16:00)

Please see attached images.
I attached the tick data as well, for June 07, 2018; instrument ES 06-18, thanks

[NinjaTrader_Jim]

Hello utilizator,

Thanks for the clear and detailed information.

There are going to be differences in order fills and strategy performance when comparing historical and realtime trades since historically; the data series that submits the order will be used to simulate the order fill instead of the realtime data. However, the difference that you are seeing may not have to do with Historical Fill Processing, but may have to do with the strategy using different Calculate modes. Historically, Calculate.OnBarClose will be forced unless Tick Replay is enabled on that chart. This would mean that when using Calculate.OnEachTick or OnPriceChange, OnEachTick/OnPriceChange is only used with "realtime" streamed data.

For the thread's reference, I have included links to publicly available documentation for for the items mentioned above.

Discrepancies: Realtime vs. Backtest - https://ninjatrader.com/support/help...ime_vs_bac.htm

Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm

Understanding Historical Fill Processing - https://ninjatrader.com/support/help...ical_fill_.htm

Please let me know if I can assist you further.

[utilizator]

Hi Jim,

Historically, Calculate.OnBarClose will be forced unless Tick Replay is enabled on that chart

.
But I did enable Tick Replay, as can be seen from my latest image attached (the Properties one). Is there anything else to be set in order for Historical to match Realtime in backtesting? thanks

[NinjaTrader_Jim]

Hello utilizator,

I see that you are using Market Replay data, but I do not see any Data Series properties showing Tick Replay being enabled. The multiple submissions would also be limited by the EntriesPerDirection property, so we may not be looking at multiple submissions at all.

Digging a little further, we can see that the SampleMultiTimeFrame strategy is submitting orders with a quantity of 1000, and we see execution markers for smaller quantities of that order. What we are looking at with this strategy are partial fills seen when data is streamed into the strategy (realtime) vs. simply filling the orders with the full quantity as seen with historical processing. I've edited my previous message to be clearer.

There isn't an option to avoid partial fills in the simulator since the result of partial fills would be closer to what would be expected with real markets.

I'd be sure to double check your strategy to be certain that the executions total the quantity you are submitting. This would be OK. If there are any other discrepancies involving multiple trades being submitted within the same bar to consider the items in my previous reply, or to confirm that there is not an additional data series that is allowing the strategy to submit multiple orders.

Please let me know if you have any questions.

[utilizator]

Hi Jim,
If I correctly understand, only 1-tick bar charts will generate the same orders in both backtesting mode and realtime/playback mode. Is this correct?

[NinjaTrader_Jim]

Hello utilizator,

Orders generated should be consistent between backtesting/sim trading/live trading so long as they are only based on price data. If the strategy uses strategy performance information we may see some differences in orders generated because of order fill differences.

Order fills will show differences when testing against a series other than a single tick series because the bar is used to simulate the order fill, not the tick data. This is where High Order Fill Resolution or adding a 1 tick data series to submit orders to will come into play. This feature or submitting orders to a single tick data series will use another data series to simulate the order fills, thus making them closer to what we would see in real markets.

More information for using High Order Fill Resolution can be found below. I've also included an example for Backtesting with intrabar granularity that can be used with Multi Series strategies.

Order Fill Resolution - https://ninjatrader.com/support/help...FillResolution

Backtesting with intrabar granularity - https://ninjatrader.com/support/help...ipt_strate.htm

Finally, my colleague Chelsea has a great resource breaking down discrepancies between historical and realtime data, and what can be done about it.

Comparing Historical Realtime and Replay Performance - https://ninjatrader.com/support/foru...d.php?t=102504

I look forward to being of any further assistance.