A Testing environment that closely mimics Real Time results, for strategies.

[rasdaka]

Hello,

I have a basic strategy that places and exits trades based on certain conditions. I ran it for a specific time period over Real Time, and when the day is over, I download the historical data, and execute the same strategy, for the same time period, using Playback. And the results are ENTIRELY DIFFERENT. There's no correlation between the two, I even added a slippage of up to 16 ticks, and still am getting an inflated success rate on Playback, that's unrealistic and never duplicated during RealTime.

Note that I'm mentioning 'Playback' and not 'Historical Data', which yields results that are even more optimistic than Playback! So, we're not even going to talk about that! I'd much rather that the 'Playback' or the 'Test' environment used to develop the strategy yield results that are more pessimistic than Real Time run, so that it has a better chance of working out during actual execution in Real Time on actual money.

If one cannot work with results that are close to realistic during Playback, how can one expect to create a strategy that is ever successful in Real Time..??

Some of the responses to other posts suggest that I add a second 1 tick series, unlock the code (not at all my preference), then place all the orders on the tick series instead of the main data series. This is probably more work that I have time to come up to speed on the expertise for, but, does this even hold the possibility of close-to-real time results!??

If this is what one has to do to get realistic playback results to develop a strategy with, they why do you even have the Strategy Builder? If realistic results are just not possible without unlocking the code during development, it seems your Strategy Builder would simply mislead and fool users into thinking they can actually create a strategy that'll work in the real world!

How have others overcome this most basic stumbling block - of having a testing environment that yields realistic results to develop with iteratively..?

Thank you.

RK.

[NinjaTrader_ChrisL]

Hi rasdaka, thanks for posting.

We received a forum post and Email from your address. While we are happy to assist on the forums or with you privately through email with our [email protected] address, we respectfully ask that you only create one inquiry. This can be privately in email or can be a public thread on the forum.

If you would no longer like to communicate publicly on the forums, please let us know so and we will be happy to converse solely through email.

Creating both a forum thread and an email with [email protected] will create two separate tickets and may cause multiple technicians to start at the beginning of reviewing your inquiry, tying up resources with our platform support and delaying our ability to respond to all customers in a timely manner.

On the strategy you are using, we have a documentation guide here on the differences you can see in comparing a real time strategy vs a backtest, please review this page:

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?discrepancies_real-time_vs_bac.htm

To find out what exactly is different between each strategy run, you can switch to the Trades view in the top left and compare the trade collection of each strategy run. This same question has also been asked many times in the past, I posted some previous resolutions below:

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1153122-wildly-different-results-in-strategy-analyzer-vs-market-playback

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1148766-why-different-results-on-different-replay-results

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1168558-different-results-with-different-windows

Kind regards,
-ChrisL

[rasdaka]

Hi Chris,
Thanks for your response. Actually, I got the info from one of the posts here on how to contact the Scripting Support department. But, you're right: no need to send a note there if I'm going to post here. Yes, there have been other posts with similar questions. However, I didn't realize that merely adding the tick data series would yield a better result. I was thinking it required unlocking the code which I wanted to do at a later time. While now, Playback results do not match RealTime result exactly, its close enough of a graph that I think I can work with it.
RK

[NinjaTrader_ChrisL]

Hi rasdaka, Im happy to help out. Please let me know if there are any questions that come up.

[IsaacBillion]

Hello,

I have a basic strategy that places and exits trades based on certain conditions. I ran it for a specific time period over Real Time, and when the day is over, I download the historical data, and execute the same strategy, for the same time period, using Playback. And the results are ENTIRELY DIFFERENT. There's no correlation between the two, I even added a slippage of up to 16 ticks, and still am getting an inflated success rate on Playback, that's unrealistic and never duplicated during RealTime.


RK.

Hello rasdaka and NinjaTrader_ChrisL
I have the same problem.
I've asked it also in this post:

https://ninjatrader.com/support/foru...arket-playback

Hello Chelsea and MatthewLesko.
I've read all the other posts about differences between RealTime vs. BackTest:

https://ninjatrader.com/support/foru...nce#post100192
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1218724-a-testing-environment-that-closely-mimics-real-time-results-for-strategies
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1148766-why-different-results-on-different-replay-results
https://ninjatrader.com/support/foru...ay-vs-backtest

And also the TickReplay Guide:
https://ninjatrader.com/support/help...ick_replay.htm
But I Still don't figure it!

I took the simple CrossOver 2 SMA strategy and enabled TickReplay . Added intrabar granularity 1 tick bar to the secondary series.
Sent the order to the secondary bar series with the method overload with barindex=1 (I know that if I'm adding the 1 tick bar there isn't difference
if i enable or not TickReplay- Correct me if i wrong)
AND I STILL HAVE MAJOR DIFFERENCES BETWEEN Strategy Analyzer Backtest (with TickReplay Enabled) and PLAYBACK!
And Both of them i use HISTORICAL data (not Replay data because i couldn't find for TQQQ Replay Data - Can someone find me ?)
So i don't understand, it said in the TickReplay guide:
"Tick Replay is used to playback 1 tick historical data to build the bars as if they had been build live, this means that tick data will be thrown as Market Data events in historical and subsequently OnMarketData and OnBarUpdate events will be called as if it was live. "

So why i'm getting total different results between PLAYBACK and Backtest with TickReplay Enabled when the both using Historical data?
I understand there SHOULD be difference between Realtime or Simulated data because of the delay in order fill in RealTime vs. backtest and Playback which
fills the orders immediately.
But why the difference between backtest and Playback??

What do i need to do that Playback and backtest will be the same??
This is Critical because i'm trying to build a strategy and I'm optimizing via Strategy Analyzer to figure out the best settings via backtest with TickReplay
Enabled to simulate market data received tick by tick and using OnMarketData event. But if after i get great result in Strategy Analyzer and now moving to
Playback in the same dates and getting TOTALY DIFFERENT so what is the point?
I need a way to activate the SAME ENGINE of PlayBack as a backtest to the whole 3 months history to figure out the best settings.
How can i do that?
Isn't it the time that NT8 will unite these engines into one backtest and give same results that developer can build by that good Strategy?
Thank you Chelsea for the great Video and demo strategy for diff between BackTest/PlayBack/Realtime :

https://www.youtube.com/watch?v=2iM5BPpcUBg

But I steel don't understand why there is a diff between BackTest with TickReplay enabled and Playback and how to achieve same behavior to closely
resemble RealTime?
Thx So much for Your help.

Issac.

[rasdaka]

Hi Issac,
Thanks for your note. I've not had the time to go through your response in detail, but I'll leave here what I took from NTSupport responses, and what's working - so far!. That's this:
- Add, to your Strategy, another data series, which is a Tick Series, but the same instrument - the Primary Instrument. Just that in itself, provides close to real results - on Playback, not Historical. I'm going with that for now. Would recommend it as well.
Hope that helps!
Regards,
RK

[IsaacBillion]

Hi Issac,
Thanks for your note. I've not had the time to go through your response in detail, but I'll leave here what I took from NTSupport responses, and what's working - so far!. That's this:
- Add, to your Strategy, another data series, which is a Tick Series, but the same instrument - the Primary Instrument. Just that in itself, provides close to real results - on Playback, not Historical. I'm going with that for now. Would recommend it as well.
Hope that helps!
Regards,
RK

Thank you Rasdaka.
Already did that. Not working. Terrible difference between backtest with TickReplay on and Playback.
On BackTest i got 3% profit.
On Playback (with same historical data) loss of 8%.
I can put here full example, but you can try on any data you like , or the support can.
Hope there will be a way to get same results on both. This is what should be. On RealTime sure there should be some difference (not to much I hope).
Waiting to the Support Respond.
Best Regards.

Isaac.

[rasdaka]

Hi Isaac,

I compare Playback and RealTime results only. I do not use BackTest results AT ALL. Yes, that means that I use Playback all the time during development, but I don't see any other option: backtesting seems consistently off the mark so much that it doesn't seem to have value during development...

Best,

RK

[IsaacBillion]

Hi Isaac,

I compare Playback and RealTime results only. I do not use BackTest results AT ALL. Yes, that means that I use Playback all the time during development, but I don't see any other option: backtesting seems consistently off the mark so much that it doesn't seem to have value during development...

Best,

RK

Hi RK.
I'm also use Playback all the time. But this is the exact point I want help from Support.
Strategy Analyzer Optimizer should be a great tool to figure the best settings for a strategy you develop.
If i take for example 2 SMA Crossing, and i want to test for best settings which period to choose and with TimeFrame to choose (I also do optimization for Data Series (=Timeframe)
for a given Instrument, I can figure it out only by Backtesting via Strategy analyzer Optimization. And I'm also enabling Tickreplay That should mimic the Tick by Tick bars forming (onMarketData is also called).
So what is the point after getting the best Settings to play now PLAYBACK and find out these settings are all wrong?
What Settings should i Pick now? It's Crazy that i need now to check each settings with PLAYBACK - IT WILL TAKE YEARS!
So I need from Support a way to correlate of find a way PlayBack will give exact or same result like Optimization.
Why it haven't done so till now?
I use for playback THE SAME HISTORICAL DATA like for backtesting/optimization. So NT8 should engage some way to activate the PLAYBCAK engine for multi settings exactly like what
is happening now with Strategy Analyzer Optimization. I thought that this is what happening with TickReplay On that the Tick by Tick process that in Playback is working now on Strategy Analyzer Optimization - but i found THAT IS NOT!
So I'm asking Support to enable this feature even if it will take more resourses. I have now AMD 16 cores and willing to buy even AMD 3990X 64 core to activate Parallel Processing to do the work in no time.
I think also NT8 need to improve the multi core support for efficiency, because I'm working from VMWare virtual machine and I check changing from 4 to 8 to 16 cores and don't see any change in performance.
So if they put the playback engine into Strategy Analyzer engine it will be great!
Thx.
Best.
Waiting to Support response.

Isaac.

[NinjaTrader_ChrisL]

Hi Isaac, thanks for posting. We have already provided everything that can be done to improve the order fill accuracy in the backtest in the first post of this thread. The best thing to do in this scenario is to look at the individual trades by switching to the Trades view at the top left of the Strategy Analyzer, compare and contrast the trades that have been taken in real-time and in the backtest and see where they differ. Typically, it's at the beginning or end of the data set, it's not effective enough to look at the summary results e.g. average profit to tell where the difference in trades is coming from.

Kind regards,
-ChrisL

[IsaacBillion]

Hi Isaac, thanks for posting. We have already provided everything that can be done to improve the order fill accuracy in the backtest in the first post of this thread. The best thing to do in this scenario is to look at the individual trades by switching to the Trades view at the top left of the Strategy Analyzer, compare and contrast the trades that have been taken in real-time and in the backtest and see where they differ. Typically, it's at the beginning or end of the data set, it's not effective enough to look at the summary results e.g. average profit to tell where the difference in trades is coming from.

Kind regards,
-ChrisL

Thanks Chris for your reply.
But I don't understand.
Can you explain why PLAYBACK (not Realtime) is DIFFERENT from backtest/optimization with TickReply Enabled?
They both use Historical data (I can't get replay data for TQQQ) so why the so drastically different results? They should be the same, aren't they?
I use always the "Trades" section in the analysis view. but how can i explain difference in trades on the same historical data??

[NinjaTrader_ChrisL]

Hi Isaac, thanks for your reply.

I will not be able to guess why custom strategy is completely different, you are going to need to review the trade list as I suggested in my previous post and study the information that is given already. A strategy running in backtesting with no tick replay, no high resolution data series, and Calculate = OnEackTick or OnPriceChange will run OnBarClose in the backtest, so that would cause a big difference in the results compared to a real time run of the strategy.

Kind regards,
-ChrisL

[IsaacBillion]

Thanks Chris.
But Sorry. You are not answering my question.
If you need , I can post here the strategy the simple sample of 2 sma crossover with historical data and see it for yourself. But you can do it yourself of any instrument with 1 day Tick data.
My question was why if i enable TickReplay on backtest, even added 2nd data series of 1 Tick there are MAJOR DIFFERENCE between the backtest/optimization and the PLAYBACK with the SAME historical data.
Is there a way to do they will be EXACTLY the same? (some hidden flag field like TickReply or something else- this is what I thought will make Backtest work with the engine of PLAYBACK - but it DOESN'T - there are TOTALLY different results between them.
Maybe NinjaTrader_ChelseaB who made the video of the difference backtest/realtime can help and show me how to work properly to get results on the optimization which will be the same when running it in Playback.
Thanks.
Isaac.

[NinjaTrader_ChrisL]

Hi Isaac, thanks for your reply.

To find out the difference in your strategy between playback and a historical backtest, you will need to use Print() and also look at the Trades view of the Strategy analyzer to see the specific difference. E.g what specifically is different? Are there more or fewer trades in either data set? Are the trades the same but the exits are different? If the former, make sure the data being backtested is exactly the same and also take note of the BarsRequiredToTrade property to make sure the strategy is not skipping trades at the beginning of the playback test. If the ladder, the strategy is likely missing an order fill estimation and a higher resolution data series is needed.

If your strategy runs OnEachTick or OnPriceChange, it needs a 1 Tick data series added to it to A. Calculate the indicators and price movement and B. To submit orders to in lieu of High Resolution Order Fills. If you enable Tick replay within the strategy analyzer, the high-resolution data series can no longer be used, so it all needs to be done through the 1 tick series for intrabar grainularity.

https://ninjatrader.com/support/foru...nce#post100192 - A helpful post from our colleague about the "historical vs realtime" differences seen.

Kind regards,
-ChrisL

[IsaacBillion]

Thanks Chris.
So you're saying that if I'll make exact BarsRequiredToTrade the outcome from PlayBack and Backtest will be the same?
I'm making limit of trades only between 9:30 - 16:00. no trades allowed before that. So in that case BarsRequiredToTrade won't matter because data starts much sooner than 9:30 (7:00). Isn't it?
Also I'm adding 1 Tick second series and working only on Calculate= OnBarClose and sending all the orders to BarIndex=1 (second series).
In that case TickReplay won't matter. So will this make difference between Playback/Backtest?
If so, what I need to do so the outcome will be the same?

Thanks for your help.
Isaac.