Playback Market Replay multi timeframe bug?

[Brillo]

I have a multi-timeframe strategy that day trades on the US Equities RTH trading hours

template using 3, 5 and 10 minute bars and swing trades on the CME US Index Futures ETH trading hours using 1440 minute bars.

When I backtest using the Strategy Analyzer I get the trades I expect. I then wanted to confirm the trades using Market Replay connection. The 1440 minute bar trades do not match.

The MR version is recording trades that begin and end at 9:30am instead of at the start of the CME US Index Futures ETH session ( 6 pm ).

Strategy Analyze enters price: 3097.50 at 11/21/2019 5pm:
Market Replay enters price: 3109.00 at 11/21/2019 9:30AM.


While debugging the Market Reply test, the EnterLong is called correctly at Time[0] == 11/21/2019 5:00 PM:



But as you can see from the performance report it's not being recorded as being filled at the time and price of the open of the CME US Index Futures session.


I have attached a video that shows this happening.
The strategy I used to create this is also attached.
AtsSampleMultiTimeFrame.zip

[NinjaTrader_ChelseaB]

Hello Brillo,

As a heads up, use null instead of Instrument.FullName.
(Tip 4 in the help guide)

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

Are you using 1 tick granularity for further accuracy with fills?

Below are links to forum threads that discuss this.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

[Brillo]

Hi Chelsea,

I watched your video and read through the information you sent me. I can see this subject is something you're familiar with!

I went back to my script and changed it so that the orders placed for the 1440 minute bars use a 1 minute bar granularity:

[LEFT][COLOR=#4D4D4D][FONT=Helvetica][SIZE=13px]EnterLong(barsIndex1m, 1, rsi.Name + "LE");[/SIZE][/FONT][/COLOR][/LEFT][LEFT][COLOR=#4D4D4D][FONT=Helvetica][/FONT][/COLOR][/LEFT]

Unfortunately, after running the changed strategy in the Market Replay with this change I see no change in the results.

The difference between with fills in the Strategy Analyzer and Market Replay are not small. The Market Replay version, which is wrong, is filling the market order at the start of the bar for the data series running on the US Equities trading hours template, 9:30 am. It should be getting 15 1/2 hours earlier at 6pm EST the previous night ). The US Equities trading hours template is used in the script for 3 other data series, the 3, 5 and 10 minute bars.

I changed the script further so that only the trades for the 1440m bars are taken. The dataseries for 3, 5 and 10 minute bars on the US Equities RTH are still loaded.

[ATTACH]n1093180[/ATTACH]

I also added a true/false property that controls whether or not to add the 3, 5 and 10 minute bars running on US Equities RTH. This way you can run two instances of the script, one with the US Equities RTH bars and one without them.
I did this and the results show that the presence of these US Equities RTH bars together with the CME US Index Futures ETH in the same strategy bring up the bug.
In the screen shot below the chart on the left runs the script with false, ( no US Equities RTH bars ) and the chart on the right is true, meaning to include them.

You can see that the script on the left records the sell order filled at the open of the bar on Oct 28. at 3024.25, while the chart on the right records it in the middle of hte CME US Index Futures ETH bar. You'll see in the trades register below that this is at 9:31am. ( 1st script would have recorded it at 9:30am but now I'm submitting the orders on the 1 minute bar for better resolution as you suggested ).



Trades earlier then 10/23 were backfilled and have the correct fill times.

If you run the script attached you should be able to reproduce the issue.

[NinjaTrader_ChelseaB]

Hello Brillo,

Do you not have access to historical tick data to use 1 tick granularity instead of 1 minute granularity for increased accuracy?

The Playback is basically the same as real-time.

When you mention the fill price is wrong, can you provide output from the output window showing this? (Right-click the output window > select Save as)

What is the current bid or current ask at the moment the order is filled in OnExecutionUpdate()?

Please include a print of the order object.

Below is a link to a forum post on using prints to understand behavior.

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/99209-date-conditions-not-working?p=791121#post791121

[Brillo]

I have changed the script to use 1 tick granularity. It takes a very long time to load. Remember my real strategy will be using at least 200 1440 minute bars. Loading a parallel set of 1 tick bars is not practical. But for this study it's alright.
AtsSampleMultiTimeFrame.2020-04-06.12-51pm.zip

The output including order objects is here:
NinjaScript Output 4_6_2020 12_50 PM.txt
The script prints 2 lines per order execution. The 1st line shows the RunDayTrades setting along with enough information to see the problem. The 2nd line is the order object you requested.

The last 2 log entries ( 4 lines ) tell the story. I'm running 2 instances of the attached indicator on a 1440 minute chart. RunDayTrades true and RunDayTrades false. The script is set to no longer actually order day trades, but it does load the 3, 5 and 10 minute bar series on the US Equities RTH session template when RunDayTrades is true.

10/22/2019 5:00:00 PM Realtime RunDayTrades?: False OrderName: 1440m_LX Market OrderTime: 10/21/2019 6:00:01 PM AverageFillPrice: 3006.5 Bid/Ask:3006.5/3006.75
orderId='312d9321a6ec479481df2782959c2a7f' account='Playback101' name='1440m_LX' orderState=Filled instrument='ES 12-19' orderAction=Sell orderType='Market' limitPrice=0 stopPrice=0 quantity=1 tif=Gtc oco='' filled=1 averageFillPrice=3006.5 onBehalfOf='' id=6464 time='2019-10-21 18:00:01' gtd='2099-12-01' statementDate='2019-10-21'

10/22/2019 5:00:00 PM Realtime RunDayTrades?: True OrderName: 1440m_LX Market OrderTime: 10/22/2019 9:30:01 AM AverageFillPrice: 3011 Bid/Ask:3011/3011.25
orderId='a7974b4043bf43efadd26125ffa9f1c4' account='Playback101' name='1440m_LX' orderState=Filled instrument='ES 12-19' orderAction=Sell orderType='Market' limitPrice=0 stopPrice=0 quantity=1 tif=Gtc oco='' filled=1 averageFillPrice=3011 onBehalfOf='' id=6465 time='2019-10-22 09:30:01' gtd='2099-12-01' statementDate='2019-10-22'

When RunDayTrades is false the order is executed at 6:00:01 PM.
When RunDayTrades is true the order is executed at 9:30:01 AM.

[NinjaTrader_ChelseaB]

Hello Brillo,

What is the current bid or current ask at the moment the order is filled in OnExecutionUpdate()?

Is the order being submitted at the same time?

What are the values used in the condition to place the order?

Have you reviewed the post I have linked that shows how to use prints to understand differences between playback, real-time, and historical?

[Brillo]

What is the current bid or current ask at the moment the order is filled in OnExecutionUpdate()?

It is in the output that I gave you

Bid/Ask:3006.5/3006.75 and Bid/Ask:3011/3011.25

Is the order being submitted at the same time?

Not sure. The order is a market order. I hear "Order Filled" occur at much different times. You can see the log entries in my previous post have a time stamp at the beginning of each line. That's the current bar Time[0]. They match. But the order filled time from the order object has the times that differ and are the exhibit of the problem.

What are the values used in the condition to place the order?

            RSI rsi = RSI(3,3);

            if (rsi[0] < 60)
                EnterLong(barsIndex1m, 1, rsi.Name + "LE");
            else
            if (rsi[0] > 70)
                ExitLong(barsIndex1m, 1, rsi.Name + "LX", rsi.Name + "LE");

I sent you the code so you can see it. Have you had a chance to import the strategy?

Have you reviewed the post I have linked that shows how to use prints to understand differences between playback, real-time, and historical?

Yes I have but I did not discover anything that helps understand why this problem is happening or how to fix it.
I have given you everything I can think of to describe the problem
a video
screen shots of the charts showing the different fill prices
all the code in an importable
log files with highlighted to show the problem

But it seems that we are not on the same page. I would expect that you at least import the strategy and run it to reproduce the problem. It's not hard to reproduce.

[NinjaTrader_ChelseaB]

Hello Brillo,

If the Bid is 3006.75 and the sell market order filled at 3006.75 this does appear correct.

Unfortunately, I cannot debug the script on your behalf. I can only guide you to fixing the script yourself.

You can also contact a professional NinjaScript Consultant who would be eager to create or modify this script at your request or assist you with your script. The NinjaTrader Ecosystem has affiliate contacts who provide educational as well as consulting services. Please let me know if you would like our business development follow up with you with a list of affiliate consultants who would be happy to create this script or any others at your request.

Would like someone to debug this script on your behalf?


That said, it does appear that the order is filling with the correct price.
Possibly the order is being submitted due to the logic.

You have provided code, you have not provided the values used in the conditions as requested.

Have you reviewed the forum post that demonstrates how to use prints to understand behavior?

What is the RSI when the order is submitted in real-time? What is the RSI when the order is submitted in playback?

Did you use prints to make sure the first historical bar is the same?

[Brillo]

Hello Brillo,

If the Bid is 3006.75 and the sell market order filled at 3006.75 this does appear correct.

Yes the price is correct for the time at which the order is recorded as a fill. The problem is that the time is wrong. So focusing on the bid/ask is not helpful in understanding the issue I'm raising.

Unfortunately, I cannot debug the script on your behalf. I can only guide you to fixing the script yourself.

I don't think there's a bug in my script. I'm not looking for a way to program around this problem. I'm reporting what I think is a bug in the Market Replay of NinjaTrader.

Would like someone to debug this script on your behalf?

No. I'd like someone debug NinjaTrader on NinjaTrader's behalf. Seriously. There's a bug in Market Replay.

You have provided code, you have not provided the values used in the conditions as requested.

I'm sorry if I'm not understanding your request about the RSI. But in the interest of getting to the point it doesn't matter what condition is used. I have changed the script to enter long on Tuesday and Thursday and exit long on Wednesday and Friday.

AtsSampleMultiTimeFrame.2020-04-07.zip

This should take any consideration of data values out of consideration.

Have you reviewed the forum post that demonstrates how to use prints to understand behavior?

I have not seen the behavior that I'm describing in any other forum post.

What is the RSI when the order is submitted in real-time? What is the RSI when the order is submitted in playback?

Did you use prints to make sure the first historical bar is the same?

If you simply run the script the way I have described the problem I'm raising will be perfectly obvious. I question is not why is the results different in the Strategy Analyzer vs Market Replay ( they are but that's distracting from the issue ). The problem is that in Market Replay, if you run the same script twice, one with suplementary data series on the US Equities RTH template and another without those suplementary data series, the trades recorded on the primary data series are different. The logic does not use the suplementary data series at all... it simply loads it. The logic only places market orders when the bars in progress are the primary and only places market orders on 1st tick for bar.

When you run the script with RunDayTrades true and again with RunDayTrades false the order is submitted at different times. With false is submitted as it's supposed to be, at the open of the CME US Stock Index Futures 1440 minute bar. With true it's submitted at the wrong time, at the start of the US Equities RTH 1440 bar. This later bar is a supplemental series that's loaded.

10/8/2019 5:00:00 PM Realtime RunDayTrades?: False OrderName: 1440m_LE Market OrderTime: 10/7/2019 6:00:01 PM AverageFillPrice: 2935.25 Bid/Ask:2934.75/2935.25
orderId='64cce2ba9ceb423a9e42ed81f078a465' account='Playback101' name='1440m_LE' orderState=Filled instrument='ES 12-19' orderAction=Buy orderType='Market' limitPrice=0 stopPrice=0 quantity=1 tif=Gtc oco='' filled=1 averageFillPrice=2935.25 onBehalfOf='' id=6466 time='2019-10-07 18:00:01' gtd='2099-12-01' statementDate='2019-10-07'

10/8/2019 5:00:00 PM Realtime RunDayTrades?: True OrderName: 1440m_LE Market OrderTime: 10/8/2019 9:30:01 AM AverageFillPrice: 2916 Bid/Ask:2915.75/2916
orderId='4f71fa9139ed4cef9a502acc780b3390' account='Playback101' name='1440m_LE' orderState=Filled instrument='ES 12-19' orderAction=Buy orderType='Market' limitPrice=0 stopPrice=0 quantity=1 tif=Gtc oco='' filled=1 averageFillPrice=2916 onBehalfOf='' id=6467 time='2019-10-08 09:30:01' gtd='2099-12-01' statementDate='2019-10-08'

[NinjaTrader_ChelseaB]

Hello Brillo,

You have mentioned there is a bug with the Playback connection with Market Replay.

Without focusing on your strategy performance (which is from the logic coded in the strategy) what is the bug you are wishing to report so I may test this on my end?

Is the data appearing on the chart incorrect?

Are the ask and bid prices incorrect as the data plays back?

What are you comparing this with to know the data is incorrect, without just looking at the strategy performance?

Specifically, what values are different?

[Brillo]

Without focusing on your strategy performance (which is from the logic coded in the strategy) what is the bug you are wishing to report so I may test this on my end?

The bug has everything to do with how strategies orders are processed within the MarketReplay. It's not possible to explain it without a strategy. I have been trying to explain it and based on the questions you're asking me it seems as though you haven't examined the material I've given you. You should see that the order fills are different if a secondary dataseries is loaded.

In the hopes that maybe you'll take a closer look I've prepared yet another video and another strategy. This strategy is very close to the NInjaTrader suppllied @SampleMACrossOver strategy. The only material change is that it can load a secondary data series if you set a property to true. The secondary data series is not used in the strategy. No actions are taken on BarsInProgress == 1. The mere presence of the dataseries causes the OnBarUpdate for BarsInProgress == 0 to occur at the time that the secondary dataseries opens it's bar and that's wrong because the secondary data series opens it's bar at 9:30 am while the primary opens it at 6pm the night before.. It's much more clear if you watch the video.

https://drive.google.com/open?id=158...eQFYF6z3PXVSCt

PLEASE WATCH THE VIDEO.

I'm attaching the strategy

SampleMACrossOverMTF.zip

as well.

[NinjaTrader_ChelseaB]

Hello Brillo,

As a heads up, when adding multiple series, all series must be in session before the script will start triggering OnBarUpdate().

From the help guide:
"If your NinjaScript object is using AddDataSeries() allowing to specify a tradingHoursName, please keep in mind that: An indicator / strategy with multiple DataSeries of the same instrument will only process realtime OnBarUpdate() calls when a tick occurs in session of the trading hour template of all added series. Any ticks not processed will be queued and processed as a tick comes in for all subsequent DataSeries."

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

Is this the issue you are having?

Are you not seeing OnBarUpdate triggering when one of the added series is outside of market hours?

I have watched the video you have linked in its entirety twice to make sure I'm not overlooking where you have prints going to the output window, but I am not seeing this.
The video mostly seems to be of the code.

[Brillo]

Yes that's the problem. Not so obvious. Doesn't happen in backfill. Good thing I tested it in MarketReplay. Is there an open enhancement request to make it work so that the ticks come in even if not all added series are in session?

[NinjaTrader_ChelseaB]

Hello Brillo,

Yes, I will add your vote to this request.

This request is being tracked with ID# SFT-3654.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm