Market Replay mysteriously 'switches' to the Realtime State

[pt-emini]

I hate starting a new thread but having now invested 2 + hours searching through pages of threads with no luck finding my needle in the haystack I decided to post my question/issue here for some helpful guidance.

Using Playback with market replay selected, I load 1 day of market replay data into a 200 volume chart of the CL on which I have an indicator with a sound alert (using PlaySound()). In the indicator I have the requisite encapsulating code as follows:

if ((State != State.Historical) && (IsFirstTickOfBar))
{
if (High[0] > High[1]) { PlaySound(@BuyAlertSound);}
}

On the Playback/Market Replay control I select a start date and time to load the chart with the market replay data. The chart starts constructing itself and as the data is loading the alert sound starts triggering in rapid fire succession. Just to be clear, playback is not running yet, all that is happening is the chart is populating, filling in the market replay data to the start time selected.

So I thought to myself, how is this possible ? I then decided to put a Print() statement just above that first if (State!=Historical...) statement to print the current state variable along with the bar number, and what shows up in the output window is about 1100 bars are loaded with the State of Historical, then all the bars after that, about another 2000 bars are loaded with a State of Realtime. Obviously these later bars that are loaded with the Realtime state are 'getting though' my protective if (State != Historical) and allowing the code inside to execute (resulting in the sound alert repeatedly triggering while the chart populates). So at the end of the day, my question here is simply, why is Market Replay changing the State variable from Historical to Realtime midway through the chart data population process ?

With all this said, this is code that I am porting into NT8 from NT7, which of course has been working perfectly fine in NT7 for many years. I have never encountered this type of behavior with Market Replay before in NT7, so assume it is either a new feature or a bug.

Thank you for your consideration and guidance.

[sledge]

Is this the new "tick replay" feature that you are encountering?

Right click, I think in the slider, and make sure current day only is checked.

[pt-emini]

Sledge - Thank you for your reply.

What I do is Connect to the Playback Connection

In the Playback control I make sure "Market Replay" is the selected state.

I then right mouse click in the window and select "Go To" , and also note here that "Playback current day" is checked (by default).

In the Go To control that pops up I set the desired date and time and click OK

The chart then populates with bars, and as this is happening the sound alert is triggering as described in my initial post. The debug output window Print() statements show the state change from historical to realtime mid-way through the chart load process. As noted, I have not started replay yet, simply selected the date and time to "Go To".

The sound issue is an annoyance, but this state change to Realtime is going to be a bigger problem when loading a chart with a strategy. The state needs to stay Historical until the entire chart is populated, and only transition to Realtime when the Playback is activated by clicking on the Play button in the Playback control window.

I have attached snapshots of the process here... I zoomed in on one image to show the demarcation line between Historical and Realtime in the debug log. All I did to create this was select a date/time in the GoTo window and click the OK button...

[NinjaTrader_MichaelM]

Hello pt-emini,

Thank you for writing in. I have confirmed this behavior and I am double checking with our product management team to determine whether this is a bug or if it is expected and if so why and what exactly is happening.

I will update this post when I have further information.
Thank you for your continued patience in the meantime.

[pt-emini]

Just an update with some new information on this issue.

During the chart loading / bar population process the method OnMarketData() is being called inadvertently as well, which is causing all sorts of havoc as one can well imagine.

In the NT8 User Guide for OnMarketData() we see the following note:

"2.By default, this method is not called on historical data"

Which obviously makes all the sense in the world. We don't want to be calling OnMarketData() when initially populating a chart with historical data. Unfortunately the actual implementation, by changing state to "realtime" while the chart is still loading, is violating rule #2 for OnMarketData().

[NinjaTrader_ZacharyG]

Hello pt-emini,

This behavior is expected.

I will be investigating further on how to avoid these real-time methods from being called when changing times on the playback controller.

[pt-emini]

The way to avoid it is for NT8 to not change state to Realtime until after the data is loaded.

That is the underlying design of NT7. If in NT8 you are now going to change state while the data is loading, then you will need to provide a new State control mechanism that we can use in a strategy to wait until the chart data load has completed before doing anything. Of course, as I demonstrated upthread, this whole situation applies equally to certain actions that might now inadvertently happen in OnMarketUpdate() unrelated to a strategy. This issue pervades the entire NT8 design.

NT7 strategies use the following line of code at the very top of OnBarUpdate()

if (Historical) return;

That is all we are required to do here in NT8, we are saying wait to do anything while the chart is being constructed, while data is initially being loaded, while the bars are populating in the chart. Once the data is done loading, then it is safe to issue alerts, play sounds, place orders, ect... There is a set of things we must never allow to happen in any strategy while data is being loaded. So we must have a control mechanism / flag to stop those things from ever happening prematurely. In NT7 that mechanism was the Historical boolean. According to your reply, we apparently have no corresponding mechanism in NT8.

So you can keep saying, that's the way it works now. And we as a user base will continue to respond, we understand this is your new design approach, and given this new design, what is the control mechanism we can use and rely on 100% of the time to work, that accomplishes the same task the Historical boolean in NT7 did ?

I think you have a significant design flaw in NT8. And as such, until we see an actual solution presented, the response "This behavior is expected" is not sufficient. It neither provides any solution to the problem, nor does it give any explanation at all that we as a user base can plan anything around in terms of moving our migration to NT8 forward.

[NinjaTrader_Brett]

PT,

Thanks for the providing the feedback. Let me take a moment to further clarify since you took the time to post your concern.

Your analysis of the NT7 underlying design is not accurate and takes the behavior out of context. However the end result (no matter the NT7 underlying design) is the same in your context so its not of value to clarify that behavior further.

For NT8 we on purpose changed this due to feedback from out client base that wanted their scripts to run live (not on historical).You however are pointing out you want and enjoyed the opposite with NT7. Seems we have clients that want both behaviors, which means we needed to provide the client some way to choose to satisfy both use cases.

I'm checking to see if anything could be done in NT8 for "choosing' now and if not I will add SFT and begin tracking for our next major release. I will get back shortly.

[pt-emini]

Brett - Thank you for the clarification in your response.

In NT7 having the Mutually Exclusive design of (Historical==true) and (Historical==false) made for a very simple way to control the startup of any strategy or indicator.

I take it from your response, in NT8 we are moving to a multi-state startup process (from a user's perspective). In this case opening up capabilities during startup to some users. I am more than happy to adapt to the new state design to facilitate the needs of everyone.

Maybe some sort of class attribute we can set / control from within the SetDefaults logic in OnStateChange() .... OldSchoolStartup = true ;

Thank you for your help.

[NinjaTrader_DaveI]

NT7 strategies use the following line of code at the very top of OnBarUpdate()

if (Historical) return;

we apparently have no corresponding mechanism in NT8.

In NT7 having the Mutually Exclusive design of (Historical==true) and (Historical==false) made for a very simple way to control the startup of any strategy or indicator.

Keep in mind that this same technique is still valid in NinjaTrader 8, with slightly different syntax. There are two States to note here:

• State.Historical (script has begun processing historical bars)
• State.Realtime (script is done processing historical bars, and has moved to realtime bars)

Thus, you can still place the code below in OnBarUpdate:

if (State == State.Historical) return;

or

if (State == State.Realtime)
{
do something...
}

Basically, all we've done here is to switch from using a bool to using an enum, so rather than checking a bool for true or false, we check the enum value currently assigned to State.

[pt-emini]

Dave - thank you for taking time to reply to my comments.

I think your response highlights the importance of this thread in the larger scheme of things. A lot of script writers new to NT8 are going to trip over this issue.

I genuinely think the design team needs to seriously consider 'falling back' to the NT7 two state mutually exclusive data load behavior as a default 'out of the box'. Making your new three stage data load behavior user selectable in some fashion, maybe as I had mentioned upthread as a simple user configurable class attribute controlled in OnStateChange(). The script writer will then need to intentionally enable the new behavior in order to use it. I would suspect 98% of average script writers will never need it for the typical indicators and strategies they will be developing. I personally cannot imagine how I would ever use it, but I have inadvertently found at least 4 different ways to trip over it.

[NinjaTrader_Brett]

pt-emini,

I have discussed with development and you can use the Playback.IsAvailable property to trigger your actions to not run while in FF on playback.

Its two staged: First set a bool if you are connected to playback or not.

	protected override void OnConnectionStatusUpdate(ConnectionStatusEventArgs connectionStatusUpdate)
		{
			PlaybackConnection = connectionStatusUpdate.Connection.Adapter as NinjaTrader.Adapter.PlaybackAdapter; 
			if (PlaybackConnection != null)
			{
				ConnectedToPlayback = true;
			}
		}

If you are connected to playback then check the Playback.IsAvailable property. If its false then you are in the middle of moving to a slider position.

if (ConnectedToPlayBack && NinjaTrader.Adapter.PlaybackAdapter.IsAvailable)
				{
					//Disable any action you dont want to trigger while FF. 	
				}

I'm evaluating doing this check for you in the SendMail, Alert, and Share adapters and logging a message however for now its a pick your poison scenario which I wanted to track feedback from customers on and will do that moving forward.

Again thanks for bringing this to our attention.