Playback -> Calculate.OnEachTick -> OnBarUpdate called before transition to Realtime

[digibob]

Hi. Issue with an early call to OnBarUpdate for a series (BarsInProgress=1) in Playback on transition from historical to realtime.

Primary series is Futures daily.

State.SetDefaults >
Calculate = Calculate.OnEachTick;

State.Configure >
AddDataSeries(null, new BarsPeriod { BarsPeriodType = BarsPeriodType.Second, Value = 86400 });
AddDataSeries(null, Data.BarsPeriodType.Second, 1);

OnBarUpdate >
if (BarsInProgress == 1 && IsFirstTickOfBar)

Print(Times[1][0]+" | "+Times[2][0]+" | State="+State);

Output:

30-Jan-20 17:00:00 | 30-Jan-20 17:00:00 | State=Historical
31-Jan-20 17:00:00 | 31-Jan-20 17:00:00 | State=Historical
03-Feb-20 17:00:00 | 03-Feb-20 17:00:00 | State=Historical
04-Feb-20 17:00:00 | 03-Feb-20 23:59:57 | State=Historical
Enabling NinjaScript strategy 'Test Series/192771176' : On starting a real-time strategy ...
05-Feb-20 17:00:00 | 04-Feb-20 18:00:01 | State=Realtime
06-Feb-20 17:00:00 | 05-Feb-20 18:00:01 | State=Realtime
07-Feb-20 17:00:00 | 06-Feb-20 18:00:01 | State=Realtime

The call in bold should take place AFTER "Enabling NinjaScript strategy" with State=Realtime. The reason it is there in the first place is because Calculate is set to OnEachTick. When the state is not yet Realtime, how can OnEachTick be already in effect.

I have separate implementations for historical and realtime - said call is wrongly routed in my code to the historical segment because it comes in too early, and then it is absent from the first session in realtime because of that, making double the mess.

Should I expect the same behavior on live run or is this only happening in Playback?

Thank you.

[digibob]

Just to clarify when I say absent from the first session in realtime, it's because of using IsFirstTickOfBar, so once the first call arrives before the transition to realtime, the next calls in realtime are ignored.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your note.

It would need to be tested if this happens in real-time data. Do you have a test script I could try along with this bar type you are adding?

I look forward to hearing from you.

[digibob]

Hi Chris. Thanks for your reply. The test file I created is deleted but the script is just that as given in the first post, to go in OnStateChange and OnBarUpdate. No need for anything else.

[NinjaTrader_ChrisL]

Hi digibobl, thanks for your reply.

I made a test script that does the same thing but I am not getting this problem. Are you sure to check for CurrentBars?

if(CurrentBars[1] < 1 || CurrentBars[2] < 1)
return;

[digibob]

The strategy runs few days historical before transition to realtime so both CurrentBars are past the beginning.

I can confirm the same is happening on live run, which is good in a sense, that playback is consistent with live run, but I don't get it. Basically what happens is, Calculate mode is changing from OnBarClose to OnEachTick before the state is changing from historical to realtime.

There is a simple workaround to deal with it though. In the script in my first post you simply compare the two timestamps. If the state is historical and the timestamps don't match you know something is "wrong". Meaning the state is already realtime.

It just takes a while to figure out what's going on when you don't expect this behavior.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your reply.

If you have a script that can reproduce the problem I would be happy to test it out. I was unable to get any such behavior out of the script that I made.

Kind regards.

[digibob]

Hi Chris. Please see the script below. I run that with ES 03-20 daily as the primary series.

I should note that the workaround I suggested earlier is not perfect as it relies on there being a bar on the 1-second time frame.
But that's beside the point.

protected override void OnStateChange()
{[INDENT]if (State == State.SetDefaults)
{[/INDENT][INDENT=2]Description = @"Enter the description for your new custom Strategy here.";
Name = "Test17";
Calculate = Calculate.OnEachTick;[/INDENT][INDENT]}
else if (State == State.Configure)
{[/INDENT][INDENT=2]AddDataSeries(null, new BarsPeriod { BarsPeriodType = BarsPeriodType.Second, Value = 86400 });
AddDataSeries(null, Data.BarsPeriodType.Second, 1);[/INDENT][INDENT]}
else if (State == State.Transition)
{[/INDENT][INDENT=2]Print("Transition");[/INDENT][INDENT]}
else if (State == State.Realtime)
{[/INDENT][INDENT=2]Print("Realtime");[/INDENT][INDENT]}[/INDENT]
 }

protected override void OnBarUpdate()
{[INDENT]if (CurrentBars[0] < 0 || CurrentBars[1] < 0 || CurrentBars[2] < 0)[/INDENT][INDENT=2]return;

[/INDENT][INDENT]if (BarsInProgress == 1 && IsFirstTickOfBar)[/INDENT][INDENT=2]Print(Times[1][0]+" | "+Times[2][0]+" | State="+State);[/INDENT]
 }

Below is the output on my machine. The timestamp in the line in bold (19-Feb-20 09:08:17) is when I enabled the strategy. To my understanding this line should print after transition, after the strategy is enabled, not before. It's called when the first realtime tick comes in, is it not?

12-Feb-20 17:00:00 | 12-Feb-20 17:00:00 | State=Historical
13-Feb-20 17:00:00 | 13-Feb-20 17:00:00 | State=Historical
14-Feb-20 17:00:00 | 14-Feb-20 17:00:00 | State=Historical
17-Feb-20 13:00:00 | 17-Feb-20 12:59:59 | State=Historical
18-Feb-20 17:00:00 | 18-Feb-20 16:59:59 | State=Historical
19-Feb-20 17:00:00 | 19-Feb-20 09:08:17 | State=Historical
Transition
Enabling NinjaScript strategy 'Test17/192771173' : On starting a real-time strategy ...
Realtime

[NinjaTrader_ChrisL]

Hi digibob, thanks for your reply

I don't see how you are getting a print for the latest second on the third 1 second series when you have this defined in the condition:

if (BarsInProgress == 1 && IsFirstTickOfBar)

By running this, I get the second the secondary bar closes (the 1 day series.) e.g.

2/18/2020 3:00:00 PM | 2/17/2020 10:59:59 AM | State=Historical
2/19/2020 3:00:00 PM | 2/18/2020 2:59:59 PM | State=Historical
2/20/2020 3:00:00 PM | 2/19/2020 3:00:00 PM | State=Historical

Please let me know if you have any further questions.

[digibob]

Hi Chris. I'm getting really confused here

I do understand what you say, the 1 day series gets called before the 1 second series does, at this point the time on the 1 day series is :00:00 while on the 1 second series it's still the prior second :59:59. Ok.

But I don't get neither your print nor mine.

I don't get lines in my print where both times are 00:00 and both are the same date.

I do get lines in my print where one is 00:00 and the other 59:59 but with the same date. This to my understanding is as it should be.

I don't get the different dates in your print. Once the 1 day series closes it holds today's date, while the 1 second series holds today's date all throughout the session, so when the 1 day series closes, the dates should be the same. What am I missing?

There are two different issues here now. The above is my asking in attempt to understand the mechanism. Separate from that is a technical issue - why are we getting different prints? Is it some definition on my machine?

[NinjaTrader_ChrisL]

Hi digibob, thanks for your patience.

I'm looking over this with my supervisor now. I am indeed seeing prints from the 1-second series that are not making sense. Ill post here when I have more info.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your patience.

I re-ran the script over some more sample data. The reason you see the second series print out 1 day before the daily series is the new day bar opens up after the session and that has the next days timestamp (remember that a daily bars only have one time stamp at the end of the day)

While running the attached script I got expected time stamps in historical mode running on a 10 minute chart:

2/19/2020 3:00:00 PM | 2/18/2020 2:59:59 PM | State=Historical
2/20/2020 3:00:00 PM | 2/19/2020 3:00:00 PM | State=Historical
2/21/2020 3:00:00 PM | 2/20/2020 3:00:00 PM | State=Historical

Try what I did and run the script through Market Replay for a few days and see how it acts in real time mode.

[digibob]

Hi Chris. I appreciate your going over this and thanks for the script. Running the indicator on the chart in Market Replay I get this:

14-Feb-20 17:00:00 | 14-Feb-20 17:00:00 | State=Historical
17-Feb-20 13:00:00 | 17-Feb-20 12:59:59 | State=Historical
18-Feb-20 17:00:00 | 18-Feb-20 16:59:59 | State=Historical
19-Feb-20 17:00:00 | 19-Feb-20 17:00:00 | State=Historical
Transition
Realtime
20-Feb-20 17:00:00 | 20-Feb-20 17:00:00 | State=Realtime // on EOD realtime

Always same dates, which is in line with what I expect and different from your print.

Could the issue be that I use end-of-bar timestamps and you use beginning-of bar? I remember seeing the setting for this somewhere though I looked now in Options menu and didn't find it.

I also tried running the indicator through the strategy but strangely there was no output from the indicator.

So this is about understanding the timestamps, and the example indicator uses Calculate.OnBarClose. There is still the issue of the early tick that comes in on transition when using Calculate.OnEachTick. I'm at a loss on both.

Have a good week.

[digibob]

End-of-bar timestamp setting is on import data. So scrap that Then I don't know what the issue is.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your reply.

I meant to set the Indicator default to OnEachTick, I have been testing with OnEachTick, sorry about that. Please try removing and re-downloading this playback data, this can be done from Tools>Historical Data>Market Replay section>Right click the data and remove. Also clear out your cache of historical data:

To clear out your cache:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Kind regards,

-ChrisL