Invalid IsFirstTickOfBar After Historical Processing

[TheTradingMantis]

I have an indicator that uses a second data series AddDataSeries(Data.BarsPeriodType.Tick,1). On the chart I have the primary data series set at 5 minutes. Calculate is set to OnEachTick.

I have noticed that IsFirstTickOfBar in the Primary data series (5 minute) is true immediately after all historical data has loaded even if the last bar is only partial completed. After historical processing, I would expect IsFirstTickOfBar to only become true when the bar currently forming is completed.

I have also checked with the second data series commented out and I get the same results.

I have attached a reduced sample of code that illustrates the problem.

[NinjaTrader_PaulH]

Hello gregschr,

Thanks for your reply.

With your code you are seeing Historical states of IsFirstTickOfBar being true.

You can observe this by these statements that use State checks and CurrentBars[0] check:

if (BarsInProgress == 0)
			{
				if (IsFirstTickOfBar && State == State.Historical)
				{
					Print("CB: "+CurrentBars[0]+ "  Historical - First Tick Of Bar"+" at "+DateTime.Now.ToString("hh:mm:ss"));
				}
				
				
				if (IsFirstTickOfBar)
				{
					Print("CB: "+CurrentBars[0]+ "  No state checked - First Tick Of Bar"+" at "+DateTime.Now.ToString("hh:mm:ss"));
				}
				
				if (IsFirstTickOfBar && State == State.Realtime)
				{
					Print("CB: "+CurrentBars[0]+ "  Real time - First Tick Of Bar"+" at "+DateTime.Now.ToString("hh:mm:ss"));
				}				
			}

On 3 minute bars of CL 03-17, here is an example output where historical bars are processed followed by the real time data.:

CB: 1678 Historical - First Tick Of Bar at 07:09:18
CB: 1678 No state checked - First Tick Of Bar at 07:09:18
CB: 1679 Historical - First Tick Of Bar at 07:09:18
CB: 1679 No state checked - First Tick Of Bar at 07:09:18
CB: 1680 Historical - First Tick Of Bar at 07:09:18
CB: 1680 No state checked - First Tick Of Bar at 07:09:18
CB: 1681 Historical - First Tick Of Bar at 07:09:18
CB: 1681 No state checked - First Tick Of Bar at 07:09:18
CB: 1682 Historical - First Tick Of Bar at 07:09:18
CB: 1682 No state checked - First Tick Of Bar at 07:09:18
CB: 1683 Historical - First Tick Of Bar at 07:09:18
CB: 1683 No state checked - First Tick Of Bar at 07:09:18
CB: 1684 No state checked - First Tick Of Bar at 07:12:00
CB: 1684 Real time - First Tick Of Bar at 07:12:00
CB: 1685 No state checked - First Tick Of Bar at 07:15:00
CB: 1685 Real time - First Tick Of Bar at 07:15:00

The output shows that at the time the code was applied, IsFirstTickOfBar is true once per historical bar (this is expected behavior) and then becomes true on the open of the first realtime bar at 7:12 and again at 7:15.

[eDanny]

This was a big problem in NT7 also where the last historical partial bar fired a FirstTickOfBar and then on the first real time tick fired it again, mid bar.

[TheTradingMantis]

Here is another sample of code that clearly shows the problem that I am seeing and that eDanny mentions was also a problem in NT7.

This code sample adds up tick volume in an additional data series set at 1 Tick. The tick volume is reset on IsFirstTickOfBar in the primary data series. The printout compares the primary bar volume to the cumulative tick volume. This is on a 3 minute bar.

Here is the normal, expected output during Historical Processing:

CB0:1046 CB1:225419 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=27
CB0:1047 CB1:225419 - Historical Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar
CB0:1047 CB1:225420 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=1
CB0:1047 CB1:225421 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=2
CB0:1047 CB1:225422 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=3
CB0:1047 CB1:225423 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=4
CB0:1047 CB1:225424 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=5
CB0:1047 CB1:225425 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=8
CB0:1047 CB1:225426 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=11
CB0:1047 CB1:225427 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=12
CB0:1047 CB1:225428 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=13
CB0:1047 CB1:225429 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=14
CB0:1047 CB1:225430 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=15
CB0:1047 CB1:225431 - Historical Processing Bars[1] - Bars[0] Volume=16; Cum Tick Volume=16
CB0:1048 CB1:225431 - Historical Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar
CB0:1048 CB1:225432 - Historical Processing Bars[1] - Bars[0] Volume=83; Cum Tick Volume=1

Here is the output during the Transition from Historical to Realtime:

CB0:1056 CB1:225967 - Historical Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar
CB0:1056 CB1:225968 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=3
CB0:1056 CB1:225969 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=4
CB0:1056 CB1:225970 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=5
CB0:1056 CB1:225971 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=6
CB0:1056 CB1:225972 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=7
CB0:1056 CB1:225973 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=8
CB0:1056 CB1:225974 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=9
CB0:1056 CB1:225975 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=12
CB0:1056 CB1:225976 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=13
CB0:1056 CB1:225977 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=17
CB0:1056 CB1:225978 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=18
CB0:1056 CB1:225979 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=19
CB0:1056 CB1:225980 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=20
CB0:1056 CB1:225981 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=21
CB0:1056 CB1:225982 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=22
CB0:1056 CB1:225983 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=23
CB0:1056 CB1:225984 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=24
CB0:1056 CB1:225985 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=25
CB0:1056 CB1:225986 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=26
CB0:1056 CB1:225987 - Historical Processing Bars[1] - Bars[0] Volume=27; Cum Tick Volume=27
CB0:1057 CB1:225987 - Historical Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar
Current State = Transition at 11:53:00
Current State = Realtime at 11:53:00
CB0:1057 CB1:225988 - Realtime Processing Bars[1] - Bars[0] Volume=28; Cum Tick Volume=1
CB0:1057 CB1:225989 - Realtime Processing Bars[1] - Bars[0] Volume=29; Cum Tick Volume=2

And here is the normal, expected output during Realtime Processing:

CB0:1059 CB1:226030 - Realtime Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar
CB0:1059 CB1:226030 - Realtime Processing Bars[1] - Bars[0] Volume=1; Cum Tick Volume=1
CB0:1059 CB1:226031 - Realtime Processing Bars[1] - Bars[0] Volume=2; Cum Tick Volume=2
CB0:1059 CB1:226032 - Realtime Processing Bars[1] - Bars[0] Volume=3; Cum Tick Volume=3
CB0:1059 CB1:226033 - Realtime Processing Bars[1] - Bars[0] Volume=4; Cum Tick Volume=4
CB0:1059 CB1:226034 - Realtime Processing Bars[1] - Bars[0] Volume=5; Cum Tick Volume=5
CB0:1059 CB1:226035 - Realtime Processing Bars[1] - Bars[0] Volume=6; Cum Tick Volume=6
CB0:1059 CB1:226036 - Realtime Processing Bars[1] - Bars[0] Volume=7; Cum Tick Volume=7
CB0:1059 CB1:226037 - Realtime Processing Bars[1] - Bars[0] Volume=8; Cum Tick Volume=8

During the transition, you can see that IsFirstTickOfBar is called and the cumulative tick volume is reset, even though the bar forming during the transition has not closed. You can also see that the Primary Bar count is incremented from 1056 to 1057, even though it is still the same bar. As expected, the Primary Bar volume is not reset until the bar actually closes.

[NinjaTrader_PaulH]

Hello gregschr,

Thanks for your reply and additional code.

The example does illustrate your point, thank-you for the effort.

After discussion here I understand that this would be expected (IsFirstTickOfBar == true on first realtime tick)

If you like I can create a feature request that says IsFirstTickOfBar will not be true when transitioning from Historical to RealTime until/unless the bar type starts its first Bar in RealTime.

[TheTradingMantis]

Hi Paul,
That feature would be great and prevent having to store temporary values to work around the issue. Please create the request.

After discussion here I understand that this would be expected (IsFirstTickOfBar == true on first realtime tick)

Actually, IsFirstTickOfBar == true just before the transition to realtime. You can see that from the output I previously posted. If would be better if IsFirstTickOfBar == true on first realtime tick. At least then I could trap/correct the issue when State == Transition or Realtime and not have to store temporary values.

While not currently an issue with the indicator I am writing, the fact that one bar actually has two indexes (1056 and 1057 in the output below), is a bit concerning. This would imply that a chart with 100 bars would actually have 101 bar indexes.

I appreciate your help!
Greg

[NinjaTrader_PaulH]

Hello gregschr,

Thanks for your reply.

After further discussion here it has been determined that your code is showing the expected processing.
The bool is set on the first realtime tick triggering the state changes that follow in your example. As the example results show at the end/beginning of a bar the bar number is incremented to 1057.

If you run your code in the middle of a 5 minute bar you see that the bar number does not change. Here is an example where we have added DateTime.Now to the prints and run the indicator with just under 30 seconds remaining until the next bar:

CB0:469 CB1:155773 - Historical Processing Bars[1] - Bars[0] Volume=20865;  Cum Tick Volume=20864 time:  1/26/2017 8:29:32 AM
CB0:469 CB1:155774 - Historical Processing Bars[1] - Bars[0] Volume=20865;  Cum Tick Volume=20865 time:  1/26/2017 8:29:32 AM
CB0:470 CB1:155774 - Historical Processing Bars[0] - Reseting Cum Tick Volume On First Tick Of Bar. Time: 1/26/2017 8:29:32 AM
Current State = Transition at 08:29:32
Current State = Realtime at 08:29:32
CB0:470 CB1:155775 - Realtime Processing Bars[1]   - Bars[0] Volume=20866;  Cum Tick Volume=1 time: 1/26/2017 8:29:33 AM
CB0:470 CB1:155776 - Realtime Processing Bars[1]   - Bars[0] Volume=20867;  Cum Tick Volume=2 time: 1/26/2017 8:29:33 AM
CB0:470 CB1:155777 - Realtime Processing Bars[1]   - Bars[0] Volume=20868;  Cum Tick Volume=3 time: 1/26/2017 8:29:33 AM

There is not a good way to have real time processing delayed until the beginning of a bar without causing other significant issues and we will remain with the current processing.

Edit - forgot to add code wrappers

[TheTradingMantis]

Hi Paul,
Thanks for your further review and explanation.

I disagree about the bar number not changing. If you look at the Bars[0] volume on bar 470 in your example, it does not reset to zero on the first tick. Instead it continues to build on the volume from bar 469. Both bar 469 and 470 are the same bar on the chart.

I understand the need to not change the behavior of this bool when transitioning to realtime. Perhaps another bool could be added to the API that would only be set on the first tick of a bar regardless of the State?

Greg

[NinjaTrader_PatrickH]

Hello gregschr,

Thank you for your response.

We will submit a feature request for your suggestion.

On the Volume using the prior values, the CurrentBars[0] has not changed and therefore is still using the volume of the historical bar prior.

Please let me know if you have any questions.

[TheTradingMantis]

I agree the volume of the prior historical bar is still being used. The problem is the CurrentBars[0] count has incremented by 1 (from 469 to 470) at the beginning of realtime processing.

For further proof, use the drawing tool ruler from the first bar to the last bar on a chart. In your example below, the ruler would have shown 460 bars. BUT, the CurrentBars[0] count goes from 0 to 460 for a total of 461 bars.

What am I missing?

[NinjaTrader_PatrickH]

Hello gregschr,

Thank you for your patience.

I see what you are detailing on the the CurrentBars[0] changing but using the volume of the prior bar. I can't replicate this behavior on my end though. I used an ES 03-17 3 Minute chart.

Can you update to the latest release of NinjaTrader 8 (8.0.4.0)?
To upgrade NinjaTrader, please follow the steps below.

• First get your license key from NinjaTrader under Help> License Key then exit NinjaTrader
• Click on the link: http://ninjatrader.com/PlatformDirect
• Enter your license key and press Submit
• Select 'NinjaTrader 7' if you are on NinjaTrader 7
• Select 'NinjaTrader 8' if you are on the NinjaTrader 8
• Select 'Download'
• Critical: Before running the installer, ensure NinjaTrader is closed.

Once complete can you advise if you still see this behavior and if so please answer the following questions.
What instrument are you testing and what data provider are you using for the Tick and Minute data?

[TheTradingMantis]

Hi Patrick,
Sorry it took so long to get back to you. I was away from my charts for awhile.

I upgraded to 8.0.4.0 64-bit and I am still seeing the same issue.

This is happening on the 6E 03-17 contract and my data provider is CQG (not Continuum).

I've also attached the NinjaScript I am using to test this.

[NinjaTrader_PatrickH]

Hello gregschr,

Thank you for your patience.

Your code looks ahead one bar when in historical at line 81:

[I]Print("CB0:"+CurrentBars[0]+" CB1:"+CurrentBars[1]+ 
" - Historical Processing Bars[1] - Bars[0] Volume="
+BarsArray[0].GetVolume([/I][B]CurrentBars[0]+1[/B][I])+";  
Cum Tick Volume="+cummTickVolume);[/I]

Is this intended? I would think you would want the current bar's volume.