Tick Bar and Intra Tick Alignment

[BraisedInBlue]

I am currently working on an indicator that is centered around Tick bars (1000 tick) that also leverages 1 Tick granularity for its processing.

For some time I have struggled to get the indicator results to be indentical between historical processing and real-time processing.

One of the struggles I am running into is data alignment between the primary tick bar (1000 ticks) and the intra ticks. Upon debugging, I ran into a oddity that I cannot explain.

Upon printing the data on each tick, I noticed a couple issues.

The primary bars index is initialized to -1 for the beginning of the historical data loaded. Not sure what the issue is with that, but that doesnt seem reasonable. The real issue is that, the primary bar's index becomes 0 on tick 983, and not tick 999 or 1000. The primary bar index does correct itself when it increments to 1, on tick 1999 (making 1998 final tick. Primary bar index moves to 2 on 2997. Technically this is an off by one error, as that puts 998 (0 index based) index counts between the primary bar being incremented. This behavior is seen in both historical and realtime processing.

The secondary (1 Tick) bar series is initialized to 1 on the first tick, which makes this alignment issue even more odd.

My hope is that there is some thing I am missing that is causing this issue. Now, I do not believe I need to use TickReplay. I am using 1 Tick bars to perform the data aggregation and analysis, but with tick bars, there is a path dependence that has be respected in order for any historical data testing to even be able to be compared to realtime analysis. In order to perform validation and verification of analysis results, I have to be able to get historical results to match realtime results, and while I can get very close, a tick being miscounted under a different bar can accumulate error and change the outcome of the analytical tool.

Please advise NT team!

[BraisedInBlue]

As I was trying to build a minimally complete example for NT's testing, by adding some more prints the results of the output changed slightly.

primary bar index still is initialized to -1, but now rolls over to 0 at 1-tick 999 as expected, however, it rolls over to 1 @ tick 1995, and then 2 @ 2977, 3 @ 3993, gets corrected by rollover 4 @ 4999, off by one on 5 @ 5998, fixed at 6 @ 6999 off again at 7 @ 7996 etc.

The secondary issue I have noticed is that the final tick isnt processed until after the primary data series roll over, so even if there wasn't the roll over issue, it would always be missing the data from the last tick of the bar.

EDIT: Secondary issue isn't an issue, the latest data for the last tick on a new primary bar can be retrieved within the primary data series OnBarUpdate().

[NinjaTrader_Kate]

Hello BraisedInBlue,

Thank you for your posts.

A few notes on things that are likely affecting your results:

In a multi-series script, CurrentBars starting value will be -1 until all series have processed the first bar. To ensure you have satisfied this requirement on all your Bars objects, it is recommend you start your OnBarUpdate() method with CurrentBars checks.

Historical bars are processed according to their timestamps with the primary bars first, followed by the secondary, which is NOT guaranteed to be the same sequence that these events occurred in real-time. If your development requires ticks to process in the same sequence historically as well as in real-time, you will need to enable Tick Replay (utilizes more PC resources).

This page of our help guide goes into what can occur when multiple bars of the primary series come in quick succession and may share the same timestamps:

https://ninjatrader.com/support/helpGuides/nt8/index.html?multi-time_frame__instruments.htm#WorkingWithMultiTimeFrameObjects

That being said, I'd like to take a look at a simplified sample of what you're attempting so I can see if you're running into any known limitations. Can you provide a simplified example?

Thanks in advance; I look forward to assisting you further.

[BraisedInBlue]

I can, would you like me to email it?

[BraisedInBlue]

Heres the test indicator. Must be attached to a 1k tick data series. My example output is too large to upload even compressed.

[BraisedInBlue]

My primary issue is the segmenting of 1k ticks into bars. It isn't consistent. I dont necessarily care about the order of the 1k ticks within the bar as long as the L1 data is correct for the tick, but I need ticks 0-999 to be what builds bar 0, ticks 1000-1999 be what builds bar 1 and so on and so forth.

[NinjaTrader_Kate]

Hello BraisedInBlue,

Thank you for your replies.

We've taken a look and it appears you're basically running into some known issues with trying to keep single tick series in line with a primary series.

I've chatted with our team here and we're working on an example, but it may take a few days to complete - it's more complicated than one would think.

I'll let you know once we've got something for you, however.

Thanks in advance for your patience; I look forward to assisting you further.

[BraisedInBlue]

So I have attempted to solve this a couple different ways, but if you have something that would work, thank you very much!

[hedgeplay]

Hi BraisedInBlue,

No idea if this will really help you ...

... but if you have not tried it yet...

1) Simply right click on the chart, and at the bottom of the pop-up menu click "Properties"

2) About 10% down in the next pop-up un-check or check "Equidistant Bar Spacing"

3) Right-click the chart and reload "All Historical Data"

Scroll left a then right .. drag the time bar to resize the width of the chart..

Any better?

HedgePlay

[BraisedInBlue]

So the issue isn't in the display/rendering of the bars, but the accumulation of ticks to build to overall 1000 tick bar.

It is related to rendering only in that you are reliant on when the primary data series rolls over (new bars) to be able to populate the latest rendered plot value (and any Series<> based management of data). This likely falls under the advanced development category of issues.

[hedgeplay]

So the issue isn't in the display/rendering of the bars, but the accumulation of ticks to build to overall 1000 tick bar.

It is related to rendering only in that you are reliant on when the primary data series rolls over (new bars) to be able to populate the latest rendered plot value (and any Series<> based management of data). This likely falls under the advanced development category of issues.

"The real issue is that, the primary bar's index becomes 0 on tick 983, and not tick 999 or 1000. .... "

I re-read your post.

I concur with your analysis. Beyond what you included above, if I recall correctly, when setting length of period to 1 day each added DataSeries may start with a slightly different tick count and time (in micoseconds) PRIOR to the open of the new session and the exact number of ticks included prior to session for each DataSeries might change by each reload or by reload of historical on different calendar market days.

I have not read and did not ask but assumed this behavior was designed to ~"fill each DataSeries tick pipeline just in advance of the session start so we ensure have stable reliable execution during the chaotic microseconds right at the session open."


I just designed around it to get my goals accomplished but thank you for raising the issue (if nothing else for clarification and discussion) in a well articulated manor.

Kate and team are on it. The topic is in good hands.

I will watch for the replies.

HedgePlay

[NinjaTrader_Kate]

Hello BraisedInBlue,

Thank you for your patience.

This indicator from our publicly available User App Share would give an example of building series with historical tick data that may be helpful to you. There's some comments in the code that explain how it handles this. I'd start with looking at how this handles ticks and assigns them to the bars:

https://ninjatraderecosystem.com/user-app-share-download/advance-decline/

Thanks in advance; I look forward to assisting you further.

The NinjaTrader Ecosystem website is for educational and informational purposes only and should not be considered a solicitation to buy or sell a futures contract or make any other type of investment decision. The add-ons listed on this website are not to be considered a recommendation and it is the reader's responsibility to evaluate any product, service, or company. NinjaTrader Ecosystem LLC is not responsible for the accuracy or content of any product, service or company linked to on this website.

[BraisedInBlue]

While this sort of can work, I need my 1000 tick bars to each contain 1000 ticks, period. This is a bug, not a limitation.