Bug: Daily data not added correctly via AddDataSeries()

[Harry]

Today the situation is more tricky.

OnBarUpdate processes the indicator correctly for ES 03-17, but fails to do so for CL 03-17.

Please find below two charts for CL 03-17. The disconnected chart is flawed. The connected chart is correct. The problem is reproducible with CL.

The output window shows that the daily bar for January 30 is not being processed by OnBarUpdate(), when the datafeed is disconnected.

However, there is no problem for ES 03-17 today. It is correctly displayed in both cases, connected and disconnected. The question here, is what makes up for the difference?

[Harry]

Just for documenting the problem.

Below I have attached copies of the output files for ES 03-17 and for CL 03-17.

You will notice that the output file shows that the daily bar for January 30 was processed for ES 03-17, but it was not processed for CL 03-17.

Therefore the the chart for ES 03-17 is correct, while the chart for CL 03-17 is incorrect.

I have simply toggled between the two instruments for producing the output files. I have not changed any of the chart settings. In one case OnBarUpdate() works correctly, in the other case the daily bar is omitted. I have no idea what causes this behavior.

[JerryWar]

Harry, If you haven't I think you need to compare the Session Templates for both symbols and see if they different anywhere.

[Harry]

Harry, If you haven't I think you need to compare the Session Templates for both symbols and see if they different anywhere.

Jerry, the issue is not related to session templates. All session templates are set to a start time of 6:00 PM EST. The problem affects all instruments. Temporarily, ES 03-17 was processed correctly for unknown reasons.

NinjaTrader does not process the last daily bar from the historical data base, when any indicator is set to Calculate == Calculate.OnBarClose and when the datafeed is disconnected. This is a problem of the bar processing engine of NinjaTrader.

Interesting enough, as soon as you connect the problem goes away.

[Harry]

Serious bug: GetBar not working correctly

Guys these are serious bugs, and so far this has been ignored. I am not asking for a ticket for a future improvement. This is not working!

I have now coded a very simple indicator to show that GetBar is not working properly. The first problem was that it retrieved future data on holidays, also affecting the Pivots indicator that ships with NinjaTrader in a way that it shows false results.

This is worse. I have applied a simple indicator to a 60 minute chart for ES 03-17 with a session template 24/5. And this is the outcome. GetBar always calls the same bar for Tuesday and Wednesday without a valid reason.


This is the code of the indicator in OnBarUpdate(): It is as simple as it can be!

lastBarTimeStamp = sessionIterator.GetTradingDay(Times[0][0]);
if (lastBarTimeStamp != currentDate)
{
    priorDayHigh     = BarsArray[1].GetHigh(BarsArray[1].GetBar(currentDate));
    priorDayLow     = BarsArray[1].GetLow(BarsArray[1].GetBar(currentDate));
    currentDate     = lastBarTimeStamp;
    Print(Times[0][0] + "    " + currentDate + "   "  +  priorDayHigh + "   " + priorDayLow);
}
PriorHigh[0] = priorDayHigh;
PriorLow[0] = priorDayLow;

Just from the code you would expect that the indicator identifies the daybreak for the trading day and then calls the prior daily bar and displays prior high and low.

In fact this is what it does:

For currentDate = Monday it displays Thursday's high and low (false)
For currentDate = Tuesday it displays Monday's high and low (correct)
For currentDate = Wednesday it displays Monday's high and low (false)
For currentDate = Thursday it displays Tuesday's high and low (false)
For currentDate = Friday it displays Wednesday's high and low (false)

It only works on Tuesdays!

I am aware that the session template 24/5 is an unusual template. I would not use it personally, but quite a few traders do use it. There is no reason that GetBar would not work on such a template.

Please find attached a chart showing the problem and the simple indicator that has been used for demonstration. All you need to do is to apply the indicator to a virgin chart with the session template 24/5.

[Harry]

Even worse: GetBar retrieving 3 identical values for three different days

When I use GetBar() with daily bars on the session template 24/7, things get even worse. GetBar will now produce identical values for Monday, Tuesday and Wednesday.

I believe that GetBar() should currently not be used at all, as it is not clear what it is actually doing.

[Harry]

Secondary daily bars cannot be correctly accessed on a historical chart

Again today with any indicator that loads a secondary bar series of daily bars:

-> when setting the indicator to Calculate.OnBarClose, yesterday's daily bar is not being processed by OnBarUpdate() but omitted, observed for all instruments and indicators

No problems with the same indicator settings

-> when connected
-> when disconnected and indicator set to Calculate.OnEachTick

Please confirm this bug.

It is currently impossible with NinjaTrader to access daily bars via a secondary bar series on a historical chart. Both methods, standard bar processing and access via GetBar() are flawed and do not work!

Due to the two bugs I have not found a satisfactory solution.

[Harry]

Yesterday's daily bar not being processed

I have made further tests.

From the two problems the GetBar() problem is less urgent. I have found a workaround for the CME holiday problem. The workaround can also be used to debug the Pivots indicator that ships with NinjaTrader.

Please find attached another indicator, where I have

-> applied GetBar() to the current time instead of the current date
-> and added a loop to avoid catching forward dates for yesterday's daily bar

However, even with the modified approach I cannot fix the problem with the daily, which is not being processed.

Whether I access the bars in BarsInProgress == 1 or whether I attempt to get hold of the bar via GetBar(), yesterday's daily bar cannot be accessed when disconnected from the datafeed.

I am aware that today's daily bar cannot be accessed in Calculate.OnBarClose prior to the close of the trading day (event driven). But I am talking about yesterday's bar which is not being processed.

Please find attached modified indicator to fix the CME holiday problem and two screenshots showing that the holiday problem is fixed, but that I cannot display yesterday´s daily high and low today.

[Harry]

Here is another document. It is now Monday 3:00 PM Eastern Time. The last daily bar that has been processed was on Thursday 5:00 PM Eastern Time (see green box). No daily bar was processed for Friday 5:00 PM at the market close (see red box). I understand that no daily bar is supposed to be processed for today prior to the close, when the indicator runs in Calculate.OnBarClose.

Problem only shows when disconnected, mostly in Calculate.OnBarClose, but also in Calculate.OnEachTick.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience and all the additional information.

I could not recreate the multiple days of incorrect High and Low and could only recreate the holiday item you detailed and I confirmed. However, I am still looking into this further with Product Management and I will let you know what we find.

For the last bar of the Daily series, who do you connect to as your data source for the data? And to confirm, this is occurring in real-time and not when reloading historical data?

I look forward to your response.

[Harry]

Patrick,

thank you for your answer.

This is one of the more malicious bugs. I think it should have absolute priority, because NinjaTrader's bar processing engine does not perform as expected.

I have this well documented. I have coded a most simple custom indicator that demonstrates that the bar processing does not work. I can reproduce it every day.

These are the findings:

(1) When disconnected from data feed, NinjaTrader does not process the secondary daily bars added via an indicator as expected: Yesterday's daily bar is being omitted.

(2) The result is that false values are displayed on the chart, as soon as the data feed is disconnected and the chart refreshed via F5.

(3) I have written BarInProgress to the output window and I have clearly documented that yesterday's daily bar has not been processed by OnBarUpdate().

It is a heavy bug and it is important to remove it, because it has further implications.

Please find below chart attached showing that I have it again, as every day!

[Harry]

Hello Harry,

For the last bar of the Daily series, who do you connect to as your data source for the data? And to confirm, this is occurring in real-time and not when reloading historical data?

I look forward to your response.

The data provider is Kinetick Realtime Data.

The bug .. yesterday's bar not being processed .. only occurs when disconnected from the data feed and chart was refreshed via F5.

[Harry]

Again same observation on Sunday evening ...

-> connected: yesterday's close is correctly displayed, OnBarUpdate() processes the daily bar from Friday

-> disconnected: yesterday's close is not displayed, OnBarUpdate() does not process the daily bar from Friday

[Harry]

Hello PatrickH,

I have brought up the subject that OnBarUpdate() does not properly process yesterday's daily bar about three weeks ago.

The issue is important and easily reproducible.

It shows that OnBarUpdate() is not working as expected.

All indicators and strategies that add a secondary series of daily bars are affected!

Do you need any further help from my side to reproduce the issue and confirm that it is a bug?

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience.

This is actually expected behavior. The daily bar for today is only built when connected to the provider (in this case Kinetick). When disconnected, the last daily bar is the previous day.
When using Calculate.OnEachTick the last daily bar in the repository cache is being used.

This is opposed to how NinjaTrader 7 handled a disconnected feed, in which it would only show the last bar close regardless of using CalculateOnBarClose = false.

For this case there is no means around what you are seeing and it would be expected that when disconnected and using Calculate.OnBarClose in NinjaTrader 8 that the current day bar is not used and thus the prior day bar is the current bar.

Please let me know if you have any questions.