Bug: Daily data not added correctly via AddDataSeries()

[Harry]

When an indicator loads daily data via AddDataSeries(BarsPeriodType.Day, 1), and in case that there is no daily data for today, BarsArray[1]GetBar(Times[0][0]) loads future data from the historical data base.

This does not make sense, as the future is not yet known.

As a consequence the Pivots indicator calculates pivot values from future data for CME traded instruments after a day without settlement.

In order to show the problem, I have coded a PriorDayOHLC indicator that uses daily data, and I have already examined the pivot values displayed by the pivots indicator that ships with NinjaTrader 8.

Monday, January 16 was a holiday (President's Day) with an early close, and there is no data in the daily data base, as expected. If I try to load daily data from Monday - which in fact is not existant - BarsArray[1].GetBars(Times[0][1]) will load daily data from Tuesday and already display it for the first price bar on Tuesday morning.

In fact the prior day OHLC and the floor pivots for Tuesday are calculated from Tuesday's daily data, although the close is only known in the evening. I will show the problem more in detail in the following posts and include a test indicator.

[Harry]

Pivots indicator not calculating correctly when using daily data

I am referring to the case where no daily data is available for calculating the pivot. The example I chose is the day after "President's Day", which is Tuesday, January 17. As we know there is no daily data for Monday, as Monday is not a separate trading day.

In this case Tuesday's pivots should be calculated from the last daily data available (Friday's data) and not from future data (Tuesday's data) which is not yet known.

Unfortunately, the NinjaTrader pivots indicator does not show the prior day high, low and close, therefore I need to perform calculations. The main pivot shown on the chart attached below is 2263.50. If you compare to daily data you will find

pivot PP calculated from Friday's HLC = (2273.50 + 2262.75 + 2272.50) / 3 = 2269,58
pivotPP calculated from Tuesday's HLC = (2270.50 + 2257.25 + 2262.75) / 3 == 2263.50

This shows that Tuesday's pivots are indeed calculated from future data (including Tuesday's close and low, which was not known in the morning). This is not as expected.

When a secondary bar series is added and data is retrieved, only such data should be used that was available at the time, when it is called. The use of future data does not make sense and would not be possible in real-time trading.

[Harry]

Test indicator shows that GetBar() does not insert daily data correctly

To make it easier to detect, I have modified the PriorDayOHLC indicator and let it load daily data instead of intraday data.

From the chart attached you can easily make out that

- the prior day low for Tuesday' is taken from Tuesday afternoon
- the prior day close for Tuesday is Tuesday's settlement price

The high is Monday's high, because the high of the double day session published after Tuesday's close was already made on Monday.

The problem here is that

priorDayOpen    = BarsArray[1].GetOpen(BarsArray[1].GetBar(Times[0][1]));
priorDayHigh     = BarsArray[1].GetHigh(BarsArray[1].GetBar(Times[0][1]));
priorDayLow     = BarsArray[1].GetLow(BarsArray[1].GetBar(Times[0][1]));
priorDayClose     = BarsArray[1].GetClose(BarsArray[1].GetBar(Times[0][1]));

collects daily data from the future instead of the past, when there is no daily data available for the day referenced by Times[0][1].

[JerryWar]

Harry,
I don't use daily data but I had an issue with Tues/Wed with intraday data for Oil, because the Holiday Template For MLK day had an issue with it. I deleted the template entry for that day and everything was good to go.

[Harry]

Harry,
I don't use daily data but I had an issue with Tues/Wed with intraday data for Oil, because the Holiday Template For MLK day had an issue with it. I deleted the template entry for that day and everything was good to go.

Thank you for your answer, but this is issue is not related to the holiday template.

This is about loading future data instead of current data on the historical section of a chart.

In my opinion it is a serious bug that needs to be removed.

[NinjaTrader_AlanP]

Hello Harry,

We are currently looking into this and will let you know when we have more information.

Thank you for your patience.

[Harry]

OnBarUpdate Bug - secondary bar series not properly processed

This is a second bug, not related to the first bug. I am talking about this case:

-> a daily bar series is added via AddDataSeries(BarsPeriodType.Day, 1)
-> the indicator set to Calculate = CalcuateOnBarClose
-> the chart is disconnected from the datafeed

In this case yesterday's daily bar is not added by OnBarUpdate() but simply omitted. Yesterday's daily bar is correctly processed, when you

-> set the indicator to Calculate = Calculate.OnEachTick (even when disconnected)
-> or when you connect to the datafeed (even when Calculate == Calculate.OnBarClose)

The bug only occurs with the combination Calculate == Calculate.OnBarClose when disconnected from the data feed. The bug is 100% reproducible.

In order to demonstrate the problem, I have coded a little indicator that will write the BarsInProgress information to the NinjaTrader output window. When you add it to a chart (do not connect datafeed) you will easily find that yesterday's daily bars is not being processed.

Here is a excerpt from the output file (primary bars 60 min, when connected):

BIP = 0 26.01.2017 23:00:00
BIP = 1 26.01.2017 23:00:00
BIP = 0 27.01.2017 01:00:00
BIP = 0 27.01.2017 02:00:00
.
.
BIP = 0 27.01.2017 22:00:00
BIP = 0 27.01.2017 23:00:00
BIP = 1 27.01.2017 23:00:00
BIP = 0 30.01.2017 01:00:00

And here comes the same output, when OnBarUpdate() is running not connected:

BIP = 0 26.01.2017 23:00:00
BIP = 1 26.01.2017 23:00:00
BIP = 0 27.01.2017 01:00:00
BIP = 0 27.01.2017 02:00:00
.
.
BIP = 0 27.01.2017 22:00:00
BIP = 0 27.01.2017 23:00:00 DAILY BAR IS NOT PROCESSED HERE!
BIP = 0 30.01.2017 01:00:00

Below you will find a simple test indicator attached and two charts.

(1) First add indicator to chart and connect to a datafeed -> chart is displayed correctly.

(2) Second disconnect from data feed and refresh chart via F5 -> chart is now false as the last daily bar has not been processed.

[Harry]

The problem is specific to adding daily bars

I have now tested a similar indicator by adding 1440 Minute bars as a secondary bar series instead of daily bars.

In this case NinjaTrader 8 behaves as expected. OnBarUpdate() processes yesterday's 1440 minute bar correctly in all cases that is

-> disconnected or connected
-> Calculate == Calculate.OnBarClose or Calculate == Calculate.OnEachTick

Only yesterday's daily bar is omitted in the case described.

[JerryWar]

I haven't looked at your script but technically with Ninja Considering tick bars the last bar is not the last bar until the first tick of the Next Bar. This may be why your seeing a difference between OnBarClose and OnEachtick. i.e. the difference between when it rolls to the next bar.

[Harry]

I haven't looked at your script but technically with Ninja Considering tick bars the last bar is not the last bar until the first tick of the Next Bar. This may be why your seeing a difference between OnBarClose and OnEachtick. i.e. the difference between when it rolls to the next bar.

I understand the event driven logic of NinjaTrader and have coded a bunch of multi-timeframe indicators dealing with the difference between historical & real-time data and the difference between OnBarClose and OnEachTick.

In this case a daily bars is not being processed, although a new tick event has been triggered. This is simply a bug.

Comparison between daily bars and 1440-min bars added confirms that the daily bars are not being processed correcty.

[Harry]

No answer regarding the second bug: OnBarUpdate() not processing all daily bars added via AddDataSeries()?

Could you confirm the problem please?

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience.

Apologizes for the delay. I will follow up here shortly on this matter.

[Harry]

Hello Harry,

Thank you for your patience.

Apologizes for the delay. I will follow up here shortly on this matter.

Thank you for your answer. Please let me know, if you need additional information.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your response.

On the first item for the holiday not having a daily bar and pulling from the future, I am working with development.

For the second item though, could you provide the steps to reproduce? I am not sure I fully understand the steps taken.
Are we setting up the test while disconnected? Or disconnecting after loading the test on the chart and then reconnecting?

I look forward to your response.

[Harry]

First step of the test:

(1) For setting up the test please open a simple chart such as a 60 min chart with a lookback period of a few days.

(2) Also open an output window, as the indicator will print the BarsInProgress information to the output window.

(3) Please connect to the datafeed to make sure that the daily data for today has been loaded.

(4) Now apply the indicator with setting Calculate = Calculate.OnBarClose (default setting) to the chart.

While you are connected everything is fine and the indicator prints the BarInProgress information as expected. The indicator value is correctly displayed on the chart.

Second step of the test:

(1) Clean the output window

(2) Disconnect the data feed.

(3) Refresh the chart via F5.

Now you will notice that the indicator displays a false value for the prior close. This is due to the fact that the last daily bar has not been processed by OnBarUpdate() as needed. This will be visible on the output window, as the BarsInProgress information for the daily bar of today is missing.

The indicator was only built for monitoring the bar processing, OnBarUpdate() does not process the last daily bar as needed, when disconnected with Calculate == Calculate.OnBarClose.

Hello Harry,

Thank you for your response.

On the first item for the holiday not having a daily bar and pulling from the future, I am working with development.

For the second item though, could you provide the steps to reproduce? I am not sure I fully understand the steps taken.
Are we setting up the test while disconnected? Or disconnecting after loading the test on the chart and then reconnecting?

I look forward to your response.