Bug: OnBarUpdate processing historical bars with State.Realtime

[Harry]

Here is another bug that I have discovered today.

NT 8.0.8.0.

OnBarUpdate processes historical bars, but the "State" is already set to "State.Realtime" when they are processed.


Reproducing the bug

To reproduce the error please proceed as follows:

(1) Open a single chart and add a test indicator that prints the "State" for each loop of OnBarUpdate().

(2) Minimize the chart (upper right corner of the chart window).

(3) Connect to a datafeed.

(4) Wait. You will now see that for the indicator on the minimized chart "State. Configure", "State.DataLoaded" and "State.Historical" are being processed. Actually each of these states is being processed twice for no specific reason, but this does not do any harm.

(5) Now open the minimized chart.

When the minimized chart is displayed, OnBarUpdate will now process all historical bars on the chart but treat them as realtime ticks with the "State" shown as "State.Realtime". This is the bug and causes further havoc.

(6) Now refresh the chart via F5.

This time "State.Configure", "State.DataLoaded". and "State.Historical" will be called only once. OnBarUpdate will now process the same historical bars of the chart again and show them with "State.Historical".


Implications

When coding multi-timeframe indicators for NinjaTrader it is important to differentiate between historical bars and real-time bars, as they do not follow the same logic. For historical bars any event becomes final with the current bar, while for real-time bars it only becomes final with the first tick of the next bar (event driven logic).

When NinjaTrader confuses historical with real-time bars, this will cause many multi-timeframe indicators to crash. Therefore it is important that the state is correctly set when a minimized chart is opened.


Further problems

There are other problems that I have discovered with minimized charts, such as

-> BarsArray[0] not being populated when the minimized chart is opened.
-> PlotSeries holding no values when being accessed.
-> 'Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Object reference not set to an instance of an object.'
-> 'A direct X error has occurred while rendering the chart: HRESULT: [0x88990001], Module: [SharpDX.Direct2D1], ApiCode: [D2DERR_WRONG_STATE/WrongState]'

Those errors do often occur, when a minimized chart is reopened when connected to a data feed. But I cannot reproduce the behavior or attribute it to specific bugs. I will come back on these when there is further evidence.

However, the false realtime state is 100% reproducible and easy to test.

[sledge]

NT 8.0.8.0.

Further problems

There are other problems that I have discovered with minimized charts, such as


-> 'Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Object reference not set to an instance of an object.'
-> 'A direct X error has occurred while rendering the chart: HRESULT: [0x88990001], Module: [SharpDX.Direct2D1], ApiCode: [D2DERR_WRONG_STATE/WrongState]'

Those errors do often occur, when a minimized chart is reopened when connected to a data feed. But I cannot reproduce the behavior or attribute it to specific bugs. I will come back on these when there is further evidence.

.

Search on 0x88990001, you will see about 7 other previous posts about this. Maybe something in those posts can help point you to something in common.

[Harry]

For the time being I would like to focus on the problem that historical bars are being processed by OnBarUpdate() in State.RealTime, when a minimized chart is opened.

This is easily reproducible and should be fixed in any case.

[Harry]

Minimized charts disappearing

I am continuing my enquiry.

I have now found out why State.Configure, State.DataLoaded and State.Historical are being called two, three, four or five times.

When I minimize the chart it sometimes disappears from the Windows taskbar. However, it is still running in the background, but cannot be accesed any more. These hidden charts (minimized but no more accessible) trigger the multiple calls of the states.

There is a possibility to reaccess the "lost" charts. I have a main workspace and a secondary workspace "Untitled" which is empty. When I toggle to the second empty workspace and then back to the primary workspace the "lost" charts become visible again in the task bar.

Actually, I have been wondering how I lost all these charts. I had first made the assumption that I had erroneously closed them instead of minimizing them. But this was not the case. They somehow were no more attached to the taskbar, but had never been closed.

This explains why the different states were called multiples times. However, this is a different bug just interfering with my tests not linked to the primary problem.

[Harry]

Here comes the next bug related to minimized charts.

After several hours of testing, I found that the sessioniterator for the secondary bars is not working when the minimized chart is reopened after connection. I have not yet tested whether this problem also occurs under other conditions.

This was the test setup:.

An indicator that is applied to a 15 min chart. The indicator loads a secondary bar series via AddDataSeries in State.Configure.

AddDataSeries(BarsPeriodType.Tick, 50);

The two session iterators are both set in State.DataLoaded:

sessionIterator0 = new SessionIterator(BarsArray[0]);
sessionIterator1 = new SessionIterator(BarsArray[1]);

I think so far it is pretty standard. Then I have tested whether the session iterator works as expected.
The test for the primary bars was done via

sessionIterator0.CalculateTradingDay(Times[0][0], timeBased0);
sessionDateTmp0 = sessionIterator0.ActualTradingDayExchange;
DateTime sessionBegin = sessionIterator0.ActualSessionBegin;
DateTime sessionEnd = sessionIterator0.ActualSessionEnd;

All trading day, session begin and session end were returned correctly.

The test for the secondary bars was done accordingly:

sessionIterator1.CalculateTradingDay(Times[1][0], timeBased1);
sessionDateTmp1 = sessionIterator1.ActualTradingDayExchange;
DateTime sessionBegin = sessionIterator1.ActualSessionBegin;
DateTime sessionEnd = sessionIterator1.ActualSessionEnd;

However, whatever was the input time Times[1][0], the ActualTradingDayExchange, session begin and session end always referred to the last secondary bar loaded via the indicator.

Example

Times[1][0] = 27.07.2017 03:56:00
ActualTradingDayExchange = 04.08.2017 00:00:00
sessionBegin = 04.08.2017 00:00:00
sessionEnd = 04.08.2017 23:00:00

This is the third bug that relates to minimized charts.

[Harry]

Summary

I know that this is complex and that you do not like it. So let me try to summarize:

(1) When a chart is minimized when not connected to a data feed, and then reopened after connecting to the datafeed, any indicator will see OnBarUpdate() processing all historical bars in State.Realtime.

(2) If that chart holds an indicator which loads a secondary bar series, the session iterator of that secondary bar series will fail - at the same time when the historical bars are processed in State.Realtime.

The two failures - false processing logic and session iterator for secondary bars not working - have caused my indicators to produce a major mess resulting in various error messages including the OnRender errors that I have posted below. So far I could not prove any failure of OnRender when the primary two errors did not occur.

(3) Minimized charts are often disappearing from the task bar and remain active in the background. This could be detected via a multiple execution of the various states detected within OnStateChange(). Toggling to another workspace and back to the primary workspace made those hidden charts reappear in th Windows task bar.

These are my findings for today. Definitely three bugs. I am happy to share further details, if needed.

[sledge]

Here comes the next bug related to minimized charts.

After several hours of testing, I found that the sessioniterator for the secondary bars is not working when the minimized chart is reopened after connection. I have not yet tested whether this problem also occurs under other conditions.

This was the test setup:.

An indicator that is applied to a 15 min chart. The indicator loads a secondary bar series via AddDataSeries in State.Configure.

AddDataSeries(BarsPeriodType.Tick, 50);

The two session iterators are both set in State.DataLoaded:

sessionIterator0 = new SessionIterator(BarsArray[0]);
sessionIterator1 = new SessionIterator(BarsArray[1]);

I think so far it is pretty standard. Then I have tested whether the session iterator works as expected.
The test for the primary bars was done via

sessionIterator0.CalculateTradingDay(Times[0][0], timeBased0);
sessionDateTmp0 = sessionIterator0.ActualTradingDayExchange;
DateTime sessionBegin = sessionIterator0.ActualSessionBegin;
DateTime sessionEnd = sessionIterator0.ActualSessionEnd;

All trading day, session begin and session end were returned correctly.

The test for the secondary bars was done accordingly:

sessionIterator1.CalculateTradingDay(Times[1][0], timeBased1);
sessionDateTmp1 = sessionIterator1.ActualTradingDayExchange;
DateTime sessionBegin = sessionIterator1.ActualSessionBegin;
DateTime sessionEnd = sessionIterator1.ActualSessionEnd;

However, whatever was the input time Times[1][0], the ActualTradingDayExchange, session begin and session end always referred to the last secondary bar loaded via the indicator.

Example



This is the third bug that relates to minimized charts.

Harry, any chance related to my secondary series bug in the indicator? I get random times stamps in replay.

I'm guessing this is replay you are using?

http://ninjatrader.com/support/forum/showthread.php?t=100377

NT was not able to reproduce.

[Harry]

Harry, any chance related to my secondary series bug in the indicator? I get random times stamps in replay.

I'm guessing this is replay you are using?

http://ninjatrader.com/support/forum/showthread.php?t=100377

NT was not able to reproduce.

No I have neither been using market replay (connection) nor tick replay (DataSeries settings).

The bugs I have detected occurend when I was connected to Kinetick Realtime data feed.

[NinjaTrader_ChelseaB]

Hello Harry,

I'm happy to test.

Do you have IsSuspendedWhileInactive set to true or false?

To confirm, when testing I will leave the workspace open, minimize the chart, disconnect and then reconnect, view the output window, then restore the window, and again view the output window, is this correct?

[Harry]

Hello Harry,

I'm happy to test.

Do you have IsSuspendedWhileInactive set to true or false?

To confirm, when testing I will leave the workspace open, minimize the chart, disconnect and then reconnect, view the output window, then restore the window, and again view the output window, is this correct?

Disconnect first, then refresh the chart via F5, then minimize the chart. Now clear the output window, reconnect, wait 30 seconds to make sure that data was loaded and OnStateChange() has been executed. Then reopen the minimized chart. When the minimized chart is opened, OnBarUpdate() will process all historical bars as real-time ticks.

IsSuspendedWhileInactive is set to true.

Test indicator

I am about to build a small test indicator to reproduce the issue. So far I have observed

-> easy to reproduce: historical bars being processed as real-time bars
-> difficult to reproduce: session iterator not working

In my complex indicator I have called the session iterator both in OnBarUpdate() and the custom OnRender() section. It seems that there is an interference between the two calls which cpntributes to the problem. Also I will test whether GetNextSession() works different than CalculateTradingDay().

It will probably take until tomorrow to build the test indicator. In its current state the session iterator still works for the secondary bars.

[Harry]

Here is the test indicator.

It behaves in a similar fashion as the indicator that I used for detecting the problems.

The session iterator problem is indeed caused by using the same session iterator in OnBarUpdate() and OnRender(). There is probably a limitation that this cannot be done.

I will try to use two different session iterators for the same secondary bar series loaded by the indicator, one for OnBarUpdate() and another one for OnRender().

[Harry]

Session Iterator problem

This is the session iterator Pproblem:

I had defined two session iterators

-> sessionIterator0 for the primary bars
-> sessionIterator1 for the secondary bars

As long as I use the session iterators in OnBarUpdate() only, they work as expected. However, in my test indicator I also called sessionIterator1.CalculateTradingDay() in OnRender(). As a consequence sessionIterator1 returned false results in OnBarUpdate().

To confirm the findings, I have then called sessionIterator0 in OnRender(). This time sessionIterator0 did not work in OnBarUpdate(), while sessionIterator1 now returned correct trading dates.

From this test it is clear that it should be avoided to use the same sessionIterator in both OnBarUpdate() and OnRender().

As a solution I have now defined a third session iterator, sessionIteratorR which is being used in OnRender(). Now there is no more interference between the different calls of the session iterators and they work as expected.

Could you please explain this behavior?

[soulfx]

Luckily i came across this as i am testing something at the moment.

Indeed why in an indicator processing does State == State.Realtime for historical bars.
I am NOT even connected to a data feed. I am printing out state in various areas to verify this.
I do NOT expect to see State.Realtime when not connected to data feed - ie idle charting mode just with historical.

Is there a global cbi field that indicates when the datafeed is connected over and above the State change?

I NEED to make a distinction for optimising / efficiency if i am connected to the market data or not.

Has something happened in the state management - i just checked the Help on State change.

Please advise - with all Harrys postings and detail why has no one responded to his questions?

thanks

[Harry]

Hello soulfx,

NinjaTrader usually distinguishes well between historical and real-time bars. I am only referring to a specific case

- connected to a data feed
- and reopening a minimized chart (or tab)

For minimized charts OnStateChange() is triggered, but OnBarUpdate() is not triggered when connecting. OnBarUpdate () is finally called when the minimized chart is reopened. And only in this specific case NT8 seems to qualify historical bars as real-time bars.

[soulfx]

Thanks Harry, but my example is not a specific case : general with printed output over and over. This needs input from ninja - i was not even connected to a datafeed - why would i get state = Realtime.