Bug: OnBarUpdate processing historical bars with State.Realtime

[Harry]

Thanks Harry, but my example is not a specific case : general with printed output over and over. This needs input from ninja - i was not even connected to a datafeed - why would i get state = Realtime.

Did you document this behavior?

[Harry]

Synchronization errors with minimized charts

I have now modified the indicators to work with historical bars which carry the state "realtime" and also used separate session iterators for OnBarUpdate() and OnRender(). While this fixes some of the errors, there are a lot of other problems that I cannot fix. I can only guess what is happening.

These are the steps one by one:

(1) Add an indicator to a chart, while being disconnected.

Everything works as expected.

(2) Minimize the chart.

With the introduction of chart tabs, lots of NinjaTrader 8 users now have minimized charts. For NinjaTrader 7 users this was rare. The charts on all non-active charts behave in the same way as minimized charts.

(3) Connect to a data feed.

For the minimized chart OnStateChange() will be called. I have checked that it runs through the states State.Configure, State.DataLoaded and State.Historical. However, OnBarUpdate() is not called.

(4) Reopen the minimized chart.

This is now where lots of synchronization errors happen:

-> OnRender() is being executed and terminated before OnBarUpdate() has been completed (I have checked this via the condition Values[0].IsValidDataPointAt(lastBarIndex) which often returns "false" showing that OnRender cannot access the plot values).

-> The completion of OnBarUpdate() takes longer for multi-timeframe indicators as the secondary bars need to be loaded.

-> Depending on the indicator all or some plots will not be shown.

-> Also there is the problem that all the historical bars are handled by OnBarUpdate() as realtime bars.

-> Only the first real-time tick which is detected after the chart has been opened the indicators will display correctly again.

Charts on other tabs and minimized charts are currently a mess.

Would somebody please confirm my findings?

[NinjaTrader_ChelseaB]

Hello Harry,

I've tested for the behavior and I am able to reproduce the behavior where the State is Realtime while historical data is being processed when the chart is restored after disconnecting and reconnecting.

I have reported this behavior and once I have a tracking ID I will post this in this thread.

However, I was not able to reproduce an issue with the SessionIterator.

https://www.screencast.com/t/F8GOxINlnSud

May I confirm I have taken the correct steps to test?

[Harry]

Hello Chelsea,

Thank you for your quick response.

I have tested your little indicator and it is easy to replicate both errors.

In OnBarUpdate() it definitely calculates a false session for the first bar. The input time is June 5 2017, 00:15, for which the ActualSessionBegin is calculated as August 7 2017 00:00. This is the bug!

I have attached a screenshot below from the output file generated by your indicator.

[Harry]

I have checked your video again. You performed the test in a slightly different fashion. Please try again:


(1) Add an indicator to a chart, while being disconnected.

You were connected when you added the indicator. Please disconnect prior to starting the test.

-> add the indicator when disconnected
-> alternatively refresh the chart via F5 when being disconnected

(2) Minimize the chart, while being disconnected.

-> must be done when disconnected

(3) Connect to a data feed.

Please wait for 15-30 seconds after connecting. It is important that data loading was completed and that OnStateChange() has been executed prior to reopening the minimized chart.

(4) Reopen the minimized chart.

You should now see the false session begin after "OBU".

[NinjaTrader_ChelseaB]

Hello Harry,

What was the value ActualSessionBegin returned without minimizing the chart when it displayed Historical?

Did this change?

Is the issue with minimizing the chart or is the issue with ActualSessionBegin and unrelated to minimizing the chart?

(I ask because in the video I have created, the value appears to remain the same before and after the chart is minimized)

[Harry]

Hello Harry,

What was the value ActualSessionBegin returned without minimizing the chart when it displayed Historical?

Did this change?

Is the issue with minimizing the chart or is the issue with ActualSessionBegin and unrelated to minimizing the chart?

(I ask because in the video I have created, the value appears to remain the same before and after the chart is minimized)

A customer of mine opens a workspace. Then connects to a datafeed. In his workspaces he has different charts that he likes to access with tabs. Tabs are basically the same as the minimized charts.

When he opens these charts the indicators are not working, because the historical bars are treated as realtime bars, and because the session iterator returns false values.

Please do the test the way I have described it, and the session iterator bug will appear. Usually the indicators work as expected and there is no problem with the session iterators. The session iterator problem comes up, when the chart is minimized when disconnected.

[Harry]

Chelsea,

The false session begin only appears after the chart has been minimized (when disconnected) and after reconnecting and then reopening the chart.

Could you reproduce the session iterator problem with the test indicator that you have coded?

The problem shows well with your indicator.

[NinjaTrader_ChelseaB]

Hello Harry,

Below is a link to a new video.

https://www.screencast.com/t/KPJHJuEecdi

Have I not understood the steps correctly?

[Harry]

Hello Harry,

Below is a link to a new video.

https://www.screencast.com/t/KPJHJuEecdi

Have I not understood the steps correctly?

Your video does not even show the result for the session iterator in OnBarUpdate(). That would be the line marked with "OBU". This is the one that is not correctly calculated.

The video only shows the results for the session iterator in OnRender(), which are marked with "OR".

You need to look at the false calculation done by the session iterator in OnBarUpdate().

[NinjaTrader_ChelseaB]

Hello Harry,

You mentioned that the value is wrong after OnBarUpdate so I was showing the most recent value from OnRender which comes after OnBarUpdate.

However, I have created a new video to scroll up and show the exact same value printed in OnBarUpdate instead of after it.

https://www.screencast.com/t/npgDO06NCgI

[Harry]

Hello Chelsea,

Here is my video. Basically I have done the same thing as you, but in the end I have scrolled up the output window and you will notice that it reads

05.06.2017 00:15:00 | State: Realtime
OR | sessionIterator ActualSessionBegin: 08.08.2017 00:00:00

which shows that the session iterator calculates August 8 as the session begin for the first bar. The first bar has a time stamp of June 5, 00:15. This shows that the session iterator does not work here.

Here is the link to the video:

http://www.screencast.com/t/oQVzvVjSlq

The flawed result appears in the end.

The session iterator will calculate correctly under normal conditions, but it does not work when the chart is minimized and when it is used in OnRender() as well.

[NinjaTrader_ChelseaB]

Hello Harry,

The session begin time does not appear to change in your video.
It is showing 08.08.2017 00:00:00 before the chart is minimized when the State is showing Historical.

Are you certain you are wanting to report that the session iterator is showing the wrong time because the chart was minimized?

[Harry]

Hello Harry,

You mentioned that the value is wrong after OnBarUpdate so I was showing the most recent value from OnRender which comes after OnBarUpdate.

However, I have created a new video to scroll up and show the exact same value printed in OnBarUpdate instead of after it.

https://www.screencast.com/t/npgDO06NCgI

Hi Chelsea,

In the code for your test indicator, you calculate the session begin for the bar #1 by applying the session iterator.

The time stamp for this bar is August 2, 2017 6:02 PM.
The ActualSessionBegin calculated from that bar is August 8 6:00 PM.

This is totally false and is clearly shown on your video.

When you calculate the ActualSessionBegin for the bar #1 with the time stamp August 2, 2017 6:02 PM, the correct session begin would be August 2, 2017 6:00 PM and not August 8, 2017, 6:00PM.

Although you have called the session iterator in OnBarUpdate(), it takes the value from the session iterator when called in OnRender(), which it should not.

[Harry]

Here is your code:

Print(string.Format("{0} | State: {1}", Time[0], State));
if (CurrentBar == 1)
{
   sessionIterator.CalculateTradingDay(Time[0], true);
   Print(string.Format("OBU | sessionIterator ActualSessionBegin: {0}", sessionIterator.ActualSessionBegin));
}

You are calculating the trading day for Time[0], which is the time for CurrentBar == 1. However, the session iterator returns the session begin for the last bar on the chart. So it does not do what it is expected to do.

Actually, the session iterator returns the same session for all bars on the chart. This is dead wrong.

The problem here is that it is being called in parallel from OBU and OR and there is some sort of interference which leads to false results.

This interference only is there when charts are minimized. Minimizing charts causes a general disorder as OnStateChange, OnRender and OnBarUpdate are not being called in the proper order.

Even with workarounds for the real-time trap and the session iterator trap my indicators do not work correctly when the chart was minimized because some of the plots have not been loaded before OnRender() is being executed.

It is a general mess and I want to report it.