Synchronicity of daily bars within intraday charts

[Harry]

This refers to NinjaTrader 6.5.1000.14
Mode connected to data feed or not connected.
Observerd on two identical linked intraday charts.

2 identical linked charts, each of them with several indicators that collect daily bars information, for example daily, weekly and monthly pivots.

When toggling between instruments, one of the connected charts will load pivots correctly, the other one will display false pivots. Result: Two charts with a 100% identical setting (same template, same data series) display different indicator values.

The false pivots will be corrected, when the indicator menu is opened, and "Apply" is selected without any changes.

When charts are simplified, either by reducing lookback period, by increasing interval, or by reducing the number of indicators, the problem does not occur.

So far the facts. Now an interpretation:

Under similar conditions I have already observed that the loading of daily bars can lag and is not fully synchronized with the loading of intraday data. I have traced this in another case by printing execution times to the output window, and this showed that the loading of the dailybars was only completed after the intraday data had already been processed via onbar update (thus not taking into account the last daily bars).

Any ideas or remedies for this one?

[NinjaTrader_VinceB]

Hi Harry,

Can you send us the exact chart set up and all other relevant informmation so we can test this on our end. Also let us know who you are connected to. Please provide as much detailed information as possible so we can set up the exactly the way you have.

Please send it to support at ninjatrader dot com and refer to this post. Someone will test for you on Monday or Tuesday when markets are open. This coming Monday it's Martin Luther King Day in the US and markets close therefore earlier. This means that we will be understaffed this day so it is possible we will not get to it then. You will receive a response no later than Tuesday.

[Harry]

Indicators compete to access daily data at the same time

Was able to produce a simplified test setup:

No connection. Two identical standard intraday charts (5 min, 45 days). One and the same indicator that uses daily bars (via Data.Bars.GetBars...) applied to each of the charts. Then I linked the charts via the colour code on the upper right corner. Finally I toggled to another instrument on the left chart. Result:

The left chart displayed the indicator values correctly.
The right chart (connected to left) displayed false values.

To further explore the problem, I created a clone of the indicator with a different name, but otherwise identical. Then I added identical Print instructions to both of the clones, each of the Print instructions including the name of the clone, allowing me to identify, which of the two clones had printed the message.

This is what I observed:

Again, the left chart is the one I used to toggle the instrument, the right chart is the connected one.

Left 21:56:10.83 OnBarUpdate first run
Left 21:56:10.83 Loading Daily Bars Start
Right 21:56:11.08 OnBarUpdate first run
Right 21:56:11.08 Loading Daily Bars Start
Right 21:56:11.08 No daily data loaded
Right 21:56:11.08 OnBarUpdate continued
Right 21:56:11.34 OnBarUpdate last run
Left 21:56:11.42 Loading daily data completed successfully
Left 21:56:11.42 Start accessing daily bars
Left 21:56:11.42 OnBarUpdate continued
Left 21:56:11.73 OnBarUpdate last run

Conclusion:

The indicator of the leading chart of the connected charts starts OnBarUpdate and loads daily data via Data.Bars.GetBars... While it is doing this, the indicator of the second charts starts OnBarUpdate and also tries to access daily data via Data.Bars.GetBars... However, because the first chart is still loading daily data, the access is denied and no daily data can be loaded to the connected chart. The indicator from the second chart therefore continues grinding its way through OnBarUpdate without accessing the daily data.

It is difficult to observe this behaviour with the pivot indicators for a simple reason: The pivots are coded in a way that they automatically calculate values from intraday data, in case that there is no daily data available. Often the pivots calculated from intraday data show identical values, so the bug cannot easily be observed.

The indicator I used to detect the problem uses daily data to perform complex calculations and for reasons of speed and memory usage will not fall back to 'CalcFromIntradayData'.

This would also confirm one of the potential reasons of the numerous crashes I have experienced with NinjaTrader. Two indicators are competing to access the same data at the same time.


Quick Test:

By the way there is a quick test to show the problem. Take two larger identical charts with some identical indicators - but only indicators that do not catch daily data via Data.Bars.GetBars... - and connect them. Then toggle to another instrument. The first chart that will be displayed is the one you changed, the connected chart will be loaded second.

Now add standard pivots to both of the charts. You will find that the order is inversed. The connected chart will display first and the leading chart comes second.

What about loading data sequentially and not open a box within a box?

[Harry]

Parallel data access causes problem

I have now also observed the problem on a single chart with several indicators accessing DailyBars, although it is much more frequent on connected charts.

Problem is that NinjaTrader accesses daily data in parallel, and that this is probably not supported by the database.

So if you are lucky, daily data is loaded. If you are unlucky it is not. I have now recoded my indicators not to display anything, in case that "DailyBars" was selected and daily data not loaded. I prefer nothing to false data. Usually, when trying several times, the chart will finally load (when loading them one by one).

I have sent further information to support, as requested below.

[NinjaTrader_Bertrand]

Harry, thanks for the indepth analysis and for the files you sent to support - just replied, let's take it from there, thanks.

[Harry]

When reloading chart via F5, daily data indicators will correctly display

Hi Bertrand,

thanks for your response. I agree that this is not a major problem for NT 6.5. In case that one of the charts is not loaded correctly, it can be reloaded via F5, and the indicators will then display correctly - with daily data loaded.

But I would appreciate, if the issue could be adressed for NT 7.

[NinjaTrader_Bertrand]

Sure Harry, I want to definitely make sure this works as expected for you in NT7 - if you have a working 7 version of the indicator (like noted in our mail conversation) please let me know what you find.

Thanks for your support

[Harry]

Linked intraday charts with multiple indicators that load daily data

Just an update:

(1) I stopped using linked charts at all -> NT crashes reduced by over 90%.
(2) Even when loading a single intraday chart with multiple indicators using daily data, some of the indicators may not display or display false values. I have therefore recoded these indicators to display a warning message on the chart in case daily data has not been loaded. Then refresh via F5, and everything will be displayed correctly.
(3) For NT 6.5. I can get along with this - although the peformance is not convincing. NT7 has a greatly enhanced architecture, so this type of error is unlikely to reproduce - and so far I could not observe it.

[Harry]

Just an example

Here is an example, how I solved the problem:

All pivot indicators - also SessionPivotsV91 - will check, whether daily data was correctly loaded to the chart. If data is not loaded, the chart displays a message as can be seen on the first screenshot below. The second screenshot, which was obtained after hitting F5, shows the correct picture.

Dangerous to rely on a tool that sometimes works and somtimes does not! The Fibonacci cluster lines are slightly out of place (as they are only calculated from intraday data) and the values calculated by the weekly pivots indicator are missing (Prior Week Low and S-Mid) on the left chart.

I hope that NT7 will be more reliable.

[NinjaTrader_Bertrand]

Harry, thanks for the update - have you tested the same scenario already on the 7 beta you've got access to?

Thanks

[Harry]

Hi Bertrand,

when testing this with NT7 I had various other problems, such as Red Crosses appearing and Interactive Brokers data not being loaded as supposed, double session break lines, etc. I also had a few problems with loading daily data, but so far not reproducible.

Will continue testing when NT 7.0.0.7 is out.

[NinjaTrader_Bertrand]

Great, sounds good Harry, please keep me posted and thanks for the support.

[Harry]

Some rules that keep me out of trouble when using daily data on intraday charts

The following observations only apply to workspaces that include several charts with indicators that rely on DailyBars, such as pivots. The problems do not occur, when such indicators are not being used.

(1) NT 6.5. When I have indicators on my charts (Pivots) that use DailyBars, I cannot reconnect to Interactive Brokers, without shutting down NinjaTrader first.

False: Start Ninjatrader -> Connect -> Disconnect -> Connect -> Several charts are corrupted (blank) and no new charts can be opened (always)
Correct: Start Ninjatrader -> Connect -> Disconnect -< Shutdown Ninjatrader -> Start Ninjatrader -> Connect -> No problem (mostly)

(2) If I start NinjaTrader, change instruments and then connect, often I will find corrupted charts

False: Start NinjaTrader -> Change one of the chart instruments -> Connect -> One or more charts are corrupted and no new charts can be opened (sometimes).
Correct: Start NinjaTrader -> Connect -> Change one of the chart instruments -> No problem

(3) Linked charts cannot be changed, when connected

False: Start NinjaTrader -> Connect -> Change instrument of linked chart -> One ore more charts are corrupted and no new charts can be opened. (often)
Correct: Start NinjaTrader -> Connect -> Unlink all charts -> Change instruments on charts one by one -> No problem.

To cope with this behaviour, I use three rules that keep me out of trouble most of the time:

Rule 1: Never use linked charts.
Rule 2: Never change instruments before connecting.
Rule 3: Once connected and disconnected, shut down NinjaTrader before reconnecting.

Some of these problems seem to spill over into NT 7. For example the current version of the NT pivots indicator will not reliably catch the DailyBars and needs refreshing via F5, but this is a different thread.

[NinjaTrader_Bertrand]

Thanks Harry, for NT 7 beta 8, did you try already with the updated Pivots indicator Dierk posted on the beta forums?

[Harry]

As higler and Fat Canary confirmed the F5 problem still persists. In fact the indicator sometimes catches daily data and sometimes does not.

Also the latest pivots do not work with sessions that start prior to midnight. Just have a look at the thread NT7 -> Pivots not working

Thanks Harry, for NT 7 beta 8, did you try already with the updated Pivots indicator Dierk posted on the beta forums?