Historical data manager not downloading all tick data

[Harry]

NinjaTrader 8.0.5.2 with Kinetick Realtime data.

I am downloading all historical data via the Historical data manager. Every day after the market close I select two instrument lists and update all tick, minute and daily data. The lookback period for downloading data is preset to one week. I have been doing this with NinjaTrader 7 for years and it worked.

Today I discovered that with NinjaTrader 8 for all instruments historical tick data was not correctly downloaded for the trading days during the time when the US and Europe had different daylight savings schedules. Examples:

YM: March 14, March 16, March 17, March 20, March 21 missing
NQ: March 14, March 16, March 17, March 20, March 21, March 22 missing
HO: March 16, March 17, March 19, March 21, March 22 missing

Analysis

As I am only downloading data via the Historical Data Manager, and as I had not opened any charts for YM, NQ and HO recently, I know that the missing data can only be attributed to the malfunction of the download manager.

The problem only occured when US and Europe had different daylight savings schedules. During this period the start of the trading day is 11:00 PM local time. And the download manager precisely failed to download the data between 11:00 PM and 12:00 AM. This happened for all futures contracts, although the dates are not 100% identical.

Only tick data was affected. Minute data was correctly downloaded.


Updating tick data for a futures contract

Let us assume that I had updated data for a futures contract yesterday prior to the market close. Market close was at 10:00 PM local time and the markets reopened at 11:00 PM (when the US observes summer time and Europe winter time). Today I am updating tick data and set the lookback period to 1 week.

Now NinjaTrader downloads all data of today (calendar day) but fails to download the missing hour from yesterday (11:00 PM to 12:00 AM) leaving a hole in my historical data.

These data holes are a very annoying feature of NinjaTrader, as they are never filled unless you open a chart and reload all historical data. And the historical data manager is definitely not working as expected.

[Harry]

Wanted to update the missing historical data. First it worked, but now it looks like Kinetick does not allow me to reload historical chart data any more.

Kinetick data throttling prevents me from updating the faulty data.

It is 6:15 AM EST.

How shall a user cope with such a tool?


Edit: It was the NinjaTrader API that got locked up when I opened a chart while the Historical Data Manager was downloading data. No data loading possible anymore. I had to shut down and restart NinjaTrader.

[Harry]

Reproducible Problem - Historical Data Manager Bug

Now here it gets interesting.

For the contract 6J 06-17 I have tick data for March 21 in my historical database from 0:00 until 22:00. The data from 23:00 to 24:00 after the technical break is missing.

Now I am running a test:

(1) I am updating tick data (only tick data) for March 21 (start date and end date) via the Historical Data Manager.

Result: The data is not updated. After the update there is no tick data for the period March 21 from 23:00 to 24:00. This is confirmed by opening a chart and by editing the historical data base. Same result for March 22. No data updated.

(2) I am now updating the tick data by opening a tick chart and reloading all historical chart data.

This time the tick data is written to the data base and the charts display as expected.


Historical Data Manager Not Working

This cleary shows that the historical data manager is not working. The data for the selected calendar day (or trading day) is not updated.

This further explains why all my charts for futures are having holes.

[NinjaTrader_RileyS]

Harry, thanks for your posts!

I'll need a little bit more information so that I may test this further on my end.

What timezone is your PC set to?

What regional setting is your PC set to? You may find this by clicking on the start menu and typing "Region" and pressing enter.

Do you receive an error on screen? Are there errors on the Log tab of the Control Center? If so, what do these errors report?

I look forward to being of further assistance.

[Harry]

My PC is set to CET (Central European Time). Usually we are 6 hours ahead of EST and 8 hours ahead of Denver. This means that the electronic open for index futures is at midnight CET (which would be 6 PM EST or 4 PM MST).

However, during the last two weeks the difference was reduced to 5 hours, as the US switched to summer time ahead of Europe. Europe has also switched last Sunday, and the difference is now back to normal.

My PC is running with German settings.

There are no error messages whatsoever. Trace and log files are clean. The historical data manager simply does not download the data. Only tick data is affected. Minute data was complete.

If you wish you may connect to my PC, and I can show you the data sitting in the historical data base not being completed.

[Harry]

Historical data manager is really bad!

I have now checked my FOREX charts.

The FOREX session starts 1 hour earlier at 5:00 PM EST compared to index futures, currency futures and oils and metals.

For FOREX all data is flawed, as the historical data manager did not download the data for the end of the trading day.

Often the hour from 11:00 PM to 12:00 AM local time is missing, This hour belongs to the old calendar day, but the new trading day and is simply omitted. I can scrap all my tick data for FOREX instruements, because it is incomplete.

Only way our of this to delete and reload all data.

The historical data manager cannot be used at all!

[chrishaiden66]

how to download all historiacal data with multiple selection

[NinjaTrader_RileyS]

Harry, thank you for your post back.

I'm currently testing all the items you brought to the table and may take time because of specific settings.

In regards to downloading historical data by multiple instruments at a time, this is currently not possible to do--would you like me to send a feature request to our development team for further consideration on this item?

[Harry]

Harry, thank you for your post back.

I'm currently testing all the items you brought to the table and may take time because of specific settings.

In regards to downloading historical data by multiple instruments at a time, this is currently not possible to do--would you like me to send a feature request to our development team for further consideration on this item?

Of course it is possible to download historical data by multiple instruments at a time. This feature was already available for NinjaTrader 7 and it is also available for NinjaTrader 8.

In fact NinjaTrader 8 is designed to download historical data for an entire instrument list. I have therefore created instrument lists for the specific purpose of downloading historical data.

There is not much testing needed to show that the historical data manager is not working. It is pretty simple to show. I invite you to have a look at my machine and my data, and you will understand it in a minute.

[NinjaTrader_RileyS]

Harry, you're absolutely correct! I had a misunderstanding, my mistake!

Let's setup a remote support session. You may launch TeamViewer by navigating to the Control Center > Help > Remote Support.

Next, please send an email to platformsupport[at]ninjatrader[dot]com. In the body of the email please be sure write ATTN Riley, include this link: http://ninjatrader.com/support/forum...d.php?p=501723, and lastly, provide the numbers that the remote support application provides.

After launching TeamViewer, please be sure to leave it open and running until I am able to connect.

[Harry]

I am sorry for my late reply.

Somehow I have missed your message, as I have not been alerted via e-mail to your message.

Please let me know, when you are ready for a remote support session.


Connection problem

Also there is another problem. Kinetick has not been stable today. When there is a temporary disconnect while I am downloading data via the historical data manager, the data manager freezes and no more data can be downloaded.

Downloading data is only possible after a restart of NinjaTrader.

This is a second problem not related to the first one.

[NinjaTrader_RileyS]

Harry, thanks for posting back.

I'll be available today until 5:30 PM EST.

To setup TeamViewer, you may launch TeamViewer by navigating to the Control Center > Help > Remote Support.

Next, please send an email to platformsupport[at]ninjatrader[dot]com. In the body of the email please be sure write ATTN Riley, include this link: http://ninjatrader.com/support/forum...d.php?p=501723, and lastly, provide the numbers that the remote support application provides.

After launching TeamViewer, please be sure to leave it open and running until I am able to connect.

[Harry]

Fat bug: Historical data base looks like Swiss cheese

Tested with NinjaTrader 8.0.6.0. and Kinetick realtime data feed.

I would like to come back because of this fat bug affecting the loading of historical tick data. The bug can be observed when the trading day starts prior to the calendar day.


Examples:

(1) I am downloading tick data for EURUSD for Wednesday April 12 via the historical data manager. Result: All data for April 12 (calendar day) has been downloaded but the last hour of tick data for Tuesday April 11 (calendar day) has been deleted by the historical data manager.

(2) I am downladoing tick data for EURUSD for the period from Wednesday April 12 to April 19 (updating my tick data) via the historical data manager. Result: all tick data for April 12 to April 19 has been correctly downloaded, but the last hour of tick data for Tuesday April 11 (calendar day) has been deleted by the historical data manager.

(3) I am opening a tick chart with a lookback period of 7 days. I am now reloading all chart data. Result: All chart data from April 12 to April 19 has been correctly downloaded, but the last hour of tick data for Apri 11 has been deleted from the historical data base.


Analysis:

The problem only affects tick data, because tick data is organized by hourly slices. When the historical data manager updates data, it seems to first remove all tick data for the selected trading day and then downloads it again from the data provider.

During this process the last hour of the prior calendar day (and belonging to the current trading day) is first removed, but not downloaded again. The result is one hour of data missing in the historical tick data base.

The missing tick data can not be immediately observed as a hole on the chart, because this hour or tick data is still sitting in the database cache (under db -> cache). However, if you change the bar period to a value that has not yet been cached, the hole in the historical data base will also cause a data gap on the chart.

Impact:

I have used the historical data manager every morning to update my data base. For my FOREX data for most of my instruments and for many days the last hour of trade data is missing. The data has been erroneously removed by the historical data manager.


Quick fix:

I am now downloading tick data by always starting the download period with a Sunday. The historical data manager does not remove data from Saturday, because there is no data that can be removed.


Please remove the bug!!!!

[Harry]

I have received information by NinjaTrader Patrick that the issue could be easily reproduced and that it it will be submitted to the development team for review.

The tracking number is QA-2934.

Thank you for your support.

[Harry]

Issue is definitely fixed with release 8.0.7.0

NT 8.0.7.0 with Kinetick Datafeed

I have tested the behavior with the last release of NinjaTrader 8, and I have not had anymore problems with the last hour of tick data being deleted for the last day prior to the download period.

Thank you for the fix!