Random Incorrect data time stamps reported by Times[0][barsAgo]

[kkc2015]

Windows 10 Pro
NT Version: 8.0.25.0 64-bit

Subject: Incorrect data time stamps
Stock symbol: ^SPSISS
Data provider: Kinetick (free EOD)

The time stamps reported by Times[0][barsAgo] show some random incorrect dates, some of which are on Sunday (non-trading days). The historical database show the dates correctly.
Could you please look into it and recommend a procedure to resolve.

Files:

1. DataCheck.zip: indicator file to reproduce the data discrepancy.
2. DataDescreancy.gif: screen showing discrepancy.
3. ^SPSISS.Last.txt: exported data from Ninjatrader.
4. ^SPSISS_generatedby_Times.text: output from the DataCheck indicator.

kkc2015

[NinjaTrader_Jesse]

Hello kkc2015,

I Would suggest to try a more simple test without a loop and just allow OnBarUpdate to process the data as the script normally does. If there is a discrepancy in data we would want to see that without any custom logic to know that the problem relates to the data. If you otherwise need the logic to see the problem then you could further debug the logic to understand why that's happening by knowing the data processed correctly when just using OnBarUpdate.

Please try a test using Print(Times[0][0]) instead of what you have to confirm if the added logic relates to the issues.

I would also suggest to delete the data/cache and redownload the data before retesting. To remove the data and cache please use the following steps and then re download/retest.

Removing Historical Data: https://ninjatrader.com/support/help...ml?editing.htm

Once the data has been deleted close NinjaTrader. Open the Documents\NinjaTrader 8\db\cache folder. Select all files then right mouse click and select “delete.”

Open NinjaTrader and connect to your data provider. Open a new chart. and reload the historical data.

Please also double check if this happens for any other instrument using your test or if this is isolated to a specific instrument.



Please let me know if I may be of further assistance.

[kkc2015]

Jesse,
Thank you for the quick response.

1. Print(Times[0][0]) was tried prior to Times[0][CurrentBars[0]-i]. I repeated the trial testing and both provided identical incorrect time stamps for the tested symbol. The most obvious visual error is when the data date is on Sunday (eg. 2011-10-01). All subsequent testing was done using Times[0][0] only.
2. Historical data have been deleted and reloaded previously and re-performed again for this testing.
3. The time stamps provide the correct dates for most of the stock symbols that I tested. When I was looking at my sector indicator, I discovered that the time stamps are incorrect for the INDEX symbols. Thus, the time stamp errors seem to be applicable to the following indices (NYSE exchange, unless noted otherwise). It is noted that the time stamp for these indices is "12:00:00 AM" as compared to "04:00:00 PM" for most other stocks. Could this cause problem with Times[0][0]?

Times[0][0] reported incorrect dates for the following:
DJUSAR.X, SPSISS.X, DJUSAT.X, DJUSBK.X, DJUSBM.X, DJUSCH.X, DJUSCYT.X, DJUSEN.X, A1HCR.X, DJUSIR.X, DJUSNCT.X, DJUSRE.X, DJUSFNT.X, DJUSIR.X, DJUSSC.X, DJUSTC.X, A1TLS.X, DJUSUT.X, C.TX60.X(TSX)

[NinjaTrader_Jesse]

Hello kkc2015,

If you are specifically using daily bars what you are seeing is right, you don't get specific times with daily bars and only the date.

Print(Times[0][0]) was tried prior to Times[0][CurrentBars[0]-i]. I repeated the trial testing and both provided identical incorrect time stamps for the tested symbol. The most obvious visual error is when the data date is on Sunday (eg. 2011-10-01). All subsequent testing was done using Times[0][0] only.

From this detail it sounds like the loop was unneeded to see a problem, going forward I would suggest to form a new sample that only displays a print of the Time to keep it simple.

The times will relate to the series being used, if you used a non granular series like Daily bars that would end up showing a date and not a time like you have described. Please try executing a 1 minute bar test and check if you can see accurate minute times, please also exclude any looping logic and simply print Time[0]. A 1 minute bar test should show times in 1 minute intervals with no seconds. A daily bar would show dates and not times as another example. Only Tick based data will show a time granular to the second/milisecond.

Another suggestion we provide is to sync the PC clock. In any situation where you have data discrepancies that is usually one of the first troubleshooting steps to ensure your PC and the data server are using similar times without large differences.

[kkc2015]

Hello Jesse,
Thank you for the suggestions. My comments are probably not very clear and may have caused some confusions. The data discrepancy issue is summarized as follows:

1. NT Version: 8.0.25.0 64-bit
2. Data provider: Kinetick (free EOD), End of Day data only, no real time data, no hour data, no minute data, no data subscription.
3. Data discrepancy:
   1. The Dates shown on the NinjaTrader (NT) Historical Data dialog are correct.
   2. The Dates for most stocks symbols reported by Times[0][0] within OnBarUpdate() are in agreement with those from the NT Historical Data dialog but are RANDOMLY not correct for these symbols (DJUSAR.X, SPSISS.X, DJUSAT.X, DJUSBK.X, DJUSBM.X, DJUSCH.X, DJUSCYT.X, DJUSEN.X, A1HCR.X, DJUSIR.X, DJUSNCT.X, DJUSRE.X, DJUSFNT.X, DJUSIR.X, DJUSSC.X, DJUSTC.X, A1TLS.X, DJUSUT.X, C.TX60.X(TSX)). Except for C.TX60.X, all other symbols are from the NYSE exchange.
   3. It is noted that the time stamp for these indices is "12:00:00 AM" as compared to "04:00:00 PM" for most other stocks. I understood that it is not possible to get a meaningful time component from the EOD date but just bring to your attention that the "12:00:00 AM" may have caused the random date discrepancy as it is sitting in the middle of the two adjacent dates.
4. As per your suggestion, I have sync the PC clock. I set the PC clock to sync with “time.nist.gov” – suggested by NinjaTrader_Marco on 2020-07-23 for the Kinetick data feed. I repeated the procedure of rebooting the PC, reloading the historical data using the NT Historical Data dialog. This did not solve the problem.

To reduce further unguided testing from my part, I would appreciate it if you could conduct a simple test to see if you can reproduce the data discrepancy as follows:

1. Create a new indicator, which prints Times[0][0] within OnBarUpdate(). For that matter, print it from anywhere in the program to suit.
2. Open a new chart with Kinetick index symbol “DJUSAR” (NYSE) – see file “DJSAR_instrument.gif”. (Daily, Days to load = 8365) – see file “DataSeries_screen.gif”
3. Check the date “2011-10-01” (close=54.73) on the Ninjascript Output, which is on Sunday. Check the date “2011-09-30” (close=54.73) on the Historical Data dialog, which is on Friday. See files “DataDescrepancy.gif” and “DataDescrepancy1.gif”. This is an example of one of many random discrepancies, which are also present on all other symbols listed in item 3b above.

[kkc2015]

Jesse,
As shown on the attached chart, please note that Date & Close shown on the DataBox and the chart are in agreement with those shown on the NT Historical Data dialog and that there is no bar with date of 2011-10-01 (Sunday). Thus, summarizing:

1. NT Historical Data dialog, DataBox and bar chart - all showing identical Date (2011-09-30) and Close (54.73). The date of 2011-10-01 (Sunday) is not shown anywhere.
2. Times[0][0] is the only function that shows the incorrect date of 2011-10-01 with Close of (54.73).

Based on the above, the date reported by Times[0][0] is probably incorrect, noting that all of the above should report the same data regardless of the actual data value.

[NinjaTrader_Jesse]

Hello kkc2015,

I tried what you had described and checked the dates mentioned but I was not able to see the problem. If you have imported data from any external source please exclude that data by deleting it and reloading historical data from the EOD connection. You may also want to temporarily test using a different timezone by changing the timezone setting in the Control center -> Tools -> Options menu. After making that change you will need to delete and reload the historical data.

[kkc2015]

Hello Jesse,
Thanks for checking the data. All data from my historical database were not imported from any external source and were only provided by Kinetick free EOD. My NT time zone is set to (UTC-05:00 Eastern Time (US & Canada), which is basically the same as for the NYSE. For the performed tests, I have already deleted and reloaded the historical data a few times. In additional, I also "Reload all historical data" from the chart.

As you have already check the data, I would appreciate it you could upload the Times[0][0] results for the Kinetick index symbol DJUSAR.X for the period from 2011-September to present.

[kkc2015]

Hello Jesse,
For the Times[0][0] results, please also include High, Low, Close and Volume data.

[kkc2015]

Hello Jesse,

After additional thorough testing and inspection of results, I found that the data provided by Times[0][0] are not consistently synchronized with those shown on the NT chart and NT database.

The findings are as follows:

1. The NT historical data and chart data are always in sync and independent of the NT Time Zone setting.
2. For the said indices listed previously, the Times[0][0] data are time zone dependent and only synchronized with NT database if the NT Time Zone setting is later than UTC-05 Eastern Time. This may be caused by data with time stamp of 12:00:00AM, which is at the junction of two days. If so, why would this only affect Times[0][0] and not the Data Box from the chart or the NT database?
3. Given that the NT historical data source is from NYSE (UTC-05), it is logical that Times[0][0] data would be synchronized when the NT Time Zone is set to UTC-05.
4. If an indicator uses Times[0][0] to perform calculations, it is imperative that the data generated from Times[0][0] be in sync consistently with those from the NT Database and chart.

The procedures to perform the testing are as follows:

1. Data provider: Kinetick free EOD (no real time, only End of Day Daily data).
2. Create an indicator (DataCheck1) to print Times[0][0] and Closes[0][0] within OnBarUpdate().
3. Delete all workspaces and config.xml / UI.xml and reinstall over the existing with the same NT ver. 8.0.25.0 64-bit again.
4. Set NT Time Zone to UTC-6 (Central Time); exit NT; reload NT; delete the ^DJUSAR data from the NT database; open new chart with symbol ^DJUSAR (Kinetick symbol: DJUSAR.X) and the DataCheck1 indicator; Reload All Historical Data; save printed data to file.
5. Repeat above procedure for NT Time Zone setting of UT-5 (Eastern Time) and UT-4 (Atlantic Time).

Testing results (self-explanatory) are shown in the attached files and summarized as follows:

1. DJUSAR_scr_UTC-06_CentralTime_OK.gif (screen dump of relevant data with NT setting of UTC-06). Dates are in sync with those of NT database and chart.
2. DJUSAR_scr_UTC-05_EasternTime_Err.gif (screen dump of relevant data with NT setting of UTC-05). Dates of UTC-05 are one day in advance of those from NT database and chart.
3. DJUSAR_scr_UTC-04_AtlanticTime_Err.gif (screen dump of relevant data with NT setting of UTC-04). Dates of UTC-04 are one day in advance of those from NT database and chart.
4. DJUSAR_UTC-all.pdf (7 pages, data listing – some truncation to minimize bulk).
5. DJUSAR_UTC-04.txt, DJUSAR_UTC-05.txt, DJUSAR_UTC-06.txt (raw data printed by the DataCheck1 indicator).
6. DJUSAR_Last.txt (raw data exported from NT database).

For now, I could work around the problem by setting the NT Time Zone to UTC-06 (Central Time) instead of UTC-05 (Eastern Time, same as for NYSE data). My computer time zone is UTC-08 Pacific Time. However, if there are any data with time stamps of “01:00:00AM”, I will have to change the NT Time Zone back to UTC-05. As a user, I would expect that the dates generated by Times[0][0] be in sync with those from the NT charts, which are the same as those in the NT database, without changing the NT Time Zone setting to cater for the data time stamp.

[kkc2015]

Hello Jesse,
The remaining files.

[NinjaTrader_Jesse]

Hello kkc2015,
Thank you for the update.

I will test what was provided and see if I can reproduce the problem on my end to report it. For the time being I could suggest using the workaround you found if that is working on your system.

[NinjaTrader_Jim]

Hello kkc2015,

I was able to have a look while Jesse was out today, and I have been able to reproduce the symptoms you describe.

I have created an internal ticket for further review.

We will update this thread as more information becomes available.

[kkc2015]

Hello, Jim.
Thanks for the update.

[NinjaTrader_Jesse]

Hello kkc2015,

After internal review the reason for some dates to be shifted can be caused by the time zone selected in the Default 24/7 template. This would be expected currently, the issue tracking id is:14526

One workaround would be to use a trading hours template such as US Equities RTH instead of Default 24x7 and you do not see the reported issue.