Ninjatrader's historical data servers didn't record every tick

[bltoby]

Hello,

The test was posted here.

http://www.ninjatrader.com/support/forum/showpost.php?p=333981

This can impair, further, backtesting accuracy on tick based chart with reloaded historical data from Ninjatrader's servers. Please look into these issues.Thanks

[NinjaTrader_Bertrand]

bltoby, thanks for posting we will look into matters and get back to you. Generally what I would not expect is tick data reloaded from a server to match 100% what you've received in realtime - this is true across data-providers / feeds.

[bltoby]

Thanks for your reply.

Why across data-providers / feeds? I am talking about my and Ninja's recording of CQG feed.

1. Since Ninja's servers are co-located with CQG datafeed and my location is remote, what more logical is my recording missed some ticks but Ninja's recording should not. But the test shows a reverse result?

2. with native timestamp, there should be no different timestamp between two recordings.

[NinjaTrader_Bertrand]

I've just replied in the other thread where you posted as well. Would you mind rerunning your 15 min test again at a later point today?

The stamps are native correct, however NT would convert to your local timezone - that's why you see a difference in comparing exported vs locally recorded data, our data manager would always export it in UTC format.

[bltoby]

I've just replied in the other thread where you posted as well. Would you mind rerunning your 15 min test again at a later point today?

The stamps are native correct, however NT would convert to your local timezone - that's why you see a difference in comparing exported vs locally recorded data, our data manager would always export it in UTC format.

I know exported data are in UTC format, before compare, I convert two recording to the same timezone, when you look at my uploaded comparing report for yesterday's test on ES, you will notice the timestamp differences were only 1 sec.

I did run a test during Non-Farm Employment news on 6E for 30 min today, from CET 14:30:00-14:59:59 (EDT 8:30:00-8:59:59).

the result:

1. 34186 identical ticks

2. no more timestamp differences

3. Ninja' servers still missed 4 ticks (CET timezone)
143448,1.3282,1
143448,1.3282,1
143448,1.3282,3
145147,1.3223,1

[elitetradernyc]

Hi, bltoby, by Ninjatrader server, do you mean Kinetick or anothertick provider? Also can you do IB and compare?

[bltoby]

Hi, bltoby, by Ninjatrader server, do you mean Kinetick or anothertick provider? Also can you do IB and compare?

I tested with CQG feed only, no cross datafeed comparing. My own RT recording and Ninja's RT recording.

[elitetradernyc]

What is an RT recording?

[koganam]

What is an RT recording?

Real Time?

[bltoby]

Real Time?

What else?

[NinjaTrader_Bertrand]

All, we looked closely into this report and were not able to reproduce here. We compared tick data in 3 ways to approach from any potential direction:

a) recording live via NT's option to 'Save chart data as historical'
b) via recording to custom txt file with the sample indicator used earlier in this thread (OnMarketData() ).
c) via exporting historical data downloaded from the co-located CQG servers

To reproduce locally, here's the steps you would need :

1. Close NT
2. Delete Cache and Tick data from the database folder
3. Reopen and connect to CQG
4. Ensure Tools > Options > Data > Save chart data as historical checked
5. Open 1 tick chart of your product (we used 6E 06-13) and apply the GomiOMD Test indicator to it
6. Let this run for approx 10 minutes
7. Close the chart and disconnect
8. Export your NT saved data via the Historical Data Manager and import into Excel alongside the txt document created by the indicator in the C:\Temp folder.
9. Important: Pick a time window in between for which you would have a common reference, a new 1 minute increment so you could establish a correct 'cutoff' point for the data tick counts, you would need to delete data prior and past to this window set for analysis.
10. Close NT again and delete cache / tick data.
11. Restart NT and connect, reopen your tick chart for today and reload the data via right click > Reload Historical Data
12. Disconnect and go to the Historical Data Manager to export the last tick data we just downloaded from the server.
13. Import this file to Excel and delete any data not falling into our defined observation window under Step 9.
14. With the Excel row count establish how many ticks were observed for each test.

>> in my attached file showing the tests we've done you will see results matching up to the tick.

Note: the export timezone from the Historical Data Manager will always be UTC, so you have to account for that when comparing the data to the indicator storing to txt in local time zone.

Hope this helps, have a great weekend.

[bltoby]

All, we looked closely into this report and were not able to reproduce here. We compared tick data in 3 ways to approach from any potential direction:

a) recording live via NT's option to 'Save chart data as historical'
b) via recording to custom txt file with the sample indicator used earlier in this thread (OnMarketData() ).
c) via exporting historical data downloaded from the co-located CQG servers

To reproduce locally, here's the steps you would need :

1. Close NT
2. Delete Cache and Tick data from the database folder
3. Reopen and connect to CQG
4. Ensure Tools > Options > Data > Save chart data as historical checked
5. Open 1 tick chart of your product (we used 6E 06-13) and apply the GomiOMD Test indicator to it
6. Let this run for approx 10 minutes
7. Close the chart and disconnect
8. Export your NT saved data via the Historical Data Manager and import into Excel alongside the txt document created by the indicator in the C:\Temp folder.
9. Important: Pick a time window in between for which you would have a common reference, a new 1 minute increment so you could establish a correct 'cutoff' point for the data tick counts, you would need to delete data prior and past to this window set for analysis.
10. Close NT again and delete cache / tick data.
11. Restart NT and connect, reopen your tick chart for today and reload the data via right click > Reload Historical Data
12. Disconnect and go to the Historical Data Manager to export the last tick data we just downloaded from the server.
13. Import this file to Excel and delete any data not falling into our defined observation window under Step 9.
14. With the Excel row count establish how many ticks were observed for each test.

>> in my attached file showing the tests we've done you will see results matching up to the tick.

Note: the export timezone from the Historical Data Manager will always be UTC, so you have to account for that when comparing the data to the indicator storing to txt in local time zone.

Hope this helps, have a great weekend.

I did take all your directions, and take into account all you mentioned factors, cache, timezone, beginn of the data/ticks rows in recorded textfile, etc

FYI, I tested many times, some in slow market some in fast market.

And I noticed your attached data sample was from relative quiet market time. IMO the test data sample should include 40K ticks or more and from fast market. My test today during NPF news time includes 40K+ ticks. And ninja's recording missed only 4 ticks, it was few but odd.

1. In relative quiet market, my recording and ninja's recording match.
2. Today's test during NFP news, as I said before, showed that no more 1 sec timestamp difference, this seemed to be figured out.
3. But at active market condition, like ES US opening, 6E during US Non-Farm-Payroll, ninja's recording did miss some ticks.
I hope you could do some more tests next week, in fast market, at news time, ES US opening time e.g.

Thanks for your support and have a great weekend too.

[NinjaTrader_Bertrand]

Thanks for your contined testing and sharing of throughts on this matter, we will run tests on news releases this week to continue investigating reproducability.

[NinjaTrader_Bertrand]

bltoby, just wanted to let you know that this has not fallen 'off our radar' - we're still reviewing and reproducing this report here. Once I have more info I will repost then.

[bltoby]

bltoby, just wanted to let you know that this has not fallen 'off our radar' - we're still reviewing and reproducing this report here. Once I have more info I will repost then.

Thanks.
IMO two things to check:
1. why there are timestamp differences between recordings with native timestamp datafeed.
2. remote pc has more tick(s) recorded than local server in fast market, maybe server is overloaded because of too much ticks from all instruments?