Does Ninjatrader drop tick in realtime?

[NickA]

I haven't tested as thoroughly Wordu has but have observed similar, you do get gaps after a reload. It's as if NT stops collecting ticks whilst the last 'chunk' of historical data is fetched leaving a gap between the end of the history and when the realtime sparks up again. This is with zenfire.

[NinjaTrader_Ray]

Ray,

NinjaTrader has an old problem with reloading historical data, at least i have this problem since 2 years:
- Take a 1 tick chart
- note the time of your computer clock and reload historical data
- when the reload is finished after a few seconds, look at the ticks around the noted time.
- look for the first tick before this time and for the first tick after this time.
- when you check the timestamps of these 2 consecutive ticks, you will see that they are 5 to 50 seconds apart, note these timestamps
- now reload once more historical data and see how many 'new' ticks appear between these 2 timestamps.
I tested the FDAX with Zenfire today and had 20 to 160 new (missing) ticks.

When i look at the timestamps, i see 2 problems:
- Ninja often doesn't reload enough data, many times the reload period stops 20-30 seconds before the start of the reload
- Ninja discards all old realtime data, and restarts collecting only after the reload is finished.

So you always have a gap after reloading data, i had many times a missing 1 minute bar on my charts.

Since tick bar charts are built by accumulating the ticks, these missing ticks really change the chart, especially if you look for bar patterns. (thats one reason why i don't use tick charts)

Just try this sequence with a 10 tick bar chart: reload data -> collect data for a few minutes -> save the chart image -> reload data -> compare

I think Ninja should not discard the old realtime data, but should keep an overlapping period with the reloaded data.

Since you have the timestamps, price and tick volume for the overlapping period, it should be possible to make a reasonable good connection between the realtime and historical data, with minimal errors.

This is expected. See my post #15 for an explanation.

[NinjaTrader_Ray]

I have been running mine all morning. Printed the streaming chart, and then re-loaded. Two completely looking charts, not even remotely the same in regards to number and size of the bars.

Update: The reason for the discrepancy was due to the fact that the real-time feed was Trading Technologies and the historical data is based off of another source. The two data sources are different and thus it is expected to have differences. Also, the real-time tick filter was enabled which would also make a difference.

[wodru]

This is expected. See my post #15 for an explanation.

I already had read your post (#13) before writing my last post.

What is expected? Up to 50 seconds gaps while reloading data to fill gaps?

Here are my questions to your statements, it would be great if you could take the time to answer the questions:

"NT will retrieve all historical data from the time of the request from the server"

Many times the reloaded period stops 20-30 seconds before the time of the request, i have just observed that again at several tests

1) Can you confirm that NT often doesn't reload enough data ?

"There will be a small gap of time while the data was loading that you will "miss" ticks"
"....since you can easily miss 50+ ticks during the reload"

Level2 data is constantly flowing, the replay files are filled, so you could keep the old realtime data while reloading

2) Do you think it makes no sense to keep the old realtime data? (maybe because of the 'joining' problem?)

"There is no way to reconcile this since NT has no way to know what ticks were missed since ticks don't have a unique id on them"
"We can't say -- We received tick # 100,000 historically and in real-time we are on tick # 100,100 thus, go back and get the missing 100 ticks"

You have the seconds timestamps for the historical and the realtime ticks. Even when the timestamps are server/local computer generated, the time 'uncertainty' should not be more than 2 seconds and you could just arrange the historical and the (old) realtime ticks by the timestamps

3) Wouldn't it be preferable to have a 2-3 seconds period of uncertain ticks instead of an up to 50 seconds gap?

I made many tests today, the gap range was from 5 to over 50 seconds, the mean value was around 30 seconds (Zenfire, 2 day reload of 1 tick bars, ES and FDAX)

4) Can you confirm that the reload of historical data can leave gaps of up to 50 seconds?

5) Do you have plans to change this behaviour?

6) Will NT7 have similiar reload gaps?

Thanks

[mccullek]

Update: The reason for the discrepancy was due to the fact that the real-time feed was Trading Technologies and the historical data is based off of another source. The two data sources are different and thus it is expected to have differences. Also, the real-time tick filter was enabled which would also make a difference.

I tested the 770 feed with Zenfire today, and again, the charts don't match. The test I did with Bertrand was only a 10 tick for a short period of time. I don't think it was long enough to be representative of the problem. I'm doing the reinstall as suggested, so we will see if this is the problem.

[NinjaTrader_Ray]

I already had read your post (#13) before writing my last post.

What is expected? Up to 50 seconds gaps while reloading data to fill gaps?

Here are my questions to your statements, it would be great if you could take the time to answer the questions:

"NT will retrieve all historical data from the time of the request from the server"

Many times the reloaded period stops 20-30 seconds before the time of the request, i have just observed that again at several tests

1) Can you confirm that NT often doesn't reload enough data ?

"There will be a small gap of time while the data was loading that you will "miss" ticks"
"....since you can easily miss 50+ ticks during the reload"

Level2 data is constantly flowing, the replay files are filled, so you could keep the old realtime data while reloading

2) Do you think it makes no sense to keep the old realtime data? (maybe because of the 'joining' problem?)

"There is no way to reconcile this since NT has no way to know what ticks were missed since ticks don't have a unique id on them"
"We can't say -- We received tick # 100,000 historically and in real-time we are on tick # 100,100 thus, go back and get the missing 100 ticks"

You have the seconds timestamps for the historical and the realtime ticks. Even when the timestamps are server/local computer generated, the time 'uncertainty' should not be more than 2 seconds and you could just arrange the historical and the (old) realtime ticks by the timestamps

3) Wouldn't it be preferable to have a 2-3 seconds period of uncertain ticks instead of an up to 50 seconds gap?

I made many tests today, the gap range was from 5 to over 50 seconds, the mean value was around 30 seconds (Zenfire, 2 day reload of 1 tick bars, ES and FDAX)

4) Can you confirm that the reload of historical data can leave gaps of up to 50 seconds?

5) Do you have plans to change this behaviour?

6) Will NT7 have similiar reload gaps?

Thanks

1. NT will reload data from the time of the request and will return whatever data is given to it from the historical data server you are connected to.

2. We currently don't use any real-time data that was streaming during the request since there is no accurate way to merge this data with data returned from the server. We could do what you suggest however, it still will be inaccurate since either we have more ticks than there should be or not enough. This is due to the fact that there is no incremental id # per tick to know what you are missing.

3. I can definitely see how this could be better for more liquid markets however, it is also can work the opposite for less liquid markets. The reason is, you could introduce a large set of duplicate trades. In short, its a pick your poison scenario.

4. For sure there can be a gap of 50 seconds and there could also be a gap of 1 second. Too many variables including your data provider, your internet bandwidth and the sheer size of the amount of data and how well that data is compressed at the server. What we do to optimize this is locally store previously requested data excluding the current day. That way you are only loading data for the current day.

5. NT7 does implement more granular loading of tick data (in hour increments instead of days) if the provider supports this and our bar building and bar caching routines dramatically speed load times so the delays you may experience are definitely minimized.