Does Ninjatrader drop tick in realtime?

[NinjaTrader_Ray]

Guys,

Fair enough, I will run a test on Monday and verify with you my results however, please do not mix apples and oranges.

- It is not fair to say that you have a problem with feeds where we already expect mismatch on tick based bars, the ones you have mentioned are PFG, Barchart and Trading Technologies --> I repeat this, you will have discrepancy in tick based bars and this is outside of our control and not a NinjaTrader issue

- You both have mentioned Zen-Fire and I will run my test here since I know that the real-time feed and the historic storage are unfiltered --> I would expect near idential if not identical tick charts here

[biorjin]

Ray, I really appreciate that if you could look into this issue.

I just made my switch to Ninjatrader (I even switched my broker to AMP future because of this) and spent a lot of time and effort to set up my system in NT last week.

I really hope this is gonna be an easy fix.

Thanks again

Ray (my first name happens to be Ray too)

Guys,

Fair enough, I will run a test on Monday and verify with you my results however, please do not mix apples and oranges.

- It is not fair to say that you have a problem with feeds where we already expect mismatch on tick based bars, the ones you have mentioned are PFG, Barchart and Trading Technologies --> I repeat this, you will have discrepancy in tick based bars and this is outside of our control and not a NinjaTrader issue

- You both have mentioned Zen-Fire and I will run my test here since I know that the real-time feed and the historic storage are unfiltered --> I would expect near idential if not identical tick charts here

[biorjin]

Hi Ray,
I was doing some research about this issue and found Jason mentioned the realtime and historic data are not supposed to be same?

05-11-2009, 06:48 AM #2 NinjaTrader_Jason
NinjaTrader Customer Service



Join Date: Aug 2007
Location: Breda, Netherlands
Posts: 3,552



Hello brima,

After you reload historical data, the data that is reloaded should be the same. There is a difference between the real-time data and historical data with Zen-Fire.

Zen-Fire does not support historical data itself. As a courtesy, we allow Zen-Fire users to make use of our historical data server. So the real-time data is provided by Zen-Fire and the historical data is retrieved from the NinjaTrader historical data server.

Make sure you plot the exact same instrument and check if the exact same chart property menu settings are used (double click chart).
__________________
Jason, NinjaTrader Customer Service
NinjaTrader is a FREE application for advanced charting, market analytics, system development and trade simulation.

View schedule of upcoming online product training events.

[NinjaTrader_Ray]

Hi Ray,
I was doing some research about this issue and found Jason mentioned the realtime and historic data are not supposed to be same?

Re-read what Jason wrote. He said, after reload it should be the same. He was just outlining the difference which is that we provid the server technology for storage and retrieval of the historic data. The data is stored from the real-time feed that you get through Zen-Fire. I will run a test on Monday and report back.

[biorjin]

Okay, after re-read it carefully, my understanding is

1. Ninjatrader stores realtime data from Zenfire in its historic data server simply because Zenfire itself does not provide historic data download service.

2. Everytime, we hit "reload the historic data", we start to download the historic Zenfire data saved in Ninjatrader server.

3, TT provide filtered (throttled) live data feed. Zenfire data is unfiltered. TT live data is supposed to be different from historic data which is stored unfiltered Zenfire data.

Do I understand it correctly?

Thanks Ray

[mccullek]

I've tested Zenfire and TT, and the historic data for both match exactly, but neither will get close to the live data. Since it appears that the historic data is provided by Ninja, then that would explain the differences.

If you are only storing one master data base for historic data for both of these feeds, then that would explain why they may not match up with their live data. What feed does Ninja use to create the historic data base, and are any filters or changes in the way this data is collected added? Maybe your historic data is wrong, and your live data is correct? I'm just working through all of this in my head, but I can assure you, these differences are not minor at all. They are major and if the historic data is correct, then there is a problem with the streaming charts.

[biorjin]

Mccullek, here is what I found after spending whole day on the forum for the answer.

Ninjatrader has its own historic database, independent from the data provider (Zenfire, TT and etc). They are doing this because Zenfire and TT are not provide historic data download indeed(???) That historic database is recorded from unfiltered realtime Zenfire data.

Because TT provides filtered (40-50% less) data, the live TT data is not supposed to be the same as the unfilter (100%) historic data from Zenfire.

But the problem is even realtime Zenfire data is different from historic data (stored Zenfire data by NT) . So this is where the big problem is.

I am gonna run TT and Zenfire data at the same time Monday, making some snapshots and do more pictures after the historic charts are reloaded. Theoretically, the bar number should be historic = zenfire live >> TT live. Based on what I found in the last 2 days, it is historic > zenfire live >> TT live.

Anyone Ray has promised to look into this, so I still have my hope there and cross my finger for a quick easy fix.

I've tested Zenfire and TT, and the historic data for both match exactly, but neither will get close to the live data. Since it appears that the historic data is provided by Ninja, then that would explain the differences.

If you are only storing one master data base for historic data for both of these feeds, then that would explain why they may not match up with their live data. What feed does Ninja use to create the historic data base, and are any filters or changes in the way this data is collected added? Maybe your historic data is wrong, and your live data is correct? I'm just working through all of this in my head, but I can assure you, these differences are not minor at all. They are major and if the historic data is correct, then there is a problem with the streaming charts.

[mccullek]

Thanks for the update. I'm hopeful Ray will find an answer as well.

[NinjaTrader_Ray]

I ran the following test this evening:

- Connected to Zen-Fire
- 10 Tick Chart
- Session time was 6:25 PM to 7:00 PM Mountain US Time
- Exclude weekend was false
- I started this chart about 6:20 PM and then just a few minutes ago, I right clicked in the chart Reload Historical Data
- My chart bars are exact as I would expect since the historical data and my real-time are based on the same feed and NT was not designed to drop ticks

See the image attached, one is the real-time chart, the other is the reloaded chart.

What this means to me is that since you are not experiencing the same, then there must be some difference between what I am doing and you are doing. So, what I ask is that each of you send in an email to our support address, in the subject include "For Ray: Dropping Ticks" and reference this thread. I will assign it to one of our reps and instruct them to co-ordinate a time with you so they can logon to your PC, check settings and try to isolate the issue.

[mccullek]

Ray, I don't think this is going to tell you anything. First, this is only a 10 tick chart. I'm trading a 770 tick chart. Also, there is no real volume during the test period. Test a bigger tick chart during the regular trading hours and see what happens.

[NinjaTrader_Ray]

Ray, I don't think this is going to tell you anything. First, this is only a 10 tick chart. I'm trading a 770 tick chart. Also, there is no real volume during the test period. Test a bigger tick chart during the regular trading hours and see what happens.

See attached image.
- 770 tick on Zen-Fire
- 7:30 AM to 8:10 AM session times US Mountain
- Not one lost tick

If you wish further personal assistance, please follow my instructions in my prior post.

[mccullek]

I have been running mine all morning. Printed the streaming chart, and then re-loaded. Two completely looking charts, not even remotely the same in regards to number and size of the bars.

[NinjaTrader_Ray]

I have been running mine all morning. Printed the streaming chart, and then re-loaded. Two completely looking charts, not even remotely the same in regards to number and size of the bars.

Thus there must be something different between what you have set up and my test scenario. I am happy to help resolve this for you, please see my instructions about sending me a support request so I can put one of my tech's on it. Thanks.

[mccullek]

OK, thanks Ray. I just sent you an e-mail.

[wodru]

Ray,

NinjaTrader has an old problem with reloading historical data, at least i have this problem since 2 years:
- Take a 1 tick chart
- note the time of your computer clock and reload historical data
- when the reload is finished after a few seconds, look at the ticks around the noted time.
- look for the first tick before this time and for the first tick after this time.
- when you check the timestamps of these 2 consecutive ticks, you will see that they are 5 to 50 seconds apart, note these timestamps
- now reload once more historical data and see how many 'new' ticks appear between these 2 timestamps.
I tested the FDAX with Zenfire today and had 20 to 160 new (missing) ticks.

When i look at the timestamps, i see 2 problems:
- Ninja often doesn't reload enough data, many times the reload period stops 20-30 seconds before the start of the reload
- Ninja discards all old realtime data, and restarts collecting only after the reload is finished.

So you always have a gap after reloading data, i had many times a missing 1 minute bar on my charts.

Since tick bar charts are built by accumulating the ticks, these missing ticks really change the chart, especially if you look for bar patterns. (thats one reason why i don't use tick charts)

Just try this sequence with a 10 tick bar chart: reload data -> collect data for a few minutes -> save the chart image -> reload data -> compare

I think Ninja should not discard the old realtime data, but should keep an overlapping period with the reloaded data.

Since you have the timestamps, price and tick volume for the overlapping period, it should be possible to make a reasonable good connection between the realtime and historical data, with minimal errors.