Does NinjaTrader drop ticks real-time ?

[aviat72]

Please address this tick based indicator issue

Now that NT7 is in production, please do address this issue of supplying historical tick by tick data. It will require some changes in the db (synching of the bid/ask/last ticks) and a finer grain time-stamp but it is absolutely critical for a lot of us. I am sure you had your reasons for keeping the replay and historical data separate but please focus on making NT's data-handling fool-proof. All the bells and whistles become irrelevant if you can not get the right data quickly.

Gomi has done a lot of work on working around this problem, but there is nothing like having native support for these features, instead of having a recorder running 24/7 or having to export the data from iRT (as I do) or having quote collector scripts.

[NinjaTrader_Bertrand]

Now that NT7 is in production, please do address this issue of supplying historical tick by tick data. It will require some changes in the db (synching of the bid/ask/last ticks) and a finer grain time-stamp but it is absolutely critical for a lot of us. I am sure you had your reasons for keeping the replay and historical data separate but please focus on making NT's data-handling fool-proof. All the bells and whistles become irrelevant if you can not get the right data quickly.

Gomi has done a lot of work on working around this problem, but there is nothing like having native support for these features, instead of having a recorder running 24/7 or having to export the data from iRT (as I do) or having quote collector scripts.

aviat72, thanks for your feedback - this is on our list to look into.

[RJay]

Now that NT7 is in production, please do address this issue of supplying historical tick by tick data. It will require some changes in the db (synching of the bid/ask/last ticks) and a finer grain time-stamp but it is absolutely critical for a lot of us. I am sure you had your reasons for keeping the replay and historical data separate but please focus on making NT's data-handling fool-proof. All the bells and whistles become irrelevant if you can not get the right data quickly.

Gomi has done a lot of work on working around this problem, but there is nothing like having native support for these features, instead of having a recorder running 24/7 or having to export the data from iRT (as I do) or having quote collector scripts.

NT,

If you are not intending to do this in the near future, I need to know.

Gomi's stuff is nice but I need true level1 and level2 recorded info.

If you are not going to do this anytime soon, I will be forced to do it myself.

I need your timetable on this.

Thanks,

RJay

[NinjaTrader_Ray]

RJay,

This is a critical item on our list however, we have a large list. At this exact time, we do not have this feature scheduled into any development plans nor can I commit to anything thus there is no way I can say it would be done in the short time.

[solvmatic]

I am a network guy and worked for a large ISP for years. I don't know if anyone ever look at this issue from network perspective. it is very normal that packets are dropped and retransmitted due to, for example, traffic shaping or policing which is very normal practice at busy hours by routers, resulting the packets arrive out of sequence.


Not a programmer, but feel interested to know

-does the market events have sequence id coming from data providor or simply time stamped
-if an event come in a little late, and the following events already went into the pipe, how Ninja process it, does Ninja drop the late event or insert them back in the place where they suppose to be?
-what the TCP or UDP port Ninja uses, is it treated as RTP traffic by the router network?

[NinjaTrader_Austin]

solvmatic, I will have someone get back to you on Monday.

[NinjaTrader_JoshP]

solvmatic,

Ticks will not be dropped by NT. It should be noted that this thread's "issue" was determined to be an improper setup and not related to any network issues or such. It was more of an apples to oranges comparison.

[NinjaTrader_Ray]

[QUOTE=solvmatic;207855]
-does the market events have sequence id coming from data providor or simply time stamped
-if an event come in a little late, and the following events already went into the pipe, how Ninja process it, does Ninja drop the late event or insert them back in the place where they suppose to be?
QUOTE]

- No sequence ids, only time stamps
- NT processes the events in the sequenece it receives them

[Flyingscots]

Hi Gomi
I am having troubles with the volume figures in your indicator GomLadder because the total volume showed by the GomLadder does not match the volume showed by the Time and Sales or by the Market Analyzer.
I double checked the txt file created by the GomRecorder and it is not correct. The file is FGBL 03-11 from 10:29:28 (GMT) to 15:58:55 (GMT): the total volume is about 288'000 contracts while the volume showed by the Market Analyzer is around 577'000. So I believe the problem lies in the GomRecoder itself. Am I wrong? Maybe I did not set the GomRecoder in the correct way?
Thanks for your answer

[bltoby]

For more clarification - here's the exact route I took to reproduce and look into -

1. Setup custom session template from 9:30 - 9:45 EST
2. Under Tools > Options > Data used 'Save chart data as historical' and unchecked 'Get data from server'
3. Connect to Kinetick only
4. Open up 1 tick chart with custom session as defined in 1. and 0 bars lookback.
5. Apply Gomi's sample OnMarketData() indicator
6. After the 15 min was finished, I imported both the custom textfile from the indicator and Historical Data Manager tick export to Excel and summed the ticks received from 9:30:00 to 9:44:59, taking into consideration that the indicator would not be called for the first 20 bars, the unstable period.

In both tick recordings I arrived at the exact same summation value.

Please let me know if you were doing something different then my above steps or if you would see different results following them exactly as outlined.

Thanks,

I did the same test with your settings.
1. Setup custom session template. I'm testing with ES too.
2. Under Tools > Options > Data used 'Save chart data as historical' and unchecked 'Get data from server'
3. Connect to CQG Live only
4. Open up 1 tick chart with custom session as defined in step 1.
5. Apply Gomi's sample OnMarketData() indicator
6. After the 5 min was finished, I saved the custom textfile from the Gomi's indicator.
7. Uncheck 'Save chart data as historical' and check 'Get data from server'
8. Open up another tick chart, reload Data from Ninjatrader's historical data servers.
9. Export reloaded historical data from Historical Data Manager->Edit, processed it with Excel, making sure the beginning of data was the same, then saved as textfile.
10. compare the two textfiles.

The testing data was for ES on 2013-06-06 CET 15:31:00 - 15:49:59 (EDT 09:31:00 - 9:49:59) on CQG Live connection

the comparing info for the ES opening 4 minutes:
1. 44403 identical ticks
2. 610 different ticks (only timestamp difference). Since CQG is sending data with native timestamp, and Ninjatrader is recording CQG Live data, why there are timestamp differences between my recording and Ninjatrader's recording?
3. Ninjatrader's recording missed/dropped 112 ticks.

the two data textfiles and comparing report attached.

[NinjaTrader_Bertrand]

bltoby, thanks for the report - would you mind emailing us under Help > Mail to Support with your trace and log files so we could review the setup?

My server management team also informed of a minor issue this morning, which has been corrected now. Could you please restart your NT and reconnecting fresh to your CQG connection and then potentially run another 15 min window test later today?

I would greatly appreciate it.