Missing Data - CQG vs Zen Fire

[bobcat70]

I have brought this up before in someone else's thread, but, I really need an answer to how to resolve this overall issue.

Issue: Historical futures data is sometimes missing or doesn't get updated after disconnected times.

Diagnosis: We have two separate accounts that allow me and another trader to compare CQG data feed with Zen Fire. We do not believe this is necessarily a feed problem from those companies, but rather how Ninja is storing the data.

Issue #1: Comparing CQG to Zen. Take a look at these two trace files that our strategy generates on a backtest. You will notice that the screen shot from Zen Fire is missing an entire day's worth of QM data on 7/8. It goes from 7/7 to 7/10. Compare that to the CQG data run using the exact same strategy. The CQG data (running on a separate box), has the 7/8 data.

Zen Fire: http://screencast.com/t/gPqfSZS56t
CQG: http://screencast.com/t/k1BUg6Pa0vT

Issue #2: CQG data is missing a single 15min bar as shown above in that screen shot at 12:00:00am.

==================

1. How do we resolve this issue? Two separate feeds, two separate boxes, same overall problem.
2. Is there a third-party program that can go in and tell me when I'm missing single bars, or an entire day?
3. We keep re-downloading instrument data either using Reload All Historical data or deleting the history from the Historical Data manager and redownloading that. Why should I have to keep doing this? And if so, at what interval - every day, every hour?
4. If I have a strategy running and an open position, then get disconnected for an hour, from my readings in the posting on this forum, I have no option to reload the missing data. It will recalculate what is there, but how can I redownload the historical data programatically? As you could tell, missing data would cause a change in entries and exits.

==================

Please understand that we have invested months of development on a variety of strategies. I have to trust that I have all the data to backtest and go live with. If I'm missing one or two bars, it could massively adjust an entry or exit.

[NinjaTrader_Bertrand]

Hi Bob, thanks for the post - first of all the historical data provided is offered as a courtesy by our data servers - it might be worthwhile to connect to a dedicated high performing datafeed like Kinetick first to provide the backfill. That being said I tried to reproduce the missing data issue here on my end and was unable - what sessions are you using in those cases and which NT7 version would be run?

There's unfortunately no 3rd party program I'm aware of for integrity checks but I do understand the need for it and have forwarded this suggestion to product development for future consideration, the same would apply to offering a programmatic data reload. Currently NT7 would reload data in those cases : http://www.ninjatrader.com/support/h...rical_data.htm (second section here please).

Thanks,

[bobcat70]

Bertrand,

first of all the historical data provided is offered as a courtesy by our data servers

Using CQG or ZenFire, all the historical data comes from your servers, not their servers? I thought CQG had their own historical data servers? I certainly have that option turned on in the connection.


That being said I tried to reproduce the missing data issue here on my end and was unable - what sessions are you using in those cases and which NT7 version would be run?

That's my point. I too can delete all the historical data, re-download it and THEN its ok. But, when in trade, how do I know if I have all the correct data, whether its your historical data or live data? We are using 24/7 Session.


I really need a solution now, not in some future version. How can I trust the data that is being saved? Is Kinetick my only solution? Do I have to pay to get 5, 15min bar data? If I have a live trade active and I lose connection and now I have out-of-date data, it would seem the only right thing to do is kill the strategy, close the positions, reload the data, and restart the strategy. Surely, all your clients are not doing this?

If downloading the data again does get this historical data, then wouldn't that suggest it was there all along? Why wasn't it saved the first time locally? This still leads me back to the issue being in Ninja, not the data providers.

[bobcat70]

Continuing the Investigation:

Take a look at my Historical Data Manager for QM 08-11 - July 8, 2011: http://screencast.com/t/d90DP3VR5V


I disconnected and re-connected. When I re-run the strategy in Analyzer, now I see the 12:00:00AM data. http://screencast.com/t/PhlEUqKQ


Its these inconsistencies that are driving us crazy.

[bobcat70]

Apparently, this specific issue was brought up in 2009: http://www.ninjatrader.com/support/f...hlight=getbars

Almost 2 years later, we still don't have a solution for us auto-traders. NinjaTrader, please help us out!!!

[koganam]

Are you sure that you have turned on the option: "Get data from server (if available)" ?

[bobcat70]

You mean this, yes: http://screencast.com/t/QfeHkio4

[NinjaTrader_Bertrand]

Bob, NinjaTrader would trigger an automatic reload of data in the cases mentioned in the helpguide -

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

Perhaps you could improve your performance seen by following the suggestion to have an MA window with an indicator added for all instruments you need and thus the cache is up to date at all times.

If you have a scenario where the automatic reload triggered failed for you, please let us know so we could look into it here.

[bobcat70]

Bertrand,

Thanks for that explaination. It was not clicking for me that you MUST have a chart open for each Strategy you are running. Could you please add a line in that page of the manual - smething like this:

NOTE: When running a Strategy, make sure you have at least one chart open for each instrument in your strategies tab. If there is a disconnection and reconnection, then the chart will reload the instrument data.

Its unfortunate though, that upon reconnection, it doesn't just reload the data for any strategy you have running, THEN recalculate.

[NinjaTrader_Bertrand]

Thanks Bob, I will surely forward the enhancement suggestions for the helpguide and recalculate handling of the strategies.

[NinjaTrader_Bertrand]

We have added this kind enhancement request now under # 1119 in our product development lists. Thanks for the contribution.

[cashinvestor]

I have had good luck with QCollector downloading esignal data. He now has a DTN version too. You just make a list of symbols and intervals and let it rip. It will suck down all the available data from the server you are logged into.

The Kinetic data feed is OEM'd from DTN IQ feed. One issue I have with Kintetic is that the connector is built-in to Ninja, so there is no login/password account. QCollector needs to login. So you will have to either get a Kinetic account with a login or go to DTN and get a real account through them, which is usually more costly. If you have esignal, then just use that login.

QCollector can now build data sets for direct import into Ninja. This guy is a great programmer and QCollector is one of the finest, most fun pieces of software I have used in the trading arena.

http://www.mechtrading.com/qcollector/dtn/ninjatrader.htm