Historical Data Downloader - If Data Already Exists

[tmfdouglas]

Hi,

Re. the behaviour of the historical data downloader, if the data already exists is it re-downloaded?

So for example if I want to download 6 months of tick data from IQFeed for all equities within the s&p 500 but some of that data exists will NinjaTrader download it again or realise that it already exists and bypass it?

If it doesn't download it again what granularity would NinjaTrader check? I.e. if there was already data for Febuary but one day of that data in the middle of the month was missing would Ninjatrader check and download that day only?

If it does download all the data within the data range specified regardless if it already exists is there any way to enhance the performance of the process? I'm trying to download tick data for the last 6 months for 500 equties, at first it maxes out the internet connection as expected but after a while the internet connection usage declines significantly and I assume is processing the data internally although this can take a significant amount of time and appears to crash (or is at least unresponsive). I've been breaking the download into week and month batches.

Also is it possible to enable further logging to see a more granular view of what the downloader is doing? Currently it just refreshes the symbol names in the bottom right hand corner but doesn't display what it's currently process, if it's downloading or processing the data etc... It crashes at times so would be useful to know where it got to so the remaining symbols / data ranges only could be restarted.

Thanks

[sledge]

Navigate to the nt8 directory in your my documents, go to db.

Note file sizes in the sub folders for instruments you are downloading.

Initiate another download.

Are they the same or doubled?

[tmfdouglas]

Thanks sledge although it's more questioning if ninjatrader knows what data it already has when initiating a historical data download and the ability to view a more granular progress update when downloading historical data.

Often the folder size has increased although it's only downloaded a subset of what it was meant to.

[NinjaTrader_EricB]

Hello tmfdouglas,

Thank you for your post! I will be researching your post and following you shortly, thank you for your patience.

[NinjaTrader_EricB]

Hello tmfdouglas,

Thank you for your post,

When downloading data via the Historical Data Manager, first the date range of data is removed, then replaced with new data.

However, I could see how an option to pick if you would like to replace all the data or only receive a notification of what data is missing would be helpful to many users instead of simply replacing the data. I will forward a feature request to our Product Managment team for future feature consideration.

The option to view the data status logging has been removed from NinjaTrader 7 to NInjaTrader 8, I'll also include a vote for that feature on your behalf as well.

Regarding the behavior you're seeing when downloading data, if it is freezing or causing the platform to crash, please perform this again then send me your most recent log and trace files along with a link to this forum post.

You will find the log file on your PC in the (My) Documents > NinjaTrader 8 > Log folder.

You will find the trace file on your PC in the (My) Documents > NinjaTrader 8 > Trace folder.

[tmfdouglas]

Thanks Eric, are there features suggestions ID's?

I'll run tests this evening and forward the logs etc...

Related to this, I'm trying to download data for SPY from IQFeed but regardless of the date range I specify it seems to only request the last 4 days from IQFeed. Am I doing something daft here? If I request tick data it will load according to the specified date range. Screenshot attached.

Thanks

[NinjaTrader_EricB]

Hello tmfdouglas,

Thank you for your reply,

I will reply again here with the request IDs are available.

I tested downloaded SPY data from our IQFeed test account and I was able to request further back, unfortunately your screen shot did not seem to attach. If this is persisting, could you please re-attach?

[tmfdouglas]

Hi, re-attached is the screenshot. It shows the download settings, the log file from IQFeed (where I've configured it to log the downloaded data and can see the end of one request and the start of the next and also the Ninjatrader minute folder for SPY.

For IQFeed, I've attached my subscriptions which should cover SPY.

Thanks

[tmfdouglas]

Also have successfully download SPY data across the date range outside of Ninjatrader to check it will come through from IQFeed. Also reset and repaired the Ninjatrader db but no joy.

Thanks

[tdouglas]

Hi,

Have ran further tests including from a Google VM instance to check if it was anything to do with my local install. Same results, can download tick data for any date range, but can only download minute data for the last 4 days from any instrument. Very odd.

It is an IQ Feed trial being used at the moment although the fact I can download tick data from further back and can download minute data through other trading software confirms that it shouldn't be effected by that.

I've emailed across all the logs including the IQ Feed log, you'll see at the end of the IQ Feed log file I enabled additional logging which includes the requests being sent to IQ Feed.

I also tried to download minute data form all s&p 500 instruments, same problem form quick scan of the data stored within NT.

Thanks

[tdouglas]

Ok, maybe forget these last few posts. I extended to IQFeed trial as a payed subscription and it can now download the additional data. Still doesn't explain why it could download historical tick data but not minute data and historical minute data could still be downloaded from other platforms. Anyhow, all's well that end well!

Feature requests are still valid.

Thanks for your time.

[NinjaTrader_EricB]

Hello tdouglas,

Glad to hear you were able to get to the bottom of the IQFeed issue.

The more flexible Historical Data Manager / data download options tracking ID is SFT-137

The status bar tracking ID is SFT-532

Thank you for your feedback!