Historical data download and maintaining

[Arpad]

Dear NinjaTrader ,

I want to run several backtests on historical ES and EURUSD data.

I have two questions regarding this:

1) How to download data from my brokerage firm?
2) How to ensure this data against deletion / corruption?

How to download data from my brokerage firm?
I want ES and EURUSD minute data from Jan.1.2009 until yesterday.
I have a demo account at PFG.
· If I got it right I have to connect to PFG, open a 1-minute chart for ES and another for EURUSD, and go back in time until Jan.1.2009. Am I right?
· Do I also have to display lots of screens during this period (go through the whole period), or is it enough to go to the start and the end of this period?
· How much disk space does 1 year of minute data require?

I want to check whether I get different backtest results with PFG’s data and with another brokerage firm’s data, so I will connect to another firm and download their data for the same instruments and same time frames.
· How can I ensure that both data sets remains on my computer and I could switch among them back and forth as I wish?


How to maintain data integrity against deletion or corruption?
· How can I be sure that the data is unmodified and not any NinjaTrader script or functionality alter or delete it?


Thanks,
Arpad

[NinjaTrader_Jason]

Hello Arpad,

• Once connected to your data feed and you perform a backtest, it will load historical data from your data feed if not present in the NinjaTrader database. It is not necessary to create a chart first. However this way you can check if the historical data is available.

• If you use NinjaTrader 6.5, you will need to merge the individual expiries manually in order to backtest such a period or else you need to perform backtests for each expiry. Please see the link below for instructions to manually merge data.
  http://www.ninjatrader-support.com/HelpGuideV6/Merge.html
  
  NinjaTrader 7 can merge expiries automatically, so just the start and end date are sufficient. Make sure 'Merge policy' is enabled for the instrument in question. Go to Control Center-->Tools-->Instrument Manager, highlight the instrument in question and click Edit.

• I am not sure how much disk space a year's worth of minute data would require, but it should not be that much.

• I suggest to create an additional instrument in the Instrument Manager for the same instrument but only map it for the second data feed. Make sure no mapping is entered next to PFG in the Misc-tab of the Instrument Editor window. Ensure the existing instrument has no mapping entered for the second data feed. That way one instruments always uses PFG as data source and the second instrument uses the other data feed.

• Please avoid sudden power outages to avoid corruption in chart data.
  

[Arpad]

Dear Jason,

Thank you for your quick and thorough answer.

I’ve read the following statement in the Historical & Real-Time Data \ When does NinjaTrader load historical data? section:

“Note: The internal cache is managed by the .NET framework so it is unpredictable when the .NET runtime will clear it. In most cases, as long as you have at least one chart of your instrument open, the instrument’s data cache will persist.”

· Does this mean if I want to be sure about the historical data I have now will not disappear sooner or later, I better make an export through Historical Data Manager?



Do you know about any possibility for
· downloading historical trading minute data in bulk, for free, so not only one or two instruments or currencies but several one at once,
or
· one instrument at a time but that one particular instrument for a large period of time, for example 5 years of EURUSD or ES data?

Any website?
Torrents? (Is this a good source?)
Any recommendation?


Thanks,
Arpad Orfi

[NinjaTrader_Jason]

Hello Arpad,

Once you load historical data in NinjaTrader, it will be saved. You can check this in the Historical Data Manager. Please go to Control Center-->Tools-->Historical Data Manager and check if the historical data is listed under the Edit-tab. If it is listed there, it is also saved.

Unfortunately I am not aware of such free data sources. You can connect to Kinetick, Yahoo and Google for free in NinjaTrader 7. They all provide historical daily data for stocks only.

I searched our forum and found two sources posted by another customer. Please see the link below at post #23.

http://www.ninjatrader.com/support/forum/showthread.php?t=13753&highlight=free+historical+data