Data loading for backtesting

[linuxrules]

This may be a dumb question but I noticed that each backtest loads the same historic data from the data feed ever time (resulting in a an error from IB soon enough) when I backtest the same future with the same timeframe.

It was my understanding - or belief - that NT would load data into the DB once and for all. Historic data will never change, right? So why load it again? I thought I could load 2 months, and then later load 2 more and so on, until I can finally test the whole range of say 6 months.

Also, I hoped that a second connection to OpenTick could overcome the limitations of IB. But NT doesn't seem to use that connection as a backup. Is this normal, or am I doing something wrong?

Thanks for any help or experience from others.

[NinjaTrader_Ray]

This should not be the case. Please try the following.

- Start NT and connect to IB
- Open SA
- Backtest the SampleMACross strategy on 1 min bars on ES 06-07 with a starte date of Monday June 4th end date of Wednesday June 6th
- It should load data and perform a backtest
- Run the same backtest again --> It should not re-load the data

Is this correct or does it reload data? If it does not reload data in this scenario, I would need an exact scenario where it does reload data when you believe it should not be.

If you want to use OpenTick, disconnect from IB and connect to OpenTick exclusively. Or, connect to OpenTick before you connect to IB. NT will then use OpenTick for both real-time and historical data requests.

Please let me know if you experience is otherwise.

[linuxrules]

Your way works fine. I tried FDAX for two months with 5min candles. This will result in the well-known error from IB. I suspect that data gets only saved if loading completes without an error. If this is correct could you change it and make it possible to fill the database in chunks?

I also noticed that when trying to backtest say April 1 to May 1 I see the dates counting starting from today back to the days that I actually need, resulting again in the pacing violation error message. I should be able to get a month worth of data regardless of which month I test, right? Well, it seems I can get about 40-50 days back from today, but nothing else.

OpenTicK reports an eror "no data" so I can't test it. It says 'connected' though. Frankly I feel I never got data from that connection ever. But I will play more with it when important issues are resolved.

[NinjaTrader_Ray]

Thanks for clarifying.

Here is how NT data loading currently works.

If you request 30 days and it does not exist in the dbase, NT will load it from the connected provider.

If you subsequently run a backtest for 30 days, NT will load data from the database since its already there.

If you run a backtest for 31 days, NT will reload all historical data for that time range.

What's happening with IB is that if you request two months and you get an error and not all the data is loaded and you re-request data, if data does not exist from the farthest day back, NT will re-load.

[linuxrules]

Oh, just as I suspected. Could you change that, or give the user the option of keeping partial data sets?

In addition there is still the problem that NT will request unnecessary data between now and the last day for backtesting, provoking IB to cut you off again and again, even though that data will not even be part of the test. Even if the user does not suffer from the limitations of an IB account it still takes time. Imagine I wanted to backtest the first week of a contract that is in its third month: I would have to wait for two months of data loading before I get what I want, even though I might already have the last 4 weeks in the database! This is also annoying for people who have historical data imported since the imported time period doesn't seem to get skipped.

Could someone take a look at that part of the program? I am sure many will appreciate it, and many will not send in questions like mine ;-)

[NinjaTrader_Ray]

I will put this on our list to review for sure.

[Gumphrie]

Losing data

Please add my 'vote'. Having the option to keep partial data sets will mean you don't lose any recent data and don't end up with gaps.

For example, I connect to my broker and collect tick and minute real time data on a contract for a couple of weeks, which I can look at on a minute chart. I then want to look at a daily chart, but don't have enough data (in fact NT doesn't show anything on a daily chart even though data has been collected), so I connect to OpenTick and get a daily chart back-filled, I then go back to my minute chart and find that my broker data has been wiped by inferior OpenTick data and there is now a gap of several minutes or more on my chart just before the current price.

Don't understand why my daily charts were not at least half-filled in the first place either. My broker is FB.

[NinjaTrader_Ray]

Daily bars are not built from real-time incoming data. Minute bars and tick bars.

Connecting to OT to download daily data will give you the daily bars but it will only reload minute or tick data for the current day IF you have a chart of one of those time frames open.