Too many Historical Data Requests

[zoltran]

I know this has been discussed before, but it's such a vexing issue I'm going to take another crack at explaining the problem.

The issue is the way amd frequency that NT makes historical requests.

Try this scenario ..
1- Make sure you have an updated chart or mktAnalayser running .. as we want to ensure that NT has the current data already. Lets say we have charts for YM, NQ and ER2 running. Use Interactive brokers or Opentick for backfill .. these are slow fillers and make the problem much worse.
2 - Open up a Strategy Analyzer
3 - Backtest YM - NT will do a backfill of todays data - WHy? As it already has a current chart????? ANd ... NT is pretty much locked up while this is happening.
3b - Look at the chart in the backtest/strategy results .. NT does a backfill again .. Why ?
3c - Change the currency display for the backtest .. Again a backfill
3d - Poke around in the strat analyzer tabs .. backfills are triggered for a number of tasks.
3d - Run another backtest on YM.. but for a different period.. and again .. a backfill

Eventually NT calms down and doesn't do the fills and/or the fills see 'shorter' in duration. For example, doing a 2nd backtest on YM with the same parms will often not request data .. but this is inconsistent.

Similar scenarios exist with adding symbols to Market analyzers, new charts etc for symbols that are already being graphed someplace else .. NT will very often go back and re-request the data ... Data it already has. And in the case of IB .. it will receive inferior data on backfill vs the RT streaming data it has already collected.

Hopefully you guys can figure a way to reduce this behavior.
I find I disconnect any backfill source when doing a lot of backtests .. it's just to slow/annoying otherwise.

[NinjaTrader_Ray]

Hi Zoltran,

3) When running a backtest, NT will load the data based on the range you have defined and for the interval base defined (tick, minute or day based intervals). For example, if your MA has 5 days (aprox 2000 bars regular session bars) of minute data loaded (in an internal data pool) and you then run a backtest for the past two days on 1 minute bars, the SA will not load any data. I just verified this. If you run a backtest on 6 days data, NT will reload the entire data set for the six days.

3b) Are you sure it does a back fill or could it be that NT re-runs the backtest which might look like a back fill? If the former, this is true and something that is on our list to change for the next major revision due out in fall.

3c and d) Same as 3b

3e) Using the scenario in 3, if I changed interval type to 2 min, it will not reload data. I change to 100 tick, it will load data since tick data for this instrument does not currently exist in the internal data pool.

[zoltran]

3a .. It might be re-running the test, but it says loading data, usually with todays date.

The net of this is ... SA is only usable for me if I disconnect the backfill source.
Especially in the case of IB, where the backfill data is of poorer quality than the RT data I have carefully collected.

Reloading data seem odd if the start date of the data is already in the DB.
Which it is in the scenario I outlined.

[NinjaTrader_Ray]

Understand. I guess what I am saying is that it should not operate the way you describe. I would need a way to reproduce this behaviour. Lets try the other way around. Let me provide you step by step what I have done and lets see if you can reproduce it or if you get the expected behaviour that I see.

- Temporarily move your Default.xml workspace file to another location
- Restart NT --> Should only see Control Center
- Open up MA, add the ER2 06-07 instrument only
- Change the MA properties "# bars look back..." to 6000
- Connect to IB
- Add an SMA indicator on 1 minute series --> Should load data
- Open SA
- Backtest the SampleMACross strategy on 1 min bars with a starte date of Monday 28th end date of Tuesday 29th
- It should backtest immediately without loading any data

Please let me know if you experience is otherwise.

[zoltran]

Ray .. just to clarify..

Have you re-created this problem using IB data ?
and/or
Do you need me to re-create following your steps ?

[NinjaTrader_Ray]

I am unable to re-create and would appreciate it if you have a few moments to spare to follow the steps as I outlined.

Thanks