programatically reload historical data?

[edstaffin]

Hi,
I am writing a strategy and it's accuracy depends on having all historical data available to calculate itself properly. I find myself frequently opening a chart, right clicking and choosing reload all historical data to pick up missing data. Is there a way to do the same thing via code in the OnStateChange event handler?
Thanks ... Ed

[NinjaTrader_PatrickH]

Hello edstaffin,

Thank you for your note.

Unfortunately, there would not be an option to do so in code.

[NinjaTrader_Matthew]

Not sure why you needed to reload historical data right after opening a chart. The data loading rules should have handled that for you. Please make sure you have your market data options set up optimally: Tools-> Options-> Market data-> "Historical" ensure you have checked "get data from server" and under "Real-time", ensure you have un-checked "record live data as historical".

If you truly find yourself missing data directly after opening a chart, then this would be a bug and would need to be reported / analyzed with reproducible steps.

With that said, there is the ReloadAllHistoricalData() method, but this was not designed to be called from an indicators in OnStateChange / Configuring an indicator or strategy. ReloadAllHistoricalData() will re-trigger the indicator to be reloaded, and therefore you'd find yourself in an infinite loop where your indicators OnStateChange is hit, data is reloaded, indicator is re-loaded, etc, etc.

There is an example of the intended usage of reloading historical data after a connection event which is outlined on this page:

http://ninjatrader.com/support/helpGuides/nt8/en-us/?reloadallhistoricaldata.htm

[DaveE]

Chart load after connection loss results in gaps in Historical Data NT8 B12

Hello Matthew,

Not sure why you needed to reload historical data right after opening a chart.

I just discovered a reason today:
I found that when adding a strategy to Strategies tab it was starting in a different position to starting the same strategy from the Chart (using exactly the same instrument and timeframe settings, this is multiple minutes say 60). I found the reason was that although the chart correctly showed all expected bars for yesterday these bars were all missing in the data loaded by the strategy in Strategies tab. So the strategy in Strategies tab was using data from say Wednesday and Friday only, but the chart strategy was using Wednesday, Thursday and Friday.

Yesterday I stopped and started NT, I think after restart the chart showed 'loading...' which indicated that it was getting the missing historical data from IB, however something in NT seems to have prevented storage to the historical data database for most of yesterday.
I kept NT on overnight and I see that it re-started recording historical data from IB into the database at the end of the day (without any intervention by me).
The nasty thing about this is that there is no error logged such as 'unable to record historical data since ...', there is no indication from the chart that there is any data (in the database ) missing, so there is no reason for someone to think that running the same strategy from Strategies tab might be using different historical data (with gaps).


I have these settings:

Please make sure you have your market data options set up optimally: Tools-> Options-> Market data-> "Historical" ensure you have checked "get data from server" and under "Real-time", ensure you have un-checked "record live data as historical".

I am using IB where historical data is a 'costly resource' since they limit the amount you can retrieve in a time period.

I looked at this example http://ninjatrader.com/support/helpG...oricaldata.htm
But putting ReloadAllHistoricalData() in all my strategies carries the risk that IB will stop responding to historical data requests. What is really needed is a function more like ReloadHistoricalDataForGapsOnly, so if no gaps were detected in the database (for the period specified in 'days to load' in strategy settings), this function would not need to make any historical data requests.

I can think of a couple of workarounds until this is fixed in NT8:

1. Run strategies from a chart that can be visually checked to have no data gaps.
2. If running strategies from Strategies tab, add code to check for data gaps in historical data. If gap detected warn user and dont allow strategy to proceed (user has to manually use Reload All Historical Data from a chart covering the period of the detected data gap).

[NinjaTrader_Matthew]

Thanks for sharing your experience and thoughts here. I still maintain, based on the designed data load rules, there is no reason you should have to reload data right after starting a strategy. The process of starting the strategy should have handled that for you - any reasons you uncovered would be a bug.

The core does not have any heuristics to check for and fill in load gaps. It is a valid request to add, and is something that has come up before even during internal discussions, but it is not a trivial implementation, so we need to push that off for another day. I have logged your request below under SFT-1371.

Rather than try to discuss possible workarounds or providing NS methods as a workaround, I would rather just focus on the issue you experienced. Any gaps as a result of starting a strategy from the strategies tab would not be expected and should be a priority to fix. Based on the info you provided, it would be difficult for our engineers to determine why that occurred so I would need a little more information to make sure we're going the right direction in our testing.

Can you elaborate on the different position start time from the strategies tab vs. the chart? What did you mean by "this is multiple minutes say 60"?

- Exactly what interval was configured from the chart and strategy tab? Daily, minute, tick?
- From the chart, were you loading based on "Days to load", "Bars to load", or "Custom Range"? What were the exact settings used?
- How many "Days to load" were configured on the strategy tab?
- Are you using the "Break EOD" property set to true or false?
- Are you certain you were using the same exact session template for both the strategies tab and the chart? The session template is a major factor as that will determine if data loads from cache or not
- Does your strategy use any supplementary AddDataSeries() calls?

Finally, can you provide your trace file from the time this occurred? There should be some bar request tracing available which could clue us into the request that resulted in gaps. You can email me directly at matthew.hendricks[at]ninjatrader[dot]com

Thanks in advanced