Historical Data gaps if network error IB NT8 B12

[DaveE]

Setup:
Connect to IB Gateway.
Have a chart: 1 minute USDJPY with a test indicator that has the following code to record connection events:

protected override void OnConnectionStatusUpdate(ConnectionStatusEventArgs connectionStatusUpdate)
		{
			Print(string.Format(" OnConnectionStatusUpdate {0:dd MMM yyyy HH:mm:ss} {1}", DateTime.Now, connectionStatusUpdate));
		}

Steps to reproduce the problem:
break the network connection (eg turn of WiFi) for a few minutes, then turn it on again:

Results:
Output shows:

OnConnectionStatusUpdate 25 Jul 2016 16:03:27 status=Connected priceStatus=ConnectionLost previousStatus=Connected previousPriceStatus=Connected errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:03:34 status=ConnectionLost priceStatus=ConnectionLost previousStatus=Connected previousPriceStatus=ConnectionLost errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:06:27 status=Connected priceStatus=ConnectionLost previousStatus=Connected previousPriceStatus=Connected errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:06:58 status=Connected priceStatus=ConnectionLost previousStatus=Connected previousPriceStatus=Connected errorCode=NoError nativeError=''

Notice that when re-connection is detected the PriceStatus shows ConnectionLost two times. Whereas since the price connection has been re-established I would expect it to show Connecting or Connected the second time.

Looking at the Chart I see the bars ending 16:05 & 16:06 are missing. These bars are also missing from Historical Data database.

If instead I disconnect IB using Control Center/Connections menu, then use the same menu to reconnect a few minutes later I get these results:

OnConnectionStatusUpdate 25 Jul 2016 16:46:17 status=Disconnecting priceStatus=Disconnecting previousStatus=Connected previousPriceStatus=Connected errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:46:17 status=Disconnected priceStatus=Disconnected previousStatus=Disconnecting previousPriceStatus=Disconnecting errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:50:34 status=Connecting priceStatus=Connecting previousStatus=Disconnected previousPriceStatus=Disconnected errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:50:52 status=Connecting priceStatus=Connecting previousStatus=Disconnected previousPriceStatus=Disconnected errorCode=NoError nativeError=''
OnConnectionStatusUpdate 25 Jul 2016 16:50:52 status=Connected priceStatus=Connected previousStatus=Connecting previousPriceStatus=Connecting errorCode=NoError nativeError=''

Here the reconnect does cause OnConnectionStatusUpdate to show priceStatus both Connecting and Connected. It also triggers some undocumented NT function that reloads just the current calendar day's historical data from IB (rather than everything covered by the chart as ReloadAllHistoricalData() would do). This has the effect of filling any minute gaps would have occurred if the disconnect was due to a network error instead.

But this is inconsistent:
Historical Data (minutes) gaps are filled by NT if user disconnects and reconnects IB using the Control Center menu (so long as the disconnect doesn't span a calendar day break).
But these gaps are not filled by NT if a network disconnect occurs (such as broadband downtime) when the reconnect occurs.
The inconsistency might be due to OnConnectionStatusUpdate event not being raised for PriceStatus==Connecting (or Connected) after NT recovers from network error.

[NinjaTrader_ChelseaB]

Hello DaveE,

This behavior is the same as NinjaTrader 7.

Manually disconnecting and reconnecting will cause your strategy to be disabled. Once re-connected this would re-download any missing historical data. A strategy when re-enabled would then restart and completely recalculate all historical data.

When experiencing a temporary connection loss, this would not cause the strategy to be disabled. The strategy would continue running and would not restart as long as the max number of restart attempts have not been hit within the set amount of minutes. The script would then continue on as soon as the connection is re-established. However, any ticks that were missed while the connection was lost, would not be redownloaded. The script also would not be restarted and would not recalculate through all of the historical data.

If you would like we can enter a feature request to have NinjaTrader re-download historical data and disable all working strategies after experiencing a connection loss.
Would you like to submit this as a feature request?

[DaveE]

If you would like we can enter a feature request to have NinjaTrader re-download historical data and disable all working strategies after experiencing a connection loss.
Would you like to submit this as a feature request?

No thanks. What you are suggesting seems to be binding the data load even tighter to strategy operation, whereas I think the data load should be more independent.

Here's an example: I have a strategy running on 60 minute TF. The strategy analysis is based mainly on High and Low information. Strategy uses ConnectionLossHandling.KeepRunning and Calculate.OnBarClose.
If the broadband went down between say 00:05 and 00:55. The current NT will:
a) Fail to raise an event at 00:55 showing priceStatus has (re)connected.
b) Fail to load the missing minute data from IB (even though it could).
c) So when bar closes at 01:00 the High and Low reported will be calculated from only 10 minutes of data instead of 60 minutes and are likely to be misleading.

Note that in this example NT could have done the historical data load of the missing minutes silently without affecting the OnBarUpdate sequence of the strategy. Or if NT raised a reliable event for price reconnection, and provided a method for reloading a range of minutes, then the user could program the data fix themselves.

Of course if the user had been running the same strategy on a 1 minute timeframe then that would be harder to deal with. Currently NT would:
a) Keep the strategy running after 00:05
b) At 00:55 the OnBarUpdates would start every minute again as though nothing had happened. But lets say the strategy relied on SMA of last 10 bars at 0:55 it thinks its looking at SMA for ten minutes ending 0:55 but its actually looking at SMA for ten minutes ending 00:05.

Perhaps a solution lies in this direction:
Regardless of whether a strategy is running. If there's say a chart requesting minute data then when price connection re-connects:
Check for missing bars since the disconnect and reload them if needed from historical data source (if reload is needed then State would need to switch back to Historical while loading these missed bars, then switch back to Realtime.
If there's also some strategy using the same instrument and timeframe it would need to do the same state change.
But if the 60 minute timeframe example strategy above was running it would not need to change its State from Realtime, since the background data loader which is working on a minute basis would have completed the load of missing minutes intrabar as far as the strategy is concerned.

If this was implemented then it would require a small documentation change. For example OnStateChange() page currently says State.Transition is only called once. But presumably if my suggestion was implemented it might be called each time there was a price re-connection.