Environment:
I am using IB Gateway 954 (dont know if problem affects other connections).
Options in use:
MarketData/Historical: Get data from server=true
MarketData/Realtime: Filter Bad ticks = true, % off=0.1, live as historical=true.
Steps to reproduce the problem
1) To keep it simple use a new instrument. I chose USDMXN for this test since I had no previous historical data.
2) Create some minute data in Historical Data. I opened a new chart for USDMXN (Bid, minute, 60, load 5 days).
3) Tools/Historical Data: Historical/USDMXN/Bid/Min/November 2016
I see populated rows for 1Nov to 4th and 6&7th Nov. For example 4 Nov shows "1260 items".
4) Now pretend there is some missing minute Bid data for 3 Nov and you want to reload it. Go to Load tab, Instrument=USDMXN (interval=minute), select Bid only, Dates 3Nov to 3Nov, press Download.
5) Back in Edit tab check the number of rows in each date under Historical/USDMXN/Bid/Min/November 2016. Serious bug: the load for 3Nov has deleted all but 1 rows for the following date. So there is just 1 row left for 4 Nov (time=00:01).
If you try to fix this by loading data for just 4 Nov you find that all seems OK but that was just luck, the loader probably tried to delete rows in the following date but there were none because its a Saturday.
This problem does not just affect the Historical Data form. It also affects automated data loading. For example if I have a strategy that adds an Ask data series to the primary Bid series then NT may decide for some reason that there is a gap in Ask minute data on a certain day in the past and just tries to get that date. But in the process it deletes Historical Data for the following date!
Same problem occurs for Ask minute data.
One result of this problem is that each time an affected strategy is run requiring Historical Data, large data gaps may shift around in an apparently random way as NT tries to fix one problem but makes things worse somewhere else.
I am using IB Gateway 954 (dont know if problem affects other connections).
Options in use:
MarketData/Historical: Get data from server=true
MarketData/Realtime: Filter Bad ticks = true, % off=0.1, live as historical=true.
Steps to reproduce the problem
1) To keep it simple use a new instrument. I chose USDMXN for this test since I had no previous historical data.
2) Create some minute data in Historical Data. I opened a new chart for USDMXN (Bid, minute, 60, load 5 days).
3) Tools/Historical Data: Historical/USDMXN/Bid/Min/November 2016
I see populated rows for 1Nov to 4th and 6&7th Nov. For example 4 Nov shows "1260 items".
4) Now pretend there is some missing minute Bid data for 3 Nov and you want to reload it. Go to Load tab, Instrument=USDMXN (interval=minute), select Bid only, Dates 3Nov to 3Nov, press Download.
5) Back in Edit tab check the number of rows in each date under Historical/USDMXN/Bid/Min/November 2016. Serious bug: the load for 3Nov has deleted all but 1 rows for the following date. So there is just 1 row left for 4 Nov (time=00:01).
If you try to fix this by loading data for just 4 Nov you find that all seems OK but that was just luck, the loader probably tried to delete rows in the following date but there were none because its a Saturday.
This problem does not just affect the Historical Data form. It also affects automated data loading. For example if I have a strategy that adds an Ask data series to the primary Bid series then NT may decide for some reason that there is a gap in Ask minute data on a certain day in the past and just tries to get that date. But in the process it deletes Historical Data for the following date!
Same problem occurs for Ask minute data.
One result of this problem is that each time an affected strategy is run requiring Historical Data, large data gaps may shift around in an apparently random way as NT tries to fix one problem but makes things worse somewhere else.
