Historical Data Manager not working as expected

[Harry]

I just tried to download data for the new contract ZB 03-17 via the Historical Data Manager. Selected the dates from 19 Nov to 3 Dec 2016.

Expected:

NinjaTrader 8 downloads contract data for ZB 03-17 from 19 Nov to 3 Dec.

Result:

NinjaTrader 8 downloads contract data for ZB 03-17 from 28 Nov to 3 Dec.
NinjaTrader 8 downloads contract data for ZB 12-16 from 15 Nov to 29 Nov

This was totally unexpected.

Also this means that in order to get my historical data, I need to

- change the global options from "MergeBackAdjusted" to "DoNotMerge"
- download the contract data as needed
- then change the global options back from "DoNotMerge" back to "MergeBackAdjusted"

Now if I download data for ZB 03-17 from Nov 19 to Dec 3 with the global option "DoNotMerge" things are not simple either.

Expected:

NinjaTrader 8 downloads contract data for ZB 03-17 from 19 Nov to 3 Dec.

Result:

NinjaTrader 8 downloads contract data for ZB 03-17 from 29 Sep to 3 Dec


Second guessing means bad design

Second guessing always leads to misinterpretation and disaster. The best example was the heuristics introduced for NinjaTrader 7 to determine which of the intraday sessions should be selected as the last session of the trading day.

For the historical data download there is a pretty clear interface showing that you are downloading data for the contract ZB 03-17 from 19 Nov to 3 Dec 2016. But this is not what NinjaTrader does!

Either it downloads data for two different contracts with a two day overlap. Or it downloads data from September to December for the selected contract.

As a user I simply expect that it does what it pretends to do. I do not need a super-intelligent tool that second-guesses my intentions and knows better than me. This approach is bound to fail.


Suggestions

(1) User interface should in any case show what it is doing and not do something different.

(2) Please separate the global data option from the data option selected for the Historical Download Manager.

Minimum requirement: Historical Data Manager shows whether it is downloading single month contract data or merge backadjusted contract data.

Best solution: Let the user choose via the Historical Data Interface (shown below), whether single month data or mergeback adjusted data shall be downloaded.

[NinjaTrader_ChelseaB]

Hello Harry,

I am unable to reproduce these results.

Below is a link to a video I have taken to compare.

http://www.screencast.com/t/u498uCnYe

May I confirm you are using the latest version of NinjaTrader 7 (7.0.1000.32) and NinjaTrader 8 (8.0.1.0)?

Are you connected to the same data source in NinjaTrader 7 as NinjaTrader 8?
(One at a time is fine)

Are the rollover dates for NinjaTrader 8 the same as in NinjaTrader 7 for the 03-17 and 12-16 contract months?
May we have screenshots to demonstrate this?
NT7 - Tools -> Instrument Manager... -> search zb -> select and click Edit
Select the Misc tab -> click the value -> "..." ellipses button for Contract months
Select 12-16 -> take a screenshot
Select 03-17 -> take a screenshot

NT8 - Tools -> Instruments -> search zb -> select and click edit
Click the value -> "..." ellipses button for Contract months
Select 12-16 -> take a screenshot
Select 03-17 -> take a screenshot

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.

Click here for instructions

Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screen shot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.

Click here for detailed instruction

Do you see any errors appear on the log tab of the Control Center?

[Harry]

Hello Chelsea,


Thank you for your detailed answer.

I was obviously wrong in stating that NinjaTrader 7 works in a different way than NinjaTrader 8. It seems that NinjaTrader 7 does the same thing without that I have ever discovered it. (I have now removed two sentences from my original post referring to NinjaTrader 7, because the statement that NinjaTrader 7 works in a different way is false. I hope you do not mind, but it makes it simpler to grasp the problem for future readers).

If you look at the screenshot below (taken from NinjaTrader 7), I have always been convinced that when selecting a contract and a range of dates NinjaTrader would actually download contract data for that range of dates.

In fact it does not do that!

I am myself an example to what extent this is confusing. I have assumed for years that NinjaTrader 7 would do what it pretended to do, just download contract data. There is no hint on that menu that it would do otherwise than shown.


Why is this a problem?

The problem is only there, if you wish to change rollover dates at a later stage. In this case there will be data gaps which can no more be closed, as the historical backfill is typically limited to a set number of days, such as 180 days for tick data.

For example right now, I have checked the rollover dates for a few contracts and found that the contract ZL 01-17 is missing for NinjaTrader 7, while it has been included for NinjaTrader 8. I have no idea why this contract is missing. I certainly did not delete it, as I have never added or removed any of the agricultural contracts.


Why did I think that NinjaTrader 8 works in a different way?

NinjaTrader 8 actually shows the download progress in the Historical Data Window. NinjaTrader 7 shows the information in a status bar in the Control Center, which is mostly hidden or elsewhere. This made it possible for me to understand that NinjaTrader 8 downloads data different from what is shown and from what I had expected. I have then checked the results by editing historical data and was asthonished.

It would be good to mention that the downloaded date depends on the global settings for the data options.

[NinjaTrader_PatrickG]

I understand your point about the potential of missing historical data once it is no longer available on the historical data servers and then trying to use different rollover dates. Also, there could be documentation that states the merge policy affects the historical data download behavior.

I'll be submitting your comments to the Development Team as a feature request. Thanks for your feedback.