Importing Historical Daily Data; Adding Expiries and Correct Offsets

[aventeren]

Howdy--

I am going to write an indicator to pull daily futures price data from Quandl, which for some instruments goes back to the 1970s and 1980s--much longer than I am able to get from Continuum. My plan is to import each contract's daily data and then to backtest this daily data. To insure a quality backtest, I've read that I should import individual contracts and then use MergeBackAdjusted, which will apply the correct Offset Value in the creation of the spliced price record. However, if I am going to import the individual contracts, I am going to need to create new expiries that go back further than NT currently lists. For example, CL's expiries and offsets only go back to 2008, however the Quandl daily data goes back to 1983.

I know how to create new expiries via Instrument Manager, but where can I find the correct Rollover Data and Offset Value for each new expiry? Are these published somewhere? Where did NT get them?

Thanks,

Aventeren

[NinjaTrader_Brandon]

Hello Aventeren,

Thanks for writing in.

These settings were all manually created by development. They use their discretion to determine when to rollover based on settlement, historically when most people rollover, etc. There is no list of when to rollover. There is no agreed upon date for rollover. The offset is manually calculated by comparing the difference from the close of the previous contract to the open of the current one. There is no list of what offset to use.

[aventeren]

Hello Aventeren,

Thanks for writing in.

These settings were all manually created by development. They use their discretion to determine when to rollover based on settlement, historically when most people rollover, etc. There is no list of when to rollover. There is no agreed upon date for rollover. The offset is manually calculated by comparing the difference from the close of the previous contract to the open of the current one. There is no list of what offset to use.

So in theory, I could just create a new CL expiry with a Rollover Date of my choosing, and then NT will automatically calculate the Offset Value by comparing the differences between the closing (i.e., settlement) prices?

[NinjaTrader_Brandon]

Hello Aventeren,

Thanks for your reply.

The offset would not be auto calculated. These offsets are manually calculated by development. Newly added expirys would need the offset manually entered.

[aventeren]

Hello Aventeren,

Thanks for your reply.

The offset would not be auto calculated. These offsets are manually calculated by development. Newly added expirys would need the offset manually entered.

Can you please provide a manual calculation example that syncs up with the CL 03-14 expiry, which I should then be able to use in an auto calc method?

[NinjaTrader_Brandon]

Hello Aventeren,

Thanks for your reply.

One day prior to the rollover date, calculate the difference between the close price of the new expiry and the close price of the old expiry. This is the Offset value.
Since the CL 03-14 rolled over on Jan 17th you would look at the close price from the 16th on the 03-14 & 02-14 and compare the difference.
You can check the close prices in NinjaTrader under Tools> Historical Data Manager> Edit

Additionally I was incorrect and there are times the offset will auto calculate:
"Although NinjaTrader will attempt to download the Offset values from the data server, if they do not exist on the data server, they will be calculated locally. Offsets are only downloaded when the "Offset" field is left blank and the rollover date matches the date defined on the server.

When NinjaTrader will calculate the Offset value locally:

• The Offset field in the Contract Months window is blank
• Historical data exists in the database for both the new and old contract near the rollover date
• The MergeBackAdjusted policy must be selected in the Data tab of the Options menu
• You must be connected to your data provider and requesting data for the instrument

How NinjaTrader will calculate the Offset value locally:

• Use the old and new expiry’s daily price data for calculations
► If daily data does not exist, use minute data
► If minute data does not exist, default Offset value will be 0
• One day prior to the rollover date, calculate the difference between the close price of the new expiry and the close price of the old expiry. This is the Offset value.
► If you wish to overwrite the calculated Offset value you can input in your own
► When using minute data, the close price at the ending time as defined in the default session template for the instrument will be used

Note: If you inputted your own Offset value, it will be overwritten by values downloaded from the data server if it exists there. To prevent this you will need to ensure that your rollover date is not the same as the ones coming from the data server.

Note: The rollover date is the date to roll into the selected contract month and NOT out of."
Understanding the Misc tab - Offset Value: http://www.ninjatrader.com/support/h...nstruments.htm

[aventeren]

Hello Aventeren,

Thanks for your reply.

One day prior to the rollover date, calculate the difference between the close price of the new expiry and the close price of the old expiry. This is the Offset value.
Since the CL 03-14 rolled over on Jan 17th you would look at the close price from the 16th on the 03-14 & 02-14 and compare the difference.
You can check the close prices in NinjaTrader under Tools> Historical Data Manager> Edit

Additionally I was incorrect and there are times the offset will auto calculate:
"Although NinjaTrader will attempt to download the Offset values from the data server, if they do not exist on the data server, they will be calculated locally. Offsets are only downloaded when the "Offset" field is left blank and the rollover date matches the date defined on the server.

When NinjaTrader will calculate the Offset value locally:

• The Offset field in the Contract Months window is blank
• Historical data exists in the database for both the new and old contract near the rollover date
• The MergeBackAdjusted policy must be selected in the Data tab of the Options menu
• You must be connected to your data provider and requesting data for the instrument

How NinjaTrader will calculate the Offset value locally:

• Use the old and new expiry’s daily price data for calculations
► If daily data does not exist, use minute data
► If minute data does not exist, default Offset value will be 0
• One day prior to the rollover date, calculate the difference between the close price of the new expiry and the close price of the old expiry. This is the Offset value.
► If you wish to overwrite the calculated Offset value you can input in your own
► When using minute data, the close price at the ending time as defined in the default session template for the instrument will be used

Note: If you inputted your own Offset value, it will be overwritten by values downloaded from the data server if it exists there. To prevent this you will need to ensure that your rollover date is not the same as the ones coming from the data server.

Note: The rollover date is the date to roll into the selected contract month and NOT out of."
Understanding the Misc tab - Offset Value: http://www.ninjatrader.com/support/h...nstruments.htm

Amazing response. Thank you very much.