Adding Historical Expiries, Rollover Dates, Weird

[aventeren]

So I'm in the process of setting up historical expiries for several contracts. I'm currently working on the ES. I'm connected to Continuum and I'm on NT7 Build 25 (most current).

I've discovered an apparent oddity that I wanted to see if there was a reason behind it--or if it's an error. Let's start with just the ES 12-XX contract's rollover dates. When you toggle back through them (i.e., 12-16, 12-15, 12-14, 12-13, etc), you'll see that they all fall on the 2nd Thursday in September, until you get to 12-06, and then the default Rollover Date is 9/7/06. So in trying to figure out why each contract fell on the 2nd Thursday but this one, I found that the apparent reason is that September 1, 2006 fell on a Friday while all of the other September 1st(s) fall on anything but Friday. So it seems weird to me that if a September 1st falls on a Thursday or a Saturday, that the Rollover Date is the 2nd Thursday but if the September 1st falls on a Friday, the Rollover Date is the 1st Thursday. It just seems weird and arbitrary, which makes me think it's an error.

For instance, if September 1st fell on a Sat or Sun, it would make sense that the Rollover Date would be the same as if September 1st fell on a Fri (i.e., first Thursday of the month), but that's not how the Rollover Dates go. The Rollover Date is only on the first Thursday if September 1st falls on Friday.

So I started with the ES 12-XX contracts (because I'm about to add expiries and data back to 2000), but I'm also noticing the same thing with the ES 09-XX contracts--although the Rollover month is then June instead of September--as well as the ES 03-XX contracts--although December instead of September. The ES 06-XX contracts never had a March 1st on a Friday, so I can't validate the same thing is going on with the ES 06-XX contract, but odds are that if there is a March 1st that fell on a Friday that the Rollover Date would be the 1st Thursday in March instead of the 2nd Thursday.

I was seeking consistency but didn't find it, which led me here. Is there a back story here or is this an error? It seems like an error.

Thanks,

Aventeren

[NinjaTrader_Brett]

Hello,

Sure would be happy to fill you in on the back story.

Rollover dates are not an exact science and we've tried a couple of methods on settings them. The reason its not an exact science is because there is no set date where all traders switch over to the next contract. There is generally only a date where the contract expires.

Traders will rollover at some point to the next contract in advance of the contract expiry. It is not possible to guess the date that the majority of the volume will switch over to the next contract in most cases so we have to make some assumptions in some cases past on past data.

Through the years we've made changes to try and more accurately define this date based on what we see in the past. ES contract is one of the easier contracts as there now is clear dates from the CME and this is what we now use to set the rollover date which corresponds to when he majority of volume switches over. This may have been different back in 2006.

You are free to adjust any rollover date in the past as you see fit as well. We generally won't change a previous date unless it is wildly off.

Let me know if I could provide any further information.

[koganam]

So I'm in the process of setting up historical expiries for several contracts. I'm currently working on the ES. I'm connected to Continuum and I'm on NT7 Build 25 (most current).

I've discovered an apparent oddity that I wanted to see if there was a reason behind it--or if it's an error. Let's start with just the ES 12-XX contract's rollover dates. When you toggle back through them (i.e., 12-16, 12-15, 12-14, 12-13, etc), you'll see that they all fall on the 2nd Thursday in September, until you get to 12-06, and then the default Rollover Date is 9/7/06. So in trying to figure out why each contract fell on the 2nd Thursday but this one, I found that the apparent reason is that September 1, 2006 fell on a Friday while all of the other September 1st(s) fall on anything but Friday. So it seems weird to me that if a September 1st falls on a Thursday or a Saturday, that the Rollover Date is the 2nd Thursday but if the September 1st falls on a Friday, the Rollover Date is the 1st Thursday. It just seems weird and arbitrary, which makes me think it's an error.

For instance, if September 1st fell on a Sat or Sun, it would make sense that the Rollover Date would be the same as if September 1st fell on a Fri (i.e., first Thursday of the month), but that's not how the Rollover Dates go. The Rollover Date is only on the first Thursday if September 1st falls on Friday.

So I started with the ES 12-XX contracts (because I'm about to add expiries and data back to 2000), but I'm also noticing the same thing with the ES 09-XX contracts--although the Rollover month is then June instead of September--as well as the ES 03-XX contracts--although December instead of September. The ES 06-XX contracts never had a March 1st on a Friday, so I can't validate the same thing is going on with the ES 06-XX contract, but odds are that if there is a March 1st that fell on a Friday that the Rollover Date would be the 1st Thursday in March instead of the 2nd Thursday.

I was seeking consistency but didn't find it, which led me here. Is there a back story here or is this an error? It seems like an error.

Thanks,

Aventeren

Rollover dates are not controlled by NT. The exchange sets them according to their rules. It might pay to go to the source.

ref: http://www.cmegroup.com/trading/equi...fications.html

[aventeren]

Hello,

Sure would be happy to fill you in on the back story.

Rollover dates are not an exact science and we've tried a couple of methods on settings them. The reason its not an exact science is because there is no set date where all traders switch over to the next contract. There is generally only a date where the contract expires.

Traders will rollover at some point to the next contract in advance of the contract expiry. It is not possible to guess the date that the majority of the volume will switch over to the next contract in most cases so we have to make some assumptions in some cases past on past data.

Through the years we've made changes to try and more accurately define this date based on what we see in the past. ES contract is one of the easier contracts as there now is clear dates from the CME and this is what we now use to set the rollover date which corresponds to when he majority of volume switches over. This may have been different back in 2006.

You are free to adjust any rollover date in the past as you see fit as well. We generally won't change a previous date unless it is wildly off.

Let me know if I could provide any further information.

Thanks, Brett. Clearly on a go forward basis most of us roll when the volume crossover occurs, however in the past all we have are the Rollover Dates for backtests. So I was just trying to understand the Rollover Date methodology as a function of understanding my backtesting data--given that I am using a MergePolicy of MergeBackAdjusted.

Can you confirm that if I change the Rollover Date to something other than what comes packaged in NT that the user edited Rollover Date will trump the NT Rollover Date?

[NinjaTrader_Brett]

I can confirm that changing the roll over date will change it in product. However there is a limitation where on re start of NinjaTrader the value will be re downloaded and it will get reset back to the server held value. To prevent this unfortunately you have to change the date as well by adjusting the date forward or backwards by 1 day.

This is resolved in NinjaTrader 8 which is in development now.

[aventeren]

I can confirm that changing the roll over date will change it in product. However there is a limitation where on re start of NinjaTrader the value will be re downloaded and it will get reset back to the server held value. To prevent this unfortunately you have to change the date as well by adjusting the date forward or backwards by 1 day.

This is resolved in NinjaTrader 8 which is in development now.

Got it. So how far back to the "re downloaded" expiries go? For instance, in my version of NT 7 Build 25, the ES goes back to 03-05. If I add a new 12-04 expiry and add a date, will that date be overridden by a NT 12-04 Rollover Date--even though my Build 25 did not include a 12-04 expiry?

We're kind of in the weeds here, but I want to make sure I don't do a bunch of work for nothing.

[NinjaTrader_Brett]

The server only has back to what you see in product when you first start NinjaTrader and look at the rollovers.

[aventeren]

The server only has back to what you see in product when you first start NinjaTrader and look at the rollovers.

...and you don't have any plans to add additional historical expiries in the future that may overwrite my new historical expiries? At least not for NT7 (I understand in NT8 this won't be an issue).

[NinjaTrader_Brett]

no plans currently.

[aventeren]

03-98 is being added as 03-2098 or 03-3098

Next up on the weird front:

When trying to add a new ES 03-98 expiry, when I type in "03-98" as the "Contract Month" and then click on the drop down arrow to expose the calendar, the date is being read as March 2098 or sometimes March 3098.

Is there a short hand way of naming the new expiries via typing or do we have to click the calendar to enter all of the expiring names (i.e., 03-98 would be 3/1/1998).

Also, I think that I have gone down this road once before, but is there a way to modify the /ProgramFiles(x86)/NinjaTrader 7/db/Instruments.txt or Instruments.srv.txt to add the new expiries? Adding these via the GUI is going to take forever!

For instance, in the Instruments.txt file, there is a line for the ES (and all of the other instruments) that contains the expiries and their Rollover Dates. Is there a way to hack this file or the NT db file to add new expiries?

Thanks,

Aventeren

[NinjaTrader_Brett]

Wow your going quite far back. I might suggest we consider a different approach that might be easier on you.

Can you please please email me at [email protected] with ATTN: Brett and reference this forum post and include your phone number.

I will give you a call tomorrow and discuss some options that might be a better way to go. Just want to make sure you don't do a bunch of un-needed work if can be avoided.

[aventeren]

Wow your going quite far back. I might suggest we consider a different approach that might be easier on you.

Can you please please email me at [email protected] with ATTN: Brett and reference this forum post and include your phone number.

I will give you a call tomorrow and discuss some options that might be a better way to go. Just want to make sure you don't do a bunch of un-needed work if can be avoided.

Done. The GUI interface on this portion of NT7 is, well, bad. Hopefully there is a work around. I've tried modifying the Instruments.txt file, but no dice. There has to be an easier way.

Look forward to your call.

All best,

Aventeren