Rollover dates and offsets

[Harry]

Need to start ths subject again, as there has been little progress over the last years. I am sorrry. Will post here.


Looking at Nasdaq futures NQ 06-18.

Possible options to calculate the rollover offset are:

Offset option 1 (calculated from settlement prices on day prior to rollover date):
June = 6949.75 / March = 6924.50 offset = 25.25 points

Offset option 2 (calculated from regular closes at 3:15 PM CT on day prior to rollover date):
June = 6949.50 / March = 6923.00 offset = 26.50 points

Offset option 3 (calculated from last traded price on day prior to rollover date):
June = 6947.50 / March = 6923.50 offset = 24.00 points

Offset option 4 (calculated from the difference of the futures contracts at the time when the cash market closed on day prior to rollover date)
June = 6955.50 / March = 6930.00 offset = 25.50 points


Questions:

I personally prefer using option 1, but agree that other methods are possible. Here are my questions

(1) Why does NinjaTrader 8 suggest an offset of 22.5 points? None of the methods results in such an offset.

(2) I do not want to use the offsets suggested by NinjaTrader. I want to use a correct offset with a correct rollover date.How may I enter the correct rollover date March 8 and the correct offset of 25.25 points for display of merge back adjusted contracts?

I need to enter March 8 as otherwise my volume indicators will not work correctly. I also need to enter 25.25 points, as otherwise my support & resistance indicators do not work correctly.

However, when I enter correct values, NinjaTrader will override these values with false values.

What shall I do to enter the values needed?

[NinjaTrader_RyanS]

Hello,

Thank you for posting.

NinjaTrader 8 calculated the rollover based on the close price of minute bars the day before rollover, which results in the 22.5 value that you are experencing.

If you wish to change the offset value, you can do so with the instructions below:

-Go to tools>Instruments and search for and select the NQ instrument and click edit
-Scroll to the bottom and mouse over the Contract months and click edit rollovers
-Select the appropriate contract 06-18 and edit the offset value and press ok

Please let me know if you have any further questions.

[Harry]

Hello Ryan,

Thank you for your answer.

This explains how that offset was calculated. The method certainly does not make sense, as the markets are not liquid at 4:00 PM CT. Therefore any offset calculated from those random values at the end of the day does not reflect the real contract offset during the liquid trading hours. Please compare

difference calculated from VWAPs : 25.0 points
difference calculated from settlements 25.25 points
difference calculated from regular close 26.50 points
difference calculated from random values end of day 22.5 points

The problem with the random values end of day is that they are not concurrent, as they do not have the same time stamp. Even CME does not publish them, but rather shows the last bid or ask, when there has been no trade. Therefore the offset provided by NinjaTrader is simply unusable. You may not allow data like that for any backtest.

Also I am unable to change the rollover offset as you suggested. Although it works temporarily, NinjaTrader will kindly override the correct offset with the nonsense value after a restart.

Not only that NinjaTrader 8 calculates pretty false offsets from random data. When I try to change those offsets, NinjaTrader 8 will override my data with the false data again after every restart.

Hello,

Thank you for posting.

NinjaTrader 8 calculated the rollover based on the close price of minute bars the day before rollover, which results in the 22.5 value that you are experencing.

If you wish to change the offset value, you can do so with the instructions below:

-Go to tools>Instruments and search for and select the NQ instrument and click edit
-Scroll to the bottom and mouse over the Contract months and click edit rollovers
-Select the appropriate contract 06-18 and edit the offset value and press ok

Please let me know if you have any further questions.

[Harry]

Here is the detailed proof, why your method for determining rollover offsets cannot work.

Let us have a look at the last transactions in NQ 03-18 and NQ 06-18 prior to the close of the trading day.

NQ 06-18: There were only 10 trades during the last 15 minutes. Here they are:

07.03.2018 22:59:33.492 6946,00 1 - this is an apple
07.03.2018 22:58:28.441 6945,25 2
07.03.2018 22:57:11.950 6947,75 2
07.03.2018 22:56:22.208 6947,25 1
07.03.2018 22:55:57.272 6946,00 1
07.03.2018 22:54:58.271 6948,00 1
07.03.2018 22:53:49.968 6947,00 1
07.03.2018 22:51:24.697 6948,75 1
07.03.2018 22:48:15.326 6948,25 1
07.03.2018 22:44:49.146 6948,25 2

NQ 03-18: These are the last 15 trades for the more liquid front month contract:

07.03.2018 22:59:59.300 6923,50 1 - this is an orange
07.03.2018 22:59:58.240 6923,25 1
07.03.2018 22:59:45.882 6922,75 2
07.03.2018 22:59:45.086 6922,00 1
07.03.2018 22:59:45.086 6922,00 1
07.03.2018 22:59:45.086 6922,25 1
07.03.2018 22:59:45.086 6922,25 3
07.03.2018 22:59:45.061 6922,25 1
07.03.2018 22:59:45.061 6922,25 3
07.03.2018 22:59:34.026 6922,75 1
07.03.2018 22:59:34.026 6922,75 1
07.03.2018 22:59:34.026 6922,50 1
07.03.2018 22:59:34.026 6922,25 1
07.03.2018 22:59:33.493 6921,50 1 - this is an apple
07.03.2018 22:59:30.121 6921,75 1

You will probably notice that the last trade in NQ 06-18 occured at 22.59:33. This trade is then compared to the trade in NQ 03-18 that occured at 22:59:59. Effectively your method takes two trades that were 26 seconds apart and compares the prices. As the market has moved during 26 seconds to a different level, you are comparing oranges and apples.

The correct way of comparing trades would be comparing two trade prices at the same time. If you did this you would have compared the transaction NQ 03-18 at 22:59:33.493 with the transaction of NQ 06-18 at 22:59:492. The price difference would then become 6946.00 - 6921.50 = 24.50, not a perfect result but already close to the actual offset.

This shows that you are using a method that produces erroneous results taking deferred trade data from an illquid market in the after hours. Please use settlement prices - they are averaged and thus error compensated -, or in case that you do not have them available, use price data with similar time stamps during liquid market hours.

The way you built offsets with NinjaTrader 7 was good enough. Why are you introducing new problems? And as a user I cannot even manually adjust the offset to get rid of the bad data.

[NinjaTrader_RyanS]

Hello Harry,

Thanks for the response


The offset in NT8 is calculated from the difference in each contract's last tick price for that day. It takes the last tick price of the 06-18 contract and the last tick price of the 03-18 contract and calculates the difference. I've attached a screenshot which illustrates how this value is determined (see below).

When reviewing the data on both instruments on a single tick chart I found the following data, which supports the data:
Last tick on 06-18 at 14:59:59.300 at 6946 on 3/7/2018
Last tick on 03-18 at 14:59:59.300 at 6423.5 on 3/7/2018

It would be expected that the offset would revert back after a restart, as NinjaTrader pulls this number from the data servers upon startup of the platform. You can manually change it when the platform is loaded for testing/charting, but there is no way to make this a permanent change at the moment.

If you would like, I can submit this as a feature reqeust for consideration in future releases.

Please let me know if you have any further questions or concerns.

[Harry]

Hello Ryan,

I understand how the offset is calculated as I have already explained it below.

Unfortunately the data that you have cited is taken from a bad source or simply invented.

Please do not tell me any stories. I use Kinetick data feed which is reliable. There is no last tick for NQ 06-18 at 14:59:59.300. The last tick which printed at 6946.00 already came in at 14:59.33.492. This was 26 seconds earlier.

Why do you maintain a position that can not be maintained? Please have a close look at what you are doing. They way the offsets are determined is simply false.

If you want to offer a professional product to your customers

- you would need to acknowledge and correct your mistakes and not defend non-defendable positions (such as a visibly false method for calculating offsets)
- you would also need to allow the user to customize the product, if this necessary to correct the false data supplied via your servers

Are you aware that I cannot run a single correct backtest on NinjaTrader 8 für futures contracts, if i use S/R and volume indicators which I do?

What shall I do to correct the bug? I have contract data for 20 futures contracts for 10 years. Do I need to enter the rollover offsets for 800 contracts every morning?

[NinjaTrader_RyanS]

Hello Harry,

Thanks for the reply.

I have attached two screenshots of a 1 tick chart for both the march and june contract of NQ, using Kinetick as a data source. I have the merge policy set to 'do not merge' to ensure only that contract's data is being displayed. I receive the same last tick price that the offset calculation uses.

Can I confirm if you see the same with a 1 tick chart using Kinetick? If not, there may be something else causing the discrepancy in data.

However, I would also note that the offset between two months is dynamic, and won't have an exact value until settlement at expiration. So how you calculate the offset before expiration can vary between data providers, software, and individual traders.

As you noted, there are many ways to calculate this offset, and none are necessarily better or worse than the other. NinjaTrader calculates it in the manner I illustrated, which is neither better or worse than the methods you described. Additionally, there is not a specific roll date for contracts, as this is up to traders. So if you were to calculate the offset on 3/8 instead, you would reach a different conclusion for the offset. Again, neither is a wrong way of doing it.

I have submitted a feature request to be able to manually change the offset between months and have it save that way when restarting. I believe this would be the best way to implement the behavior you are looking for.

Please do not hesitate with any other questions or concerns.

[Harry]

Hello Ryan,

Thank you again for your patience.

However, you charts do not show anything.

Please open the data box and check the time stamp of that last tick for NQ 06-18.You may also check the time stamp by editing the historical data base.

For NQ 06-18 there is no tick at 14:59:59.300. The last tick plots at 22:59:33.492.. Please have a look at the chart attached.

As a side note: Please have a look at the green bar on the chart. It belongs to the old session, but it is plotted after the session break. This looks like another NinjaTrader bug.


Dynamic offset:

I agree that the offset is dynamic. However, for futures there is something called the fair value. The fair value depends on the stock price and interest rate levels between the two expiries. Fair value is used for arbitrage, and therefore the correct offset moves inside a channel around fair value.

The crime that NinjaTrader developers have committed is that they compare two trade prices with different time stamps. If I do this the resulting offset is no longer inside the channel, but it may lie outside. If you wish, I may take random time stamps over the entire day and calculate the spread between NQ 03-18 and NQ 06-18 based on real transactions. If I do that I will get values close to the difference between the settlement prices for the two contract months. In my first post I had already performed different calculations based on VWAP, settlement, regular close.

From these calculations you may estimate that the offset is around 25.25 +/- 1.25 points. The arbitrage channel would therefore lie between 24 and 26.5 points. However, the value supplied by NinjaTrader comes out too low. It is below the lower bound of the arbitrage channel. It should lie inside!

I repeat the crime that has been committed:

-> The offset is calculated during after hours when the new front month is illiquid
-> The offset is calculated from two transactions that were 26 seconds apart

This can't work.

[NinjaTrader_RyanS]

Hello Harry,

I've included a screenshot showing the simultaneous price of the march and june contract today at a random time. It shows the offset value to be 22.5 which happens to be exactly what NT8 has, though I'll concede this was luck in timing that it was exact. However, it does illustrate that the offset value can vary fairly significantly. Today if using any of the methods you provided, those would seem inaccurate. However, yesterday yours could have been considered more accurate.

All this being said, I don't believe the offset discrepancy should adversely affect your backtesting or charting.

Could you further elaborate on the issue this is creating, so I may try and find a solution or alternate way to achieve the results you need?

Thank you for your patience-

[Harry]

Hello Ryan,

Hello Harry,

I've included a screenshot showing the simultaneous price of the march and june contract today at a random time. It shows the offset value to be 22.5 which happens to be exactly what NT8 has, though I'll concede this was luck in timing that it was exact.

Sorry, there is no screenshot.

All this being said, I don't believe the offset discrepancy should adversely affect your backtesting or charting.

The issue has a direct impact on all plots that overlap with the prior period. For example

- the weekly pivot calculated for the prior week will be false as some of the input values are false
- the monthly pivot from Feburary will not be correctly shown today, but it will be three points off
- the monthly VWAP that acts as support will be false
- the quarterly VWAP calculated from the beginning of this year will be false

All these indicators will show false values, when the offsets are not correctly determined. NinjaTrader becomes useless as a tool for professional charting.


Why at all, do we need to discuss this?

NinjaTrader 7 has shown correct values since 2010 and it still does!

Why do you change a winning concept and start supplying off-market values?

[NinjaTrader_RyanS]

Hello Harry,

I have attached the screenshot again as it appears it didn't attach properly, my apologies.

I believe the best way to solve your problem would be to customize the offset to your desired value, as expressed earlier.

While you noted it does revert when you restart, you can manually change the rollover date, and adjust the offset to make this change permanent. The rollover date can be adjusted in the same place as the offset.

If you do not change the rollover date, it will revert to the original offset value on every restart, so please ensure this is done.

You may want to recalculate the offset based on the new rollover date, but this would be at your discretion.

Be sure to save your workspace after doing so.

Thanks again for your patience and your input on this matter. If we can be of any further assistance please let us know.

[Harry]

Hello Ryan,


I am trying to help to communicate customer dissatisfaction with your product on behalf of several customers.

I wanted to communicate two things

- that your method of calculating offsets is flawed and results in false charts and indicator values

- that the user cannot change the false offset without changing the rollover date


You are just defending your product, unwilling to go ahead and communicate the necessary changes to NinjaTrader development.

NinjaTrader 8 is not working correctly.

Today it is not even possible to enter correct rollover date and offset manually.

What are you going to do about that?


For further evidence I have added a chart with the 1-second spreads calculated from NQ 03-18 and NQ 06-18 over the last 15 minutes of regular trading on Wednesday March 7. As you will notice the spread oscillates around 25.5 and is mostly contained within the arbitrage interval between 24 and 26.5.

The spread has never even come close to the value of 22.5.

Why don't you simply admit that you have got it wrong and make it better!

[NinjaTrader_RyanS]

Hello Harry,

I've noted your displeasure with the current offset calculation, and have submitted a feature request on your behalf to be able to change the offset value without changing the rollover date. Once I receive a tracking number for this request, I will update you with this info.

I understand your frustration with the way the value is calculated, but please note this is a subjective matter, and using a different method may adversely affect other users' indicator calculations, as they may want to use the calculation method already in place.

I believe the best resolution for all parties is to be able to manually adjust the offset value without changing the roll date, so hopefully this can be implemented in future releases.

However, the current solution would be to change the roll date and adjust the offset. Is this method not working for you? I have verified this works on my end. I would be sure to save your workspace after doing so.

I look forward to hearing from you.

[Harry]

Hello Ryan,

I am indeed frustrated. The matter is not at all subjective, because the method that is in place results in false offsets. I have taken all my time to explain that it does not work and the spread chart also proves it visually. You are not listening and pretending that there are users that need false offsets.

Now you tell me that in order to fix the offset I need to change the roll date. However, if I change the roll date and enter the correct offset, this will result

- in correct prices
- but false volume

If I keep the rollover date this will result

- in false prices
- but correct volume

The user is caught between the devil and the deep blue sea.


Example for false volume:

I am not going to show it for NQ, as I do not want to change the rollover date to a false date. I am taking FDAX, because you have already preset a false rollover date, making it easier for me to show evidence for false volume.

For FDAX the preselected roll date is today, March 15. The correct roll date would be tomorrow, March 16. Now please have a look at the chart, and how this affects volume!

Today's volume seems to be a meagre 32% of average volume. Why is this? Easy answer. You have selected the false front month contract. FDAX rolls on Friday. If you had selected the correct front month, today's volume would have been nearly twice as high as the volume shown on the chart.

This shows the problem I am having with the customization of NinjaTrader 8.

Nota: With FDAX you got it totally wrong. This is also good because in that case I may enter both rollover date and offset manually as needed without NinjaTrader overriding the correct values.

[emuns]

Hello Ryan,

I must agree with Harry. The way how the offset is calculated in NT8 is very awkward. I really don´t understand how you can say, that it has no impact on charting or backtesting?! On the contrary, the impact is absolutely crucial!

You suggested to submit feature request to be able to change the offset value without changing the rollover date. It could be fine but I don´t understand why should I be penalized for my effort to get "correct" values. Let´s say that I would like to backtest some strategy. I would like to do it on more futures markets on longer history. So I would need to find the "correct" offsets for many markets and many rollover days. Than calculate it, and than manually change NT default settings… It is really Sisyphean task!

There should be some settings in, let´s say, Tools > Options > General, where would be "Offset calculation" option. There would be drop down list with possible calculations settings. If you don´t like it, suggest another solution. Or change the default settings.

And I have understanding on Harry´s insisting to not to take it as a feature request. It is a request for posibility to easily change settings which is at least very questionable now.

What is the offset settings in NT7 and why you changed it?