weird problem with pre 2005 ES data

[qewcool]

Hi

I have pre 2005 ES data that I have imported into ES contracts using NT6.5

I migrated to NT7 and did a DB import from NT6.5

all the contracts seems to be imported fine and i see the contracts under instrument manager.

But NT7 only defines roll over dates for contracts up until 2005. Anything previous to 2005 we have to define the rollover dates ourselves under instrument manager.

Now this happened to me on the 12-04 contract

1) WHILE NT7 is DISCONNECTED

I open a chart for 12-04 and everything is ok. I close that chart.

2) I define the rollover dates under instrument manager for 12-04
i open again a chart for 12-04 and everything is ok. I close that chart.


3) I CONNECT NT7 to zenfire

4) I open again a chart for 12-04.
NT7 seems to not know that the data is already there in the DB and tries to get data from the data servers and an error returns saying the data is not available on the server.
And at the end it displays a blank chart ... it does not use the data that is supposed to be already in the DB
I close the chart

5) I DISCONNECT NT7 from the data server

6) WHILE DISCONNECTED, I reopen a chart for 12-04 and now it is also blank .... seems like NT7 deleted the data that I had ???
I close the chart.

7) I open a 09-11 chart with custom range going from now to the 12-04 contract and the data seems to still be there ... the chart displays fine
I close the chart

8) I restore the DB from a backup that I had before all of this. I restart NT7 with the restored DB

9) after the DB restore and WHILE NT7 is DISCONNECTED
I open again a chart for 12-04 and everything is ok. I close that chart. Seems like the DB restore fixed it and that the DB was messed up somehow.

but now I am pretty sure that if I define the roll over dates, connect to zenfire and open a 12-04 chart the same problem will reappear.


Is this a bug ? how do I solve this ?

the problem seems to start at point 3) and you can see from my description that it does not behave like it should.

Now with the DB restore I have to go and redefine the roll over dates for the pre 2005 contracts but I am afraid that if I do that and connect to zenfire again I will mess the DB and need a restore again.

Thanks for your help.

[NinjaTrader_Jason]

Hello qewcool,

Could you please send me the historical data for the 12-04 expiry so I can test it on my end. Please send it to support[at]ninjatrader[dot]com. Put 'Attn Jason' in the subject header and reference this forum thread. If the file is larger than 3 MB, please use www.filexpressit.com to send me the data.

What End date did you use when you create the ES 12-04 chart?

Please include what rollover date you setup for the 12-04 expiry as well.

[qewcool]

I guess I would have to do an export to get you the data in text form ?

I used an end date of 11/04/2004. From middle October 2004 to middle of november 2004.

Rollover date for 12-04 was 9/9/2004. I left the offset blank.

I will try again and not open a 12-04 chart while connected to see what happens. Just get the chart from the 09-11 contract to 2004.

and another scenario would be to put something in the offset instead of blank so that it doesnt try to get the data from the data server ?

Is there a way to tell NT7 to calculate the offset locally instead of going to get the offset from the data server ?

Thanks

[NinjaTrader_Jason]

Hello qewcool,

You could send me your (My) Documents-->NinjaTrader 7-->db-->minute-->ES 12-04 folder as well.

Please let me know the outcome when you only create an ES 09-11 chart that goes back to 2004.

If no offset exist on the data server, NinjaTrader will calculate the offset locally. Please see the link below under 'Understanding the Misc tab' for more information.

http://www.ninjatrader.com/support/helpGuides/nt7/editing_instruments.htm

Am I correct you do not have historical data for the 09-04 expiry, so offset remains on zero?

[qewcool]

I have sent you a zip of ES 12-04 folder.

I have data for 09-04 but im pretty sure the offset field stays at 0

anything prior to 03-06 has 0 in the field (ie all the contracts in 2005) even though i have loaded a chart going from 2011 to 2004 so dont know why it is not calculating it locally

[qewcool]

anything prior to 03-06 has 0 in the field (ie all the contracts in 2005) even though i have loaded a chart going from 2011 to 2004 so dont know why it is not calculating it locally

i can confirm that the offset for contracts 03-06 to 03-05 (basically the whole year of 2005 and 1st contract of 2006) are set to 0 on the data servers ... I have redone everything and put in manually my own calculated offset and once I connect to the data servers ... those above contracts gets put an offset of 0 can you correct that on the zenfire data servers ?

the manually entered offset for the prior contracts 2004, 2003, 2002 etc seems to not be erased once connected.

I did not risk opening a chart for these past contracts for fear of corrupting my DB again and having to restore. will wait for your input with my sent zip file.

Thanks

[NinjaTrader_Jason]

I will forward an email to development regarding the offset values for expiries before 2006 to include it on our data server.

When I tested on my end (opening a chart for past contracts only), I did experience the error message that the requested historical data was not available, but it did display the historical data I manually imported in the chart.

I suggest to create a backup file for all settings and subsequently test creating such charts. If the issue recurs, you can restore your backup file to correct the issue.

[qewcool]

When I tested on my end (opening a chart for past contracts only), I did experience the error message that the requested historical data was not available, but it did display the historical data I manually imported in the chart.

did you open:
1) a 09-11 contract to 2004 or a recent contract extending back to 2004
2) or the 12-04 contract itself ?

the problem seems to occur in situation 2) but not in situation 1)

can you confirm what you tried ?

Thanks

[NinjaTrader_Jason]

I opened the ES 12-04 contract itself. It displayed the requested data in the chart. I used a custom date range of 10/4/2004 and 11/14/2004.

[NinjaTrader_Jason]

The suggestion to include offset values for expiry month 03-06 and earlier has been assigned to ID# 1047.

[qewcool]

Ok thanks for your help

I dont know why you have a different result ... must be related to the DB somehow.
I imported the values from NT6.5 while I can guess you started from a blank DB from a fresh install of NT7. Dont know if that makes a difference.

i will try again after a backup in the weekend.


for ID1047 : a suggestion for a quick temporary fix would be to not put the offset to 0 on the data servers and put it as blank instead so that the manually entered offset values on the customer end is not overwritten as soon as the client connects to the data server. After that you can take your time to calculate the proper values.

[NinjaTrader_Jason]

Yes, I renamed the database I was using at that time and so created a new empty database. Please let me know if the issue persists in the weekend.

Thank you for the additional suggestion regarding the offset values. I have forwarded it as well.

[qewcool]

for ID1047 : Just had to disconnect and reconnect to the Data servers and I got a bunch of pop up for all contracts prior to 06-06 even though I had no chart open that go back to before 2006.

I also got a bunch of this message in the log tab:
"error on loading chart data for "ES 12-04 Globex" Ninjatrader data server do not support this instrument. no data available."

one message for each of the contract prior to 06-06. each referencing the corresponding contract not the 12-04 contract as in the example.

But I had no chart opened covering any of the periods from these contracts ... why is NT7 trying to load chart data on its own without me opening a chart requiring this data ??



I go to check the roll over offsets that I put in manually and they are all put back to 0 now for all contracts prior to 06-06

I guess the problem is not limited to the year of 2005-2006 only then .... it affects everything prior to the 06-06 contract

I guess I cant enter the offset manually then.

2 questions :

1) I had no chart opened covering any of the periods from these contracts ... why is NT7 trying to load chart data on its own without me opening a chart requiring this data ??

2) What is the logic of absolutely getting the offset values from the data servers ? .... seems like it takes away control from the user and it prevents us from correcting the values when the ones on the data server are wrong ?

Thanks

[NinjaTrader_Brett]

Hello,

Thanks for the forum post:

1) I had no chart opened covering any of the periods from these contracts ... why is NT7 trying to load chart data on its own without me opening a chart requiring this data ??

NinjaTrader would not, you either had a chart open in a workspace in the background or a single chart pulling back a large number of days back to load would cause this. Please close all workspaces under FIle->Workspaces under save workspaces as. Workspaces listed under here are open in the background.

2) What is the logic of absolutely getting the offset values from the data servers ? .... seems like it takes away control from the user and it prevents us from correcting the values when the ones on the data server are wrong ?

This has already been submitted to development with ID:1047. As current setup these older contract will have their values overwritten by the server. Unfortunately you would not be able to use auto merge to go this far back at this time.

Let me know if I can be of further assistance.

[qewcool]

Hello,

Thanks for the forum post:



NinjaTrader would not, you either had a chart open in a workspace in the background or a single chart pulling back a large number of days back to load would cause this. Please close all workspaces under FIle->Workspaces under save workspaces as. Workspaces listed under here are open in the background.





This has already been submitted to development with ID:1047. As current setup these older contract will have their values overwritten by the server. Unfortunately you would not be able to use auto merge to go this far back at this time.

Let me know if I can be of further assistance.

Thanks for the reply

Regarding the 2 points :

1) I am sure I did not have any opened chart going back that far. It could be that it was trying to load the offset to see if they exist on the data servers and thus it should not reoccur.

2) If we are not able to use auto-merge for pre 2006 contracts then is there another solution ? I tried to look for the manual merge feature as in NT65 but that option is now gone. So what else can we use ?

if nothing else is available, then what is the ETA on ID:1047. Just asking because it seems to be a minor change to do (just enter the right offsets for the contracts on the data servers or let us override the "fetch offset from data server" policy for each specific contract).

I can even send you my calculated offsets for the earlier contracts if you want so you dont have to load the data and calculate it.