Trouble Merging/Rolling futures

[zoltran]

I'm trying to merge the June futures contracts into the new Sept expiry and am having some problems.
Using YM as an example..
I have YM 06-07 data back to Mar 8th.
I have YM 09-07 data as of Jun 8th
I have both in instrument manager.

When I try to merge .. I get messages such as 'No Day Data, No Minute Data, No Tick Data .. etc,
Followed by 'Minute Data, Tick Data successfully merged ...'

However, the new contract doesn't have any new data in it at all.
I've tried various combinations of to/end dates with no luck

Screen shots attached.

[NinjaTrader_Ray]

If you open a YM 09-07 1 chart right now with 15 days back, what do you get on the chart?

[zoltran]

Hmmm
It seems to be working now
You have to reset the chart (open/close or switch to another instrument and back)

But I was pretty sure I'd already done that...

Anyways, no worries now.

(this could be an easier process though ... It wouldn't be hard to pre-populate the some of the fields in the merge dialog. For example, if I select 'ES 06/07' it would be nice if NT pre-populated the target with the ES 09/07. The roll-over info is in the instrument mgr. )

[yeneuro]

I've tried to merge 06/07 contracts into 09/07 for ym es and er2 without success.

For ym 09/07 I only get the 09/07 data
For er2 09/07 I only get the 09/07 data
For es I do get the merged data but data for the day of 06/07/2007 is missing enen though for range settings/days back is set to 15


Any ideas ???

[NinjaTrader_Dierk]

Let's start simple:
- say you have 1 minute data on YM 06-07 and YM 09-07 and you want to merge YM 06-07 to YM 09-07
- export YM 06-07, minute data (Tools->Historical data export) to file #1 and inspect the file is data really is there on a given day (e.g. 2006-06-01)
- export YM 09-07, minute data (Tools->Historical data export) to file #2, inspect 2006-06-01 and see if the data there is different or if there probably isn't any data at all
- now merge YM 06-07 into YM 09-07 with From date < 2006-06-01 and To date > 2006-06-01 (Tools->Historical data->Merge)
- now again export YM 09-07, minute data (Tools->Historical data export) to file #3, inspect 2006-06-01 and see that the data matches file #1 data on 2006-06-01

-> merge was successful

[zoltran]

Trying to roll currencies today. Starting with 6C 0607 to 6C0907.

6C 0907 (sept) already has been collecting data .. ie it was up on a MA and has data from 05/23 in it .. (this is the thin, pre-roll-over data, and it should be replaced by a merge with the prior contract month).

March .. 6C 0607 has good data back to March

So far ..
I've tried merging ... no luck. Various error msgs stating there isn't any tick/minute/day data for the 'to' security. See screenshots in my original post.

Opened new charts, restarted NT .. but the only data in 6C 0609 is the original thin data I collected.

Tried the export/import trick ...
- Export 6C 0607 - Verified that there is valid data back to March
- Imported into 6C 0907 .. Imported ok
- But 6C 0907 still shows the old data

Doesn't matter if I open/close a chart, or even restart NT.

So .. maybe it's going someplace, who knows, the whole NT database/intra-day storage is a mystery.

fwiw, ..I tried to upload the export file, but the forum wouldn't let me.. my 340K Zip file exceeded some limit.

[NinjaTrader_Dierk]

>>> - Imported into 6C 0907 .. Imported ok
Importing won't do it. You need to Tools->Historical Data->Merge

Please retry and proceed exactly as per my previous post. If problem persists:
- please PM me both data files file #1 and file #2 to "dierk AT ninjatrader DOT com"
- which concrete date should I look into?

[NinjaTrader_Dierk]

(deleted my prior post): We revisited the issue and came to the conclusion that existing merge behavior is correct: You need to have some data on the "To" series after the merging date, since there will be an offset calculated and the "From" series is merged. The offset logic to "equal out" different price levels on different contract expiries.

However, we will provide a clearer message and adjust the documentation.

[zoltran]

OK retried with YM.
Merged YM 0607 to YM0907 .. from 03/12/07 up to last friday 06/08/-7
That is the range of my existing ym 06/07 contract.

Only this time .. NT is doing one of it's extraneous data retrievals ..
and it's downloading data from opentick for both contract, even though there is data already there. Disconnect opentick and retry .. then merge completes correctly.

This further underscores my thoughts that many things in NT are best done WITHOUT a historical dataprovider connected.



I've attached a screenshot showing the download message.

Also attaching a screen shot showing the msg-dialogbox corruption after performing multiple merges.

If you're in changing things .. it would be nice if the target symbol name defaulted/sync;d to the source name.

[NinjaTrader_Dierk]

Can't read images attached, since they are too small.

[zoltran]

That's very odd, as the originals are larger/normal size. Looks as though something on the forum software altered them, but that doesn't make sense either.

Anyways, I've zipped and reuploaded the one showing the way messages 'stack' as you merge several symbols.

[NinjaTrader_Dierk]

Thanks for posting bigger images. This is a bug which will be fixed with next update. Thanks for reporting this one.

[JeffGus]

merging more than two data sets

I have been able to roll one contract.....into the other. That is no problem. So march into june on the ES, ER2 markets. How do I roll more past data into this merged data of 6 months? I would like to get at least a year of data. I tried to Merge dec into march than merged that into June and it stuck on window that says : Loading tick bars data.."

How do we get one two two years of data rolled together. Also I have tick data thet we purched. Can I use that data file to run test within NT.

[NinjaTrader_Dierk]

It might take very long to merge tick data. Don't forget the ES has more than 100,000 ticks per day.

Also: You will run into memory issue if you try to merge tick data for one year in one step, splitting it up into several steps should help.

[JeffGus]

Using the merge tool.

How do I get this done. I want to merge the dec contract with march.

using er2 data .........On the monthly drop down window.....is this considered the 3-07 contract eventhough is is really 12-06 merged with 3-07.

I than did 3-07 with 12-8-06 date to 3-8-07 data into the June Contract and it ran for 15 min with no results. Is that the proper way to roll 6 months of data into the june contract and than use the same process into the current contract.

When I complete that process eventually can I cal or save that data file to access again and not have to go through this process?