Merged contract not charted correctly with minute timeframe?!

[alchymist]

(NT Version: 6.5.1000.6)

When merging ES 09-08 into ES 12-08 I found the following:
(Also applicable to other contracts)

Merging procedure starts and terminates correctly.

Charting the merged contract (which should reach back now to 13th june) with a minutes timeframe (e.g. 60 min) doesn't work correctly.

The chart picture seems to be drawn correctly between 13th june and 27th june. (candles and volume)
Then from 30th june to 10th september there seems to be displayed the original ES 12-08 (as it looked before merging) and from 11th sept onwards the chart looks ok again.

Now something strange:

If the same merged contract is charted with a 3600 second - instead of a 60 minute timeframe, the whole chart looks correct from 13th june up to the very last tick.

Also, if charted as volume based chart ( VOL 200.000 ) everything looks ok.

Maybe this is a bug in the new version, as I didn't experience this behaviour before.

Thank you for clarification

[NinjaTrader_Jessica]

When you run the Merge to your get a pop-up message mentioning any data that was not able to Merge?

It sounds like your Tick and second data are merging but the minute is not.

Can you attach a screen shot of the Chart?

[alchymist]

Thank you.......

the pop-up message says:

No day data available for "from" instrument
Minute data successfully merged
Tick data successfully merged

My data feed is a ZenFire test account.

In order to be sure, that the data base is not the cause of the trouble, I installed NT completely new "from scratch", the data base being completely empty at the beginning of the merging test.

First I loaded ES 09-08 then I loaded ES 12-08 then I started merging ES 09-08 (from 13th June to 11th Sept.) into ES 12-08.
Merging was completed with the above message in the pop-up window.
After this I opened a chart window and displayed ES 12-08 as a 60min, 3600s, and a VOL-200.000 chart.
You can see the result in the attached screen shots.


Very much appreciating any hint to solve this mystery. :-)

[NinjaTrader_Ben]

Hello,


I am sorry, I may not understand you completely.

What steps are you taking to produce the "no day data available"?

This is probably produced because you have not imported daily data.

I do not see anything wrong with the charts. Can you point out what I am not seeing?

What exactly is the issue you have having that I can resolve for you?

[alchymist]

In a nutshell:

After merging "ES 09-08" into "ES 12-08" the merged price and volume data are not charted correctly by NT.

There are many inconsistencies in the different charting modes (time-, tick-, and volume based)

Attached please see the 90min chart of "ES 09-08" at the left.
At the right you see "ES 12-08" - charted AFTER!!! merging it with "ES 09-08"

Both charts should look basically the same if everything was correct.
They obviously do not!
There is a "gap" between 06.30.2008 and 09.10.2008.
Price and volume within this "gap" represent the data of the ORIGINAL ES12-08 BEFORE merging took place and NOT the data of the source contract ES09-08.

My best guess:
Either the data base gets somehow corrupted and out of sync - or the procedure which reads the database and draws the chart has a bug.
The problem can only be observed with merged contracts.

Chart configuration:
start date: 12th of june 2008
end date: 16th of september 2008
time base: 90 Minutes
Cursor position: 27th of June 2008


Other details in order to reproduce the effect:
--------------------------------------------
NT Version 6.5.1000.6
New installation, database empty at the start of the test.
Merging data:
ES09-08 from 06.13.2008 to 09.11.2008
merged into
ES12-08


I hope I could clarify the problem.

[NinjaTrader_Ben]

Hello,

I am working on this. I will be with you as soon as I get more information.

[alchymist]

Thank you,......

I just realized, that another user seemed to have the same problem.
Look at this:

http://www.ninjatrader-support.com/vb/showthread.php?t=8677

[NinjaTrader_Jason]

Hello alchymist,

I tested merging on my end and it worked as expected. Please follow the steps below and check if it will work on your end.

First delete all historical data for the ES 12-08 via Tools-->Historical Data-->Delete.

Connect to Zen-Fire.

Open two charts:

- ES 09-08 chart beginning at 6/13 ending at 9/11

- ES 12-08 chart beginning 9/12 to present

Have charts opened and merge the data (see screenshot).

The merge should take a few seconds to be finished.

Open up a new chart for the ES 12-08 and check if all is displayed correctly.

[NinjaTrader_Ben]

Hello,

I was able to reproduce your issue and Jason's below.

What you may be doing is pulling more data than you have merged. Only pull up data prior to 6/13 when you open up the ES 12-08 chart in the final step of Jason's example below.

[alchymist]

Thank you Jason & Ben / The mystics continue....

First the good news:
I was able to reproduce Jason's example.
This prescription seems to work relatively stable.



And now the bad news:
If I open additional VOLUME or TICK charts on the SAME merged data, they are displayed somehow corrupted.

See the attachment below:

1) Newly generated ES 12-08 / 90min as per Jason's advice.
Ranging from 6/13 to 09/18 (Chart is OK)

2) VOL-100.000 Chart of the SAME ES 12-08!!!
(Messed up between 7/4 and 9/11, 9/16 and 9/17 missing)

3) VOL-100.000 Chart of ES 09-08
(Messed up between 7/18 and 9/10)


How can this be?
Both VOLUME charts are based on a correct? data base (At least the merging pop up window said tick and minute data successfully merged, AND price and volume seem to be perfectly ok judged by the 90min chart.)

To make the whole thing even more mysterious and confusing:
One test-run out of about 50 has been successful. That means the time based charts AND the VOLUME based charts were ok.
That successful merging procedure concerned the 6E 12-08 contract.

?!?!

[NinjaTrader_Jason]

Hello alchymist,

I display a similar chart when changing the ES 12-08 90 minute chart to a 10,000 volume chart.

From 8/12 to 9/11, it seems each day has less than 10,000 volume. I tried reloading the historical data, but the data display remained the same. This would indicate the data is being provided this way to NinjaTrader by the data feed.

However, upon reloading the historical data, data for 9/16 and 9/17 appeared in the chart.

I suggest to reload the historical data and check if you display data for 9/16 and 9/17.

[alchymist]

Hello Jason,


I QUOTE you:
From 8/12 to 9/11, it seems each day has less than 10,000 volume. I tried reloading the historical data, but the data display remained the same. This would indicate the data is being provided this way to NinjaTrader by the data feed.
UNQUOTE


As I understand the flow of the NT Software:

1) Merging concatenates the last and the present contract.
That means:
All original data of ES12-08 BEFORE! the rollover date (09/12) are
replaced by the corresponding (and adjusted) data of ES09-08.

2) From this time onwards the data base should no longer hold the original
ES12-08 data.
Therefore you should not be allowed to RELOAD any data being situated
BEFORE! the rollover point into ES12-08 (09/12), because RELOADING
would again replace the merged data by the original (unmerged) data of
ES12-08 which are then supplied by the ZenFire feed at the time of
RELOADING.

Besides:
It is possible to RELOAD the last 3 days, which were not correctly drawn by the VOLUME-chart, because they are positioned AFTER the rollover point. RELOADING those data therefore will do no harm to the merged ES12-08 data base.


3) If once the two contracts are merged, it should be the
responsibility of the data base to keep the data secure and
uncorrupted.

4) The charting procedure is simply reading the data out of the data base
and drawing a nice picture (hopefully :-))
It should normally not be allowed to make any changes to the data
base.


Facts:
We see that merging was obviously successful "somehow" .....
The proof for this is our 90min chart. It shows a 100% correct time series in price and volume.
So it is absolutely clear, that the data base keeps the correct data.

Questions:
Sorry to repeat myself - but how can it be, that a second chart (which necessarily pulls the data out of the SAME data base) shows a completely different volume compared to the first chart.
If you examine this "low volume" section closely, it proofs to be a perfect copy of the original unmerged ES12-08 contract. In price and in volume.
How can THIS be??
The original price and volume data of ES12-08 should have been replaced at the time of merging. That means that the old data should have been wiped out in the data base. (overwritten, replaced what so ever...)
How can those (wiped out) data then partially show up on a chart??

Conclusion:
After dozens of test runs and checking nearly any possible combination of setups, I lost confidence that the problem may be solved the "easy way".
Meaning stupidness on my side to operate the software. :-)
It seems pretty clear to me that NT has major problems (instability) with those MERGING and DATA BASE handling topics.

As a user I have a dream:

Imagine you choose "Merge" from a menu.
A window opens
You enter the source contract
You enter the starting date
You enter the rollover date
You enter the destination contract
You press "ENTER"

Nothing more - the rest is done by the software .........

No more fumbling with chart open, chart close, connecting, disconnecting,
entering "from" - "to" dates in charts as well as in the "merging dialogue" and so on.
Hope this becomes true one day :-)

[NinjaTrader_Jason]

Hello alchymist,

I will need to consult with our Development team regarding this case. As soon as we finished consulting, I will post the outcome in this thread.

[alchymist]

Thanks a lot Jason for evaluating this,


I run NT since mid 2007 in parallel to eSignal and really love this piece of software, - especially in tandem with the ZenFire feed........

....if there was not this quarterly repeating rollover pain. :-)

Unfortunately my trading style requires at least 2 months of uninterrupted, consistent, time- and volume based data history.

Therefore very much appreciating your further investigation.

[NinjaTrader_Jason]

Unfortunately, upon switching interval type (tick, minute and daily) NinjaTrader will retrieve historical data from the data feed. Tick data is used to create volume charts.

I will forward your comments and suggestions in regards to the merge feature to our development team and ask them if they can add this to the list of future considerations for the software.