Deleting old unused data.

[8DTK8]

I just stumbled upon the delete feature after merging some futures contracts.

It dawned on me that all of the old contracts are probably taking up a ton of space and have no use since I roll them all into the front month.

Unfortunately, I can't remember when I started doing all of this and I'm quick to take out symbols when a contract expires since I already have all the data merged into the new contract.

Is there any way I can see which contracts / instruments have data still in them? I'm going to guess that having minute and daily data for a decade worth of trading for every contract of the year for the same underlying is what's probably making my backups take over an hour and what is making the backup file massive.

I hope there is a better way of deleting them than deleting Mar, Jun, Sep, Dec contracts for every year that has passed.

Also, what does "repair DB" in the "misc" tab of "options" do.

Many thanks in advance,

[NinjaTrader_Bertrand]

8DTK8, I believe the best approach would be exporting the data you really need at this point in time (Tools > Historical Data > Export) and then starting over with a fresh database file and reimporting this exported data then. Otherwise you would need to export the data to check which contracts hold data and which not, NinjaTrader 7 will offer a more convenient way of handling this - a Historical Data Manager -

http://www.ninjatrader.com/webnew/NT7/NinjaTrader7.html

New Historical Data Manager
The Historical Data Manager allows you to import/export and edit historical data. The image below depicts a historical tick table with an excluded record (bad tick) marked in red and a changed record which is marked in yellow. If historical data is reloaded it will not overwrite any of the changes, exclusions or data additions. Changes logs are maintained and original values can be easily restored.

Repairing the database is a process to compact / optimize the mdb file and recover it from minor corruptions experienced.

[8DTK8]

Thanks Bertrand,

How do you delete the database? Is it "Reset DB", "Reset Instruments" or something else?

I'd do guess and check, but it seems rather risky given its nature.

Thanks again.

PS - with all the talk of NT7, which sounds fantastic, is it going to be a pretty smooth transition with the code that your users have already written or are there changes to the platform that will require revisions to our code?

[NinjaTrader_Bertrand]

First, please export all data needed and then exit NinjaTrader and go to -

MyDocuments\NinjaTrader 6.5\db and rename the NinjaTrader.mdb to for example NinjaTraderOld.mdb.

Upon restart NinjaTrader will now create a new database file, unfortunately prior execution data as well as saved ATM strategy templates would not be persisted with this step.

Then you can reimport the needed data...

Generally we expect code to migrate over well to NinjaTrader 7 - however of course method signatures were changed / updated / amended and thus scripts could need a revision to make them 'fit' for NinjaTrader 7 -

http://www.ninjatrader-support2.com/vb/showthread.php?t=21016

[8DTK8]

After a long while, I gave this a try.

I bumped into the following problem. Here is the snip from the log file.

9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 470695. CLose value needs to smaller/equal high value: 20090615 125300;109.6875;109.6875;109.65625;109.71875;196
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 470701. CLose value needs to smaller/equal high value: 20090615 125900;109.65625;109.65625;109.65625;109.6875;54
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 470710. CLose value needs to smaller/equal high value: 20090615 130800;109.625;109.625;109.625;109.65625;17
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 471226. CLose value needs to smaller/equal high value: 20090615 231400;109.8125;109.8125;109.78125;109.84375;94
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 471246. CLose value needs to smaller/equal high value: 20090615 233400;109.65625;109.65625;109.65625;109.6875;28
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 471269. CLose value needs to smaller/equal high value: 20090615 235700;109.40625;109.4375;109.40625;109.46875;14
9/9/2010 10:29:03 PM|3|4|ZB 12-10: Illegal close value in line 471271. CLose value needs to smaller/equal high value: 20090615 235900;109.5;109.5;109.5;109.53125;43

It just goes on and on like this. It exported just fine. I'm not really sure why it just chose some random spot to quit importing because there seems to be more than enough consecutive errors.

Thanks in advance.

[NinjaTrader_Bertrand]

8DTK8, so you were exporting from 6.5 and then reimporting in NT7 or also in 6.5? Could you perhaps attach a portion of the original file used?

Thanks

[8DTK8]

Hi Bertrand, from 6.5 and back to 6.5.

Are you requesting the exported file? It's pretty huge. Here are a few of the corresponding lines.

20090615 235400;109.46875;109.46875;109.4375;109.4375;105
20090615 235500;109.4375;109.4375;109.4375;109.4375;28
20090615 235600;109.4375;109.4375;109.375;109.375;176
20090615 235700;109.40625;109.4375;109.40625;109.4375;14
20090615 235800;109.4375;109.5;109.4375;109.5;35
20090615 235900;109.5;109.5;109.5;109.5;43 <<< looks like this is offender matches the last entry of my previous cut and paste.

The line is exactly as it says. The close for some reason is higher than the high.

Please let me know if there's anything else I can provide.

[NinjaTrader_Bertrand]

Thanks, the export is indeed not producing valid lines - do you still got the original database and could try to repair it, the source data looks unfortunately corrupted.