Error message: historical data in your database is corrupted

[adamus]

I am trying to run an optimisation on a large historical data set and it has so far crashed 4 times after only 10 to 20% of the job.

This last time NT7 told me the historical data in the database for the symbol I was running on is corrupted.

I heavily suspect that isn't corrupted, rather the error derives from memory and other resource problems during the big optimisation.

The data that I am testing on has not been updated for 3 months, so it hasn't been written to and I doubt this new machine has developed a hard drive problem already. That's why I think it's not actually database corruption.

I have looked at the chart for the instrument and all the data looks ok.

It would take me a long time to reload the data - it's forex tick data from 1997 to the present, one big zip from the provider plus multiple little update files.

How else can I check whether this 'historical data corruption' is a false alarm?

Thanks

[NinjaTrader_Jason]

Hello adamus,

Can you please tell me what the error message exactly states or provide me a screenshot.

Did you monitor memory usage during the optimization using the Windows Task Manager. Did it max out the memory?

Please note that historical data is not saved in the database in NinjaTrader 7. This was the case with NinjaTrader 6.5. Historical data is saved at (My) Documents-->NinjaTrader 7-->db-->daily/minute/tick folders.

What happens if you restart NinjaTrader and subsequently create a new chart for the same data set - does the error persist?

[adamus]

This was the last crash:

26/01/2011 12:47:36 Default Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.

26/01/2011 12:58:10 Default Unable to write cache data

The mention of corruption was from the time before:

26/01/2011 09:08:02 Default Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.

26/01/2011 09:08:26 Default Historical data in your database is corrupted.

But the actual pop-up error had far more detail - it said which instrument and the time period start and finish that were corrupted.

Unfortunately I can't reproduce it now.

I didn't monitor the memory usage but the trace file is full of System.OutOfMemory exceptions.

I restarted NT7 and was able to create a chart of the whole period at the 5min bar level that did not look corrupted at all, although I only checked by eye.

So if the historical data is not in the database, why is the error message telling me that the historical data in the database is corrupt? Is this an out-of-date piece of code which is displaying the message? Or are you saying that there's more than one type of historical data which coudl get corrupted?

Thanks

[NinjaTrader_Jason]

The out-of-memory exceptions indicate the optimization takes up too much memory from your computer. To avoid these errors, please use less optimization parameters or less historical data.

The data corruption error message itself is outdated. It refers to the historical data you have saved in the daily/minute/tick folders.

I assume you used a tick based interval during your optimization. Could you please create a tick chart instead of a minute chart. Check if the data corruption error message persists.

[adamus]

I ran the optimization with only 1 possible parameter combination so it would only do one loop. The error occurred again. Please see the attachment for the original error pop-up.

The following errors were logged:

26/01/2011 14:11:40 Default Unable to load server side config file: Exception of type 'System.OutOfMemoryException' was thrown.
26/01/2011 14:12:27 Default Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.
26/01/2011 14:13:45 Default Historical data in your database is corrupted.
Please reload historical tick data for '$AUDJPY_DTC' from 31/12/2000 to 01/01/2009.
Tools->Historical Data->Reload...

[adamus]

Also, in response to your question whether I can display the tick bar chart of the same length bars and instsrument that I am using for the strategy analysis, yes I can, for shorter periods.

I am trying to run the optimisation from 2001 to 2008. It seems to be too long. I tried it on another instrument, and got a different error message.

First it said "Unable to read cache", and then came the pop-up as attached.

[NinjaTrader_Jason]

Am I correct you experience the data corruption error if you create a tick chart from 2001 to 2008? Do you experience the out-of-memory exceptions as well?

Can you please send me your log and trace files for the days you experienced these issues. You can find log and trace files at the following locations.

Start Menu--> My Documents--> NinjaTrader 7--> Log--> log.20110126 for today. (log.yyyymmdd for any other date)

Start Menu--> My Documents--> NinjaTrader 7--> Trace--> trace.20110126 for today. (trace.yyyymmdd for any other date)

Please send them to support[at]ninjatrader[dot]com. Put 'Attn Jason' in the subject header and reference this thread.

[adamus]

Hi Jason,

please find the log and trace file atttached in the zip.

I only received the data corruption error message when running the strategy analyzer.

When loading a tick series bar chart for the whole period in question for $AUDJPY_DTC which was allegedly corrupt, that is 2001-2008, NT7 just crashed, no error given.

However I was able to load the chart for shorter periods than that after rebooting.

So the corruption message only came during the optimization.

[adamus]

I am able to run the backtest on this 1 tick bar time series and instrument and it runs until it gives me an orderly data corruption error without crashing the whole app.

This way I can narrow down the date range hopefully to the exact point in the data that is corrupt and repair it by hand.

[NinjaTrader_Jason]

I suggest to avoid running such large optimizations, since they result in out-of-memory exceptions anyway and only such tests produce the historical data corruption error.

It seems like the out-of-memory exceptions causes historical tick data to become corrupted. Am I correct the 'historical data corruption' error only occrus after you experienced out-of-memory exceptions?

As per your latest post, did it refer to the same historical data set as the prior error?

How did you obtain the historical data? Did you manually import it or did you load it from a connected data feed?

[adamus]

Hi Jason,

I can't say with any confidence where or when the historical data corruption occurs because at this point fortunately it has only happened once, but the corruption continues to affect further optimisations and backtests.

To answer my own question from the first post, I am now convinced that there is corruption in the historical data. I think I can isolate exactly when it is so that I don't have to reload all the data.

This is because the error only occurs when I run a backtest over the same instrument and the original period that I had for the optimization run which first resulted in an error. That was the period 2000 - 2008. However I can run a backtest on 2000 - 2006 without the error so I know the corruption lies in the 2006 - 2008 period, and I am going to reload that.

It also seems that the other instruments were not affected, luckily.

It is also worth noting that the optimisation cannot deal with the error as capably as the backtest. The backtest fails with the corruption message, but the optimisation might give me several different memory errors first. Each time I have to reboot to otherwise the application just hangs if I try to run a backtest or optimisation again.

The data was obtained by downloading the tick data as a zip file and importing it.

[NinjaTrader_Jason]

I see. Good to see you isolated the issue to 2006 - 2008. Please let me know if the data corruption error recurs after reloading the corrupted historical data.

Optimizations will use more memory than backtests when applied on the same data set. So the out-of-memory exceptions can be as expected in this situation.

[adamus]

OK, I'll let you know. I hope it doesn't recur, but of course if it has happened once then there is no reason to expect it not to happen again.

To be honest, I am surprised that the crash resulted in corruption in the historical data. The whole optimisation is surely just a read-only operation, so how can it actually write back corruption into the historical data files that it's only meant to read? This is surely a bug?

It seems I might have to keep my back-ups more regular and more conveniently located if I have to deal with this again.

[saltminer]

I have also seen this problem, in NT7.0.1.3. I was using the Genetic Optimiser on daily data over a twenty year period, with two instruments in the BarsArray.
It happened on
1) imported data
2) live connection to Kinetick
3) live connection to eSignal.
Different instruments selected for each test. So I don't think it is the data.

Entries in log include:
28/03/2011 2:44:45 AM Default ESignal.StandardAdapter.OnBar: Access to the path 'C:\Users\myUser\Documents\NinjaTrader 7\db\day\JBL\2008.Last.ntd' is denied.
28/03/2011 2:44:45 AM Default Historical data in your database is corrupted.
Please reload historical daily data for 'JBL' from 1/01/2000 to 31/12/2008.
Tools->Historical Data->Reload...

Cheers,
saltminer

[NinjaTrader_Jason]

Hello saltminer,

The first log entry you included is likely related to the data corruption in your situation.

28/03/2011 2:44:45 AM Default ESignal.StandardAdapter.OnBar: Access to the path 'C:\Users\myUser\Documents\NinjaTrader 7\db\day\JBL\2008.Last.ntd' is denied.

Can you please check if your virus scanner or any other application scans/locks files in the C:\Users\myUser\Documents\NinjaTrader 7 folder. NinjaTrader will need to write to this folder, however it is blocked because another application access/scans files in this folder at the same time. This causes the log entry message and likely the data corruption as well.

Please make sure to configure your virus scanner or other application (automatic backup software for example) so that none of the files in directory above would be scanned/locked/accessed or remove that directory from the scanning list.