Crash downloading market replay data

[Brillo]

I was downloading data for CL 12-16 for 7/20/2016 - 7/27/2016.
Tick and Minute, Bid, Ask, Last
First I tried with no connection.
Tried with Market Replay.
In one of those states the button was disabled ( can't remember which )
And in the other state I got a message box saying to connect to my data provider.

Connected to Rithmic.
Hit download. NT8 disappeared. No message. Trace file shows contention accessing a file.
See attached.

[NinjaTrader_KennethD]

Hello,

Thank you for your post. As per the log another process was accessing one of NinjaTrader files. You may need to add the ninjatrader folders located at C:\Users\your username\Documents\NinjaTrader 8 to exception lists of any antivirus, back up software, etc in order to ensure nothing else is accessing these files. Please contact us if we may be of any further assistance.

[Brillo]

Hi Ken,
A few comments:

1. The contention for file access should not cause NinjaTrader to crash. It should handle the exception.
2. I'm pretty confident about my theory the file was being accessed concurrently by 2 separate threads within the NinjaTrader process. I know the message text says "another Process" but I've seen that exception before and it will say that even if the concurrent access is from 2 different threads. That's how the security for kernel objects in Windows works anyway... it's on a thread by thread basis. Poorly worded message by the Microsoft engineer.

How did the exception happen? Not sure but I think that NT was updating the files through the Rithmic connection on behalf of the requesting chart that was loaded with the instrument. I had just re-started NT, connected to Rithmic ( with the chart open ) and immediately went to download the data. Maybe that caused it.

Bottom line: NT should not crash. That exception should be handled.

[sledge]

How did the exception happen? Not sure but I think that NT was updating the files through the Rithmic connection on behalf of the requesting chart that was loaded with the instrument. I had just re-started NT, connected to Rithmic ( with the chart open ) and immediately went to download the data. Maybe that caused it.

Do you have record for market replay data turned on by chance?

NT7 has a message which prevents download while recording is on.

Just a wild guess. I haven't tested this part of NT8.

[Brillo]

Yes that check box is checked "Enable market recording for playback". Also, I have checked "Record live data has historical".
Some other solution than force the user to uncheck the box for playback recording would be better. It's annoying to uncheck that box. NT should just take care of coordinating multiple concurrent downloads.

[NinjaTrader_Matthew]

Yes that check box is checked "Enable market recording for playback". Also, I have checked "Record live data has historical".
Some other solution than force the user to uncheck the box for playback recording would be better. It's annoying to uncheck that box. NT should just take care of coordinating multiple concurrent downloads.

We've addressed a few scenarios around this for the next release

#10483 Historical data recording thread locking .ncd file

To be specific, the issue is not around recording market replay data, so you should not need to uncheck that box.

The specific issue is around "Record live data as historical", since there were some issues with reloading the same data that is being recorded. The temporary solution is to uncheck that option until the next release is out. You can continue to record market replay data..

For what it is worth, in most cases we review users have that option checked, it is usually unnecessary. While there may be a very specific reason you are recording live data as historical, but in principle, the option was designed for users who do not have access to historical data. If you are offered historical data from a provider, you should consider disabling this option since it uses unnecessary PC resources and can lead to gaps in your data cache.

[Brillo]

Thanks for you update. That's good to know it will be fixed.

And thank you for your advice about not recording as historical. However I think it's better to advise recording it.

Correct me here if I'm wrong about my understanding:

1. Charts, Market Analyzer, and any other client request such as internally loaded data series pull their data from the files in the "cache"

db\cache

The folders in there have file sets for the particular session template, and instrument combination that have been requested.

For example:
NinjaTrader 8\db\cache\CME US Index Futures ETH.Eastern Standard Time\TICK\ES 12-16

2. If the data is not in the cache then the cache is re-built from the historical files in

db\minute or
db\tick


Replay data get's it's own historical file format stored in
db\replay

3. if the data is not in the historical files then it's requested from the current market data connections in the order the connections where established. ( except if it's a Replay Connection )

If it retrieves it from the market data connection it can build the cache and the initial request is honored. Further, if the user has selected "Record market data as historical" then it's also written to the historical files, where, in the future step #2 will succeed.

So by unchecking record market data as historical there's no chance to restore the cache from the historical data. The only option is to get it again from a market data provider. For tick data this means only about 6 months or so of data.

[NinjaTrader_Matthew]

No, that is not exactly true. There is a physical cache on disk, and a memory pool. All real-time market data makes it into the memory pool, and eventually to the cache. Data from the cache eventually makes it the files on disk .This happens regardless if you have that option checked. Enabling the option writes the data in real-time to disk at the expense of CPU resources.

Furthermore, when using 'record live data as historica' in combination with a historical data provider, there is a higher chance you will find a gap between session end last trading day and begin of the actual file if there are gaps in the current day file (due to restarting or disconnecting).

[Brillo]

Data from the cache eventually makes it the files on disk .

Which files do you mean? The ones in cache or the ones in minute and tick? The reason I ask is that the once in cache folder are specific to the session template. If, they're only in there you can't switch to a different session template and get deep historical data for it. Is this a concern?

This happens regardless if you have that option checked. Enabling the option writes the data in real-time to disk at the expense of CPU resources.

I would like to avoid that since that's the case!

Furthermore, when using 'record live data as historica' in combination with a historical data provider, there is a higher chance you will find a gap between session end last trading day and begin of the actual file if there are gaps in the current day file (due to restarting or disconnecting).

[NinjaTrader_Matthew]

The .ncd files, the raw data files that either come from the provider or recorded in real-time. The cache files are simply segments of those .ncd files which have already been built to speed up the process of generating segments of days instead of having to read the whole .ncd file for a period. I'm not sure what your concern is, but it does not trigger a concern in my mind.