Slow Tick Replay

[gomifromparis]

Hi,

I find tick replay to be slow.
I built a very simple test indicator that does nothing, but in tick replay mode

Read CL from may 8th to June 18th

results :
ticks : 5849264 , volume : 7497777 , time :00:00:03.8579406
ticks : 5849264 , volume : 7497777 , time :00:00:03.8843013
ticks : 5849264 , volume : 7497777 , time :00:00:03.8675965
ticks : 5849264 , volume : 7497777 , time :00:00:03.6222143

4 second load time.

I tested the same indy on NT7 with GomRecorder that reads the ninja tick files (the Ninja ntd files)
results :
ticks : 5842233 , volume : 7489700 , time :00:00:00.2975015
ticks : 5842233 , volume : 7489700 , time :00:00:00.2951910
ticks : 5842233 , volume : 7489700 , time :00:00:00.2877999
ticks : 5842233 , volume : 7489700 , time :00:00:00.2883060

0.30 second load time

Any way yous guys can speed up the file reading, because if we have a couple tick replay indicators on charts, it's going to slow down charts a lot... And I'm really impatient to get rid of the GomRecorder stuff..

Or am I doing it wrong ?

Thanks

[NJA_MC]

I have noted similar performance and think this is one they need to look at. Seems slower than NT7 in my mind, thanks for the data to support that feeling.

[sledge]

The quantity of data looks the same (<1%difference).

Isn't the new stuff broken down further than seconds? Is this the issue here?

[NinjaTrader_Matthew]

We're running analysis on the performance, however:

The quantity of data looks the same (<1%difference).

Isn't the new stuff broken down further than seconds? Is this the issue here?

This could come into play since the tick data is stored down to a sub-second granularity (DateTime.Tick), which leads to more data to process. As a result of this, any tick testing from NT7 to NT8 is not an apples to apples comparison and we can expect NT8 to take longer to process all of that information.

I cannot speak on NT7 GOMI framework to NT8, but would love to hear GOMI's input here.

I will get back to you guys on this thread once we've ran our test cases we're prompting as a result of this thread.

[gomifromparis]

To isolate the impacts of the new file format, I plugged a NT 8 file format reader to GomRecorder on NT7.

Results : on CL 08-15 , 10000 tick, from May 05 to June 19.

NT7 reading NT7(ntd) files using GomRecorder
ticks : 6202905 , volume : 7956246 , time :00:00:00.3674868
ticks : 6202905 , volume : 7956246 , time :00:00:00.3163918
ticks : 6202905 , volume : 7956246 , time :00:00:00.3270841
ticks : 6202905 , volume : 7956246 , time :00:00:00.3200613

NT7 reading NT8 (ncd) files using GomRecorder
ticks : 6206651 , volume : 7960748 , time :00:00:00.6937655
ticks : 6206651 , volume : 7960748 , time :00:00:00.7031810
ticks : 6206651 , volume : 7960748 , time :00:00:00.6912060
ticks : 6206651 , volume : 7960748 , time :00:00:00.7079216

NT8 reading NT8 files
ticks : 6206508 , volume : 7960559 , time :00:00:04.3398267
ticks : 6206508 , volume : 7960559 , time :00:00:04.2863122
ticks : 6206508 , volume : 7960559 , time :00:00:04.4037290
ticks : 6206508 , volume : 7960559 , time :00:00:04.3538043

[honvl]

To isolate the impacts of the new file format, I plugged a NT 8 file format reader to GomRecorder on NT7.

Results : on CL 08-15 , 10000 tick, from May 05 to June 19.

NT7 reading NT7(ntd) files using GomRecorder
ticks : 6202905 , volume : 7956246 , time :00:00:00.3674868
ticks : 6202905 , volume : 7956246 , time :00:00:00.3163918
ticks : 6202905 , volume : 7956246 , time :00:00:00.3270841
ticks : 6202905 , volume : 7956246 , time :00:00:00.3200613

NT7 reading NT8 (ncd) files using GomRecorder
ticks : 6206651 , volume : 7960748 , time :00:00:00.6937655
ticks : 6206651 , volume : 7960748 , time :00:00:00.7031810
ticks : 6206651 , volume : 7960748 , time :00:00:00.6912060
ticks : 6206651 , volume : 7960748 , time :00:00:00.7079216

NT8 reading NT8 files
ticks : 6206508 , volume : 7960559 , time :00:00:04.3398267
ticks : 6206508 , volume : 7960559 , time :00:00:04.2863122
ticks : 6206508 , volume : 7960559 , time :00:00:04.4037290
ticks : 6206508 , volume : 7960559 , time :00:00:04.3538043

Hi Gomi,

Is it possible to provide the file format reader for ncd files for GomRecorder? CQG tends to disconnect every weekend for maintenance, causing some loss of premarket GomRecorder data. It is difficult to backfill GomRecorder Millisec files, but easy to run NT7 and NT8 in parallel with NT8 backfilling ncd files from the CQG historical server. Any help would be appreciated.

[wbennettjr]

I noticed a huge slowdown when going from beta 6 to beta 7 (or it may have been 5 to 7).