All GomRecorder Indicators

[superarrow]

Data not saved

Gomi,
Disabling time filter had no effect. Further, I removed the recorder and kept the CD with recording status true, file format binary and ran replay for couple of days. The binary file was the same size at the end of the replay even though "Recording Binary OK" was showing all along. Pressing F5 erased all the CD data and reset to zero. This shows no data was saved.
FYI I am testing it on NT B5

[gomifromparis]

Maybe file is corrupt ? Could you try removing it ?

EDIT : Also keep in mind you can't "backfill" data preceding last written date. You can only write data past the last written date.

[superarrow]

Yes, file was corrupt. Good call. Now it seems to be running as expected. Glad with it.

Ninja is a good product but it's you Gomi that give it that edge for traders like myself. Thank you, for w/o your contributions, other charting platforms would have drawn my interest.

SA

[gomifromparis]

Thanks

I added the GomFileConverter in post #1 with documentation in post #5

[dandxg]

Gomi you really went over the top with the volume ladder. I am sure many a MD customer is singing the praises of Gomi. Thanks so much.

[FulcrumTrader]

BRILLIANT work again Gomi!!!

[deanz]

lastprice

Calculation Mode: UpDownTick
volume is up if price>lastprice

How do you calculate lastprice say on a 10 minute bar, is lastprice=opening price or can you look at tick data inside every bar ?

.

[aviat72]

Gom:

I really love what you have done. The next step should be increasing the stability and reliability of the on disk data-storage infrastructure.

I also have had a variety of problems recording data. The time-stamp would update when I do an reload of NS or restart NT but the size would not change and so on.

As you suggested it might be a case of binary file corruption.
-----------------------------------------------------------------
The next step in the development of the GOM tick data-storage system is to add some fault-tolerance and error-detection & reporting mechanisms.

-> Print out a warning to the output window if the binary file reader/writer discovers an error. Produce an audio clue (a beep) to warn users to check.

-> The next step, of course would be to recover as much data as possible from the corrupted file.

-> When it comes to reducing the chance of corruption, I am not a C#/MS guru but perhaps doing periodic flushing of the write buffer might help reduce the chances of corrupted files.
---------------------
An alternative approach is to have a flow which tries to ensure that the bin file remains uncorrupted as much as possible.


(a) We could first save the tick information as a text file and then run a utility to convert into the binary format and then append it to the existing binary file for that instrument. The text format will allow manual repairs of the tick data and the conversion to binary will be useful for archival purposes. Further we can maintain different text files for different sessions (timestamp the name) so that the binary can be incrementally updated

(b) The dispose() method of the RecorderIndicator can call the function to perform the text to bin conversion for that session. If the function fails to convert the text file it prints a message and exits without corrupting the binary file, and printing a text message in the log/output window and also the $GOMDIRECTORY about what was wrong. Instead of working directly with the bin file, create a fresh copy while updating, and then rename it to the original bin file after its post-update intrgrity has been verified.
---------
Added later:

I noticed that you have already added the converter in the fifth post of this thread (which was previously reserved for future use). Much of what I have suggested above can now be automated using some Window Scripts (which I am not good at writing (hint)...)

Also noticed that the ConversionUtility is not using the $GOMFOLDER env variable to produce the output file. Also the Window title "Form1" could be changed (yes, yes I am being picky, but this is a thing of beauty and any blemish stand out).

(c) We would also need a command line version of the GOMTXT2BIN converter which can be called to convert the text files to bin after the text file has been fixed.

[gomifromparis]

Calculation Mode: UpDownTick
volume is up if price>lastprice

How do you calculate lastprice say on a 10 minute bar, is lastprice=opening price or can you look at tick data inside every bar ?

.

lastprice is the last tick price, so it looks at every tick inside the bar.

[Sergio66]

It seems to me that the SMoving Average is not customizable. The parameter is (5) on default and I am not able to change the settings

[gomifromparis]

It seems to me that the SMoving Average is not customizable. The parameter is (5) on default and I am not able to change the settings

Yep, sorry, bug in 6.5...

[gomifromparis]

Nice post

Some thoughts :

1. This "corruption" thing seems rare, I haven't encountered myself, but there are some messages on the board mentioning it... It might be a bug but it's hard to analyze since I've never encountered it.

2. The write buffer is flushed on every tick.

3. Automatic process in Dispose() may be more a problem than the solution. File reading/writing may be quite time consuming with large files or small CPU computers, so shutown or indicator removal will take a long time, eventually users will kill the process and perfect files will become corrupt

4. If you really need file completeness I don't think you can avoid a backfill external process. I use QCollector to downlaod daily/weekly the data, and I construct my historical file myself. Who knows when your computer will reboot/NT will crash/power will shut down etc..

5. I think a more robust solution for file backup/restoration would be to install a software like Crashplan. This sofware will allow you to backup the file as frequently as you like, even every 5 minutes if you wish. You will be able to restore any version from the past. It incorporates file deduplication so only modified parts will be saved. It is able to backup open files using Volume Shadow Copy. It is able to backup on another computer in case your disk crahes. Great software, but with free version you'll only be able to backup daily.

6. Command line for the converter is a good idea. Had the idea but have been more or less lazy :-)

Gom:

I really love what you have done. The next step should be increasing the stability and reliability of the on disk data-storage infrastructure.

I also have had a variety of problems recording data. The time-stamp would update when I do an reload of NS or restart NT but the size would not change and so on.

As you suggested it might be a case of binary file corruption.
-----------------------------------------------------------------
The next step in the development of the GOM tick data-storage system is to add some fault-tolerance and error-detection & reporting mechanisms.

-> Print out a warning to the output window if the binary file reader/writer discovers an error. Produce an audio clue (a beep) to warn users to check.

-> The next step, of course would be to recover as much data as possible from the corrupted file.

-> When it comes to reducing the chance of corruption, I am not a C#/MS guru but perhaps doing periodic flushing of the write buffer might help reduce the chances of corrupted files.
---------------------
An alternative approach is to have a flow which tries to ensure that the bin file remains uncorrupted as much as possible.


(a) We could first save the tick information as a text file and then run a utility to convert into the binary format and then append it to the existing binary file for that instrument. The text format will allow manual repairs of the tick data and the conversion to binary will be useful for archival purposes. Further we can maintain different text files for different sessions (timestamp the name) so that the binary can be incrementally updated

(b) The dispose() method of the RecorderIndicator can call the function to perform the text to bin conversion for that session. If the function fails to convert the text file it prints a message and exits without corrupting the binary file, and printing a text message in the log/output window and also the $GOMDIRECTORY about what was wrong. Instead of working directly with the bin file, create a fresh copy while updating, and then rename it to the original bin file after its post-update intrgrity has been verified.
---------
Added later:

I noticed that you have already added the converter in the fifth post of this thread (which was previously reserved for future use). Much of what I have suggested above can now be automated using some Window Scripts (which I am not good at writing (hint)...)

Also noticed that the ConversionUtility is not using the $GOMFOLDER env variable to produce the output file. Also the Window title "Form1" could be changed (yes, yes I am being picky, but this is a thing of beauty and any blemish stand out).

[mpe66]

It seems to me that the SMoving Average is not customizable. The parameter is (5) on default and I am not able to change the settings

Gomi,
I really appreciate your work. I suck at programming and was wondering if it would be easy to fix the GomCDSMA? Also, could it be changed to an EMA?

Thanks,
Mpe

[gomifromparis]

I dug out the constant delta bar type and posted it on first post. Documentation and chart is a few posts later.

[gomifromparis]

Sorry I have no time but you can modify this code in Initialize ()

    
#if !NT7
  indic=SMA(GomCD(GomCDCalculationModeType.BidAsk,GomCDChartType.CumulativeChart,GomFileFormat.Binary,GomFilterModeType.None,1,false).DeltaClose,5);
#endif

remove the 5 and type period
replace SMA with whatever you want

Sorry

Gomi,
I really appreciate your work. I suck at programming and was wondering if it would be easy to fix the GomCDSMA? Also, could it be changed to an EMA?

Thanks,
Mpe