Let me preface by saying a few things to bear in mind before anything else:
- I know that I operate MA far outside what it normally gets, but that's what I need for my trading strategy
- I am a very experienced computer professional
Now to some other operating parameters:
- My machine has 4 cores (i7) and 24GB RAM and drive C: is a regular disk (not SSD). Running Win 7 Pro
- I use IQFeed as a data source
- I load up MA with about 500 instruments - all the constituents of the S&P 500
- I have 5 indicators loaded (all of them custom)
- All indicators are configured to calculate on bar close (not tick)
- The indicators look generally back between 15 and 120 bars, with max look back on one of them set to 360
- I have 3 charts open all of them linked to the selected instrument on the MA and each with a bunch of indicators - all of them 1min charts
- Importantly, NT is configured to save chart data as historical as well as to record for market replay
- I've been using IQFeed with a custom Java application for the last two years for the same instruments, and was logging the data stream while running indicator logic in the application. The system had no problem keeping up with the data and was sustaining 120ms latency between transaction time and write time. I was using my own storage format, but clearly it shows that dealing with the amount of data should not be a problem for a system that is designed for this number of instruments (same system, same network, same everything - just a different application)
And some symptoms:
- Indicators on the MA can be up to 5 minutes behind real time
- Charts seem to be updating OK
- CPU utilization is pretty low (never above 20%)
- IO on the C drive seems to peg often due to seek time (not throughput) with the files experiencing the worst response time all being in the NT DB. This high IO is only if the MA is open. If I close it then it calms down (not surprising). It appears to be tick data that's getting write delays most of the time.
- NT as a whole (the whole application) would enter a semi-frozen state every once in a while (maybe once an hour or 2) but remains generally responsive most of the time. The problem seems to be affecting almost only the MA
I would venture to guess that the MA is affected directly due to saving chart data as historical (have not experimented yet with turning that off for reasons I will explain). This would imply that the data saving (especially ticks) is done in the same thread as the indicators, thus delaying their execution. The system is clearly IO bound, and more specifically bound by disk latency and not throughput.
Now, in parallel to this, I've been having a different problem with the historical data manager. It has not been stable downloading data for the same list of instruments and hangs entirely quite often (sometimes hanging the entire NT process). For the data manager, I get no problems if I download only minute and daily data. But if I ask for ticks or I ask for ask/bid data then it almost never manages to complete the download. This too, is something I do regularly for my Java app with no problem (I backfill my DB on a tick level from IQFeed to my custom DB and I have no problem). So I am really relying on MA to collect the data for me as i cannot rely on the historical data download. I need the historical data for backtesting (market replay is not very important to me actually, but strategy testing is very important).
So I would assert that I have no problem with the machine, network, or data provider - as all of these are eliminated by the trouble free Java implementation. I would concede that it is likely that switching to an SSD for storage may solve the problem (as it is a seek time problem) - but that would be both painful and expensive.
So after this long introduction, here are my questions:
- Given that I do want to record the data for strategy backtesting purposes, in the options/data tab what is safe to turn off and would it have any effect? (e.g. turn off record for market replay and leave save chart data as historical on).
- Is there any plan for making MA more scalable than it is in a future version (e.g. do these writes in a background thread, or use a less seek heavy storage format)
- Is the problem with the historical data manager downloads familiar to you? If so, is there a solution for that or alternatively, can I do strategy back testing for periods that have only minute data and no tick data?
Thanks
Arnon
Comment