NT, SPY, tick data and memory usage

[AnotherTrader]

I’m trying to run backtests for an intraday strategy:

• single instrument- SPY, i.e. SPDR S&P 500 ETF
• single timeframe – 15 second bars

I have observed the following:

A)
If I run backtest over a whole year of data, the strategy trades M times. However, if instead, I run the backtest over smaller periods of time (say two weeks), and repeat the process 26 times to cover the same whole year period as above (i.e. 26 x 2 weeks = 52 weeks = a year), the strategy trades a total of N times, where N > M by about 20%. So over the same period of time, and with the same data, I get a different number of trades depending on how I treat the test time period.

B)
I also notice that in the single 52-week backtest case, the charts for some days are missing whole sections of 15-second bars (which I guess will be related to whatever reason is causing fewer trades too in this case!). In the case of the two week backtests on the other hand, all charts on all days are fine, without missing bars as far as I can see.

My guess)
My guess is that the issue is related to my testing computer being overwhelmed by SPY tick data. SPY shares are, I believe, on a daily basis among the single most actively traded securities anywhere in the world, on any exchange.

In my personal experience, a typical trading day (04h00 – 20h00 EST) will see between 500,000 and 2,000,000+ SPY ticks. So a lots of ticks! A day’s SPY tick data in .txt format will be a file typically between 10 MB and 20+ MB.

So, two weeks of SPY data will be up to 200 MB of data (10 x 20 MB).
A full year, 5 GB of data (250 x 20 MB)
A full two years, 10 GB
Etc ….

My questions)
1. Does NT load all tick data required for a backtest into memory, all at once?
2. If so, then what specification in terms of memory should my testing computer have in order for me to be able to run backtests using multiple years of SPY tick data?
3. Does NT 7 offer any enhancements over NT 6.5 in this regard?

[NinjaTrader_Jason]

Hello AnotherTrader,

1. When you perform a backtest it will first load all required historical data. Once all data has been loaded, the strategy will be backtested.

2. Unfortunately we do not have such information listed on our end. It is hard to say how much memory you exactly need. Since you will perform very large backtests, I recommend to use of computer with as much memory possible.

3. NinjaTrader 6.5 supports multi-core/multi-processor hardware. NinjaTrader internally supports multiple threads of executions which would be run by the Windows OS on different cores/processors. However, NinjaTrader 6.5 is not optimized to make full usage of a multi-core/multi-processor hardware.

NinjaTrader 7 has optimized multi core/multi-processor support. This would allow you to make full usage of a multi-core/multi-processor hardware on backtesting multiple instruments or running multiple iterations of a strategy optimization at the same time.

However, due to the nature of the Window OS, NinjaTrader 7 would not be able to fully use all available multi-cores/multi-processors at any given time.

[AnotherTrader]

Thanks.

2. ... Since you will perform very large backtests, I recommend to use of computer with as much memory possible…

I’d like to try to get some estimate in advance of what might be needed, before just going out and buying a more powerful computer in the hope that it’s powerful enough.

To hold 10 GB – 15 GB of SPY tick data in memory (2 – 3 years), is that going to require a mid range/top range say DELL computer? Or will it require a chunky server? Or would the matter require some sort of super computer???? I have no sense of what would be required? Perhaps some other forum contributors have a sense?

3. …NinjaTrader 7 has optimized multi core/multi-processor support. This would allow you to make full usage of a multi-core/multi-processor hardware on backtesting multiple instruments or running multiple iterations of a strategy optimization at the same time.
However, due to the nature of the Window OS, NinjaTrader 7 would not be able to fully use all available multi-cores/multi-processors at any given time…

OK, I’ll try on NT 7 to see if it’s any better.

[NinjaTrader_Jason]

Yes, before looking into purchasing another computer, I suggest to test NinjaTrader 7 first and check if the annual backtests do not miss any tick data.

[AnotherTrader]

3. ...NinjaTrader 7 has optimized multi core/multi-processor support. This would allow you to make full usage of a multi-core/multi-processor hardware on backtesting multiple instruments or running multiple iterations of a strategy optimization at the same time.

However, due to the nature of the Window OS, NinjaTrader 7 would not be able to fully use all available multi-cores/multi-processors at any given time.

On the basis of your comment above that not all memory gets used, if I was looking for the capability to hold at least 2 years of SPY tick data (requiring say 10 GB) in memory, would 12 GB of RAM be enough for this?

NT (v7) would be the only application running, and I would have Windows 7.

[NinjaTrader_Jason]

Unfortunately I cannot tell for certain that amount of memory is enough, but I assume it will be enough to run such a backtest.

Please note that 32 bit operating systems are limited to use 3,5 - 4 GB RAM per process. 64 bit operating systems will allow much more RAM to be used per process. I recommend to go with a 64 bit operating system when choosing that amount of RAM.

[AnotherTrader]

... OK, I’ll try on NT 7 to see if it’s any better...

Just reporting back on where I’ve got to:
I had a third-party compress the SPY tick data (any continuous sequence of ticks at the same price was converted to a single tick with same total volume as all the combined ticks), so that approx 5GB of data (1 year of SPY tick data) was reduced to approx 500MB (in the case of SPY, this sort of compression doesn’t affect the generation of 15-second bars, which is what I need).
When loaded into NT7, the backtest ran without a problem.
So some combination of tick compression and using NT7 (rather than NT6.5) resolved the issue without needing to purchase a more powerful machine.

[NinjaTrader_Matthew]

Hello AnotherTrader,

Thanks for reporting back with your results.