Optimizer and Amazon EC2

[nicbizz]

Hi,

My home PC is an i7-920 @ 2.67Ghz, with 12GB RAM running Win 7 64-bit.

Optimization seems to take a long time on my machine, so I was wondering if it would be faster if I lease some processing power on Amazon's EC2 cloud server.

The 2 options I'm looking at is either high RAM or high CPU resources (both run Win Server 2008 R2):

1) High-Memory Double Extra Large Instance 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform

2) High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

My question is:

1) Would NT's Optimizer be able to use the multiple cores in Amazon's virtual server?

2) If #1 is possible, then from a cost-benefit standpoint, would higher RAM or higher CPU (more virtual cores) be preferred?

3) There is also a Cluster option from Amazon: Cluster Compute Quadruple Extra Large 23 GB memory, 33.5 EC2 Compute Units, 1690 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet - would NT's optimizer able to fully utilize the resources available?


Thanks!

-Nick

[NinjaTrader_Brett]

Hello,

Thanks for the note.

Optimization is multi-threaded in NinjaTrader therefor you would get some gain out of doing this.

I could not theorize on what would be better here as there are a lot of factors at play.

There has been some guys that have posted on this forum in the past with their results on doing this. If you would like to research more information I would suggest searching for these posts so you can find the user feedback and results in doing this as we do not have any official benchmarks on what is best here..

-Brett

[nicbizz]

Hi Brett,

Generally though, if you had to choose, would it be better to have higher RAM or more cores/computing power, when it comes to optimization?

-Nick

[NinjaTrader_Brett]

Not sure I would favor one over the other, need to make sure you have balance of both so that you dont have any area to get bogged down.

-Brett

[koganam]

Hi Brett,

Generally though, if you had to choose, would it be better to have higher RAM or more cores/computing power, when it comes to optimization?

-Nick

Generally, with today's processor speeds, the CPU spends an awful lot of time twiddling its thumbs while waiting for data to be moved. You want a reasonable number of cores, depending on how much data you are crunching, then go for max memory. There have been quite a few tests (sorry, I cannot give you exact sources offhand), that have shown that diminishing returns come quickly with cores. In fact, I remember seeing one study, where admittedly, the processing was heavily dependent on shifting data, that showed that moving from 2 cores to 4, produced less than a 8% decrease in time spent per run.

I guess what that shows is that which gives you the most bang for the buck depends on the data set. For NT, I am inclined to go for memory, rather than CPU cores. Remember that NT uses a lot of DataSeries etc, which are fancy lists being held in memory. However, you do have to take into account the constraints on how much memory is used when you are using a .NET application.

All of which is to say that I fear you may not really get much for your buck either way by going to the cloud. .NET memory constraints, and the kind of data crunching that happens in an NT optimization, kind of negate the advantages of throwing more computing power at NT, than is present in a typical trader's station today.

[sledge]

You already have:

# of Cores
4# of Threads
8

Is your machine hurting in any instance? Out of memory? Can you still watch blueray and burn a dvd at the same time or play a game with out issue?



There's a few complaint posts about cpu usage only at 33% or something with many cores.

Hi,

My home PC is an i7-920 @ 2.67Ghz, with 12GB RAM running Win 7 64-bit.

Optimization seems to take a long time on my machine, so I was wondering if it would be faster if I lease some processing power on Amazon's EC2 cloud server.

The 2 options I'm looking at is either high RAM or high CPU resources (both run Win Server 2008 R2):

1) High-Memory Double Extra Large Instance 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform

2) High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

My question is:

1) Would NT's Optimizer be able to use the multiple cores in Amazon's virtual server?

2) If #1 is possible, then from a cost-benefit standpoint, would higher RAM or higher CPU (more virtual cores) be preferred?

3) There is also a Cluster option from Amazon: Cluster Compute Quadruple Extra Large 23 GB memory, 33.5 EC2 Compute Units, 1690 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet - would NT's optimizer able to fully utilize the resources available?


Thanks!

-Nick

[nicbizz]

Thanks for all the info and advice.

I've noticed that there are many instances when my RAM usage goes to 90+% whereby CPU stays at around 30%. And then, there are other times when both stay at above 60-70%. I suppose there are some areas where RAM becomes a bottleneck, and the CPU stays idle until it catches up.

I'm gonna run a few test using the biggest, meanest option on Amazon EC2, and see if there are any significant improvements in time needed to complete optimization. As it stands, my Strategy that's attempting to optimize all 500 stocks of S&P 500 could not be completed even after 4 days of running on my home PC.

-Nick

[hazza147]

Hi Nick,

2 Things:

Firstly how is testing going with the Amazon EC2 cloud solution?

Secondly, can a strategy with so many parameters that you clearly have in place for it to take 4 days for optimization really work in the long term? I'm very interested to know and if possible can you attach an example screenshot of your Cum. Profit?

I'm intrigued to say the least!

Regards,
Harry

[wallsteetking]

Hey I already tried this and it did not work for me. I choose the best one top of the line from AMZON EC2.

[wallsteetking]

hey if you figure it out please let me know thanks

[nicbizz]

Forgot I had this question outstanding.

Yes, optimization worked quite well using Amazon EC2 - I went with the High-Memory Quad Extra Large, which is 26 CPU units (8 cores), and 68GB RAM.

Can't say for certain if it runs faster than my physical PC, but it gives me the option of running a few optimizations concurrently while still leaving my PC available for trading and other uses.

Be prepared for the high costs though. Amazon nickels and dimes you on everything. In 1 week, running 3-4 different machines at various time landed me with a bill of a couple hundred bucks.