Optimizing on a 2xXeon with 48 cores is 4 times slower than on a household I7!

[nikolaalx]

Hello,
I have been trying to understand and build a server that I could use for backtesting and optimizing.

I bought a Dell Precision T7610 which has a
2x Intel Xeon E5-2697 v2 @ 2.69 GHz - 24 cores, 48 threads
Just for the reference, a single Xeon E5-2697 v2 CPU has a score of 9338 multi-thread and / 1742 single-thread according to Passmark (https://www.cpubenchmark.net/cpu.php....70GHz&id=2009)

The retail price of the Xeon CPU alone until Apr 2019 was approx $2500 per on a single CPU.

On the other side, the I7-4790k has 4 cores / 8 threads and a 8096 multi-thread score / 2485 single-thread score according to Passmark (https://www.cpubenchmark.net/cpu.php....00GHz&id=2275)


Considering the much higher productivity and performance capability of the server, I was expecting that the NT7 would be able to perform much faster.

I was surprised to see the opposite. An optimization that normally takes 25 minutes to complete on my I7 is taking approximately 1.5 hours to complete on the 2x Xeon server with the exact same settings.

I am using genetic optimization.

I was further surprised to see that on the Server, NT7 was utilizing only a single thread which resulted in ~4% utilization of the CPU capacity.
On the I7, it was again utilizing a single thread to near 100%, which resulted to ~12.5% CPU capacity.

Why is the optimization not utilizing 100% of the threads? Not being able to utilize more than 5% on a modern computer is really frustrating.

I am attaching a screenshot of the single thread resource meter from the Xeon Server.

What kind of computer do I have to buy to be able to speed up the optimization?


p.s. I also noticed that If I buy a $50 VPS and a $200 VPS, optimization speeds are almost identical. I assume this is because of the poor utilization of the multi threading again.

[bltdavid]

What kind of computer do I have to buy to be able to speed up the optimization?

This kind of question has little merit unless you positively state which version of
NinjaTrader you intend to run on it. Because NT7 vs NT8 matters here.

Assuming NT7, my opinion is that the higher priced multi-core Xeon CPUs are wasted w/NT7.
Just get the best Skylake (or later) i7 you can -- forget the i9's for NT7-specific workloads.

However, if this machine will be running NT8 (or will in the future) then CPUs with many cores
should have a distinct advantage on NT8.

See this post,

https://ninjatrader.com/support/forum/forum/ninjatrader-7/platform-technical-support/89513-best-computer-system?p=750357#post750357

[bltdavid]

After the Skylake i7, the best purchase is the fastest disk setup you can afford,
such as a 2 TB SSD, esp NVMe M.2. Make sure this is your boot drive.

Then, a huge amount of RAM -- 32GB is probably a good minimum.

[Downloading huge amounts of historical data for multiple instruments can
really use up a lot of RAM -- so a paltry 16GB machine may be burdensome
on enough occasions to make it quite noticeable. Btw, the 32-bit version of NT7
hangs on these huge downloads, it easily hits 3GB+ memory usage and then
just stops working, so avoid that problem by always running the 64-bit version
of NinjaTrader 7]

[nikolaalx]

This kind of question has little merit unless you positively state which version of
NinjaTrader you intend to run on it. Because NT7 vs NT8 matters here.

Assuming NT7, my opinion is that the higher-priced multi-core Xeon CPUs are wasted w/NT7.
Just get the best Skylake (or later) i7 you can -- forget the i9's for NT7-specific workloads.

However, if this machine will be running NT8 (or will in the future) then CPUs with many cores
should have a distinct advantage on NT8.

See this post,
https://ninjatrader.com/support/foru...357#post750357

The issue with migrating to NT8 is that it is far more advanced and far less user friendly.

Also, the necessity to refactor almost 100% of the code makes something which is impossible for me. I consider NT8 a completely different product than NT7 as the barriers for entry (for me personally) are far too great.

Having this In mind, I will need to figure out the most effective way to optimize while continuing to work on NT7.

I really think that the genetic optimizer on NT7 does not use multiple threads. I was able to validate this by doing an experiment with renting 7 different cloud compute windows VPS.

For the record here's the performance:

1 CPU ($6 per month) - 38 minutes (running at 95% of CPU capacity)
2 CPU ($24 per month) - 28 minutes (running at 51% capacity)
3 CPU ($48 per month) - 28 minutes (running at 38% capacity)

For comparison, the same optimization on my 2x Xeon Server takes approximately 1.5 hours to complete.

And on the I7-4790k it takes ~16 minutes

I am using a genetic optimizer with 250 generations x 250 size each (62,500 total output) on a high frequency model with little data (3 days of 30 sec bars). Disk size and RAM are not really a factor in my case. At least I haven't observed any major difference.


The threading utilization of NT7 is a mess. I really wish the team could patch NT7 so that it has better efficiency with multi-threading...as otherwise, the whole software system would very soon be technically worthless as it would be impossible to use it with modern computers and would fail to meet the needs of people working with more intensive optimization.

I find NT7 brilliant since it has the perfect balance for flexibility, ease of use, and generally little to no coding required for any person to design his own strategy. I really recommend to NT staff to consider an upgrade of NT7 and stop asking people to move to NT8, because NT8 is an entirely different product. This is also the reason why you see less usage on the NT8 compared to NT7.

[bltdavid]

Searching for more information, I found this interesting post which mentions Process Lasso,
which sounds like it could be useful, esp for NT7.

They have both Free/Pro versions -- Pro version is $26 for single PC, or $36 for all PCs
in your house.

I've never used it myself (heck, I'd never even heard of it), but it seems like a worthwhile
experiment.

[bltdavid]

The threading utilization of NT7 is a mess.

The threading code is 10-15 years old.
Written during a time when the emphasis on multiple cores was just, uh ... 2 cores.
Which meant (to most development efforts of that era) the thinking was:
"Why bother? Let's do what we can with anything simple, but the best
CPUs only have 2 cores anyways, and 4 cores won't be mainstream
for many years."

I really wish the team could patch NT7 so that it has better efficiency with multi-threading...as otherwise, the whole software system would very soon be technically worthless as it would be impossible to use it with modern computers and would fail to meet the needs of people working with more intensive optimization.

Upgrading. NT7. Will. Never. Happen.

The entire focus is NinjaTrader 8. Full stop. End of story.

I find NT7 brilliant since it has the perfect balance for flexibility, ease of use, and generally little to no coding required for any person to design his own strategy. I really recommend to NT staff to consider an upgrade of NT7 and stop asking people to move to NT8, because NT8 is an entirely different product.

I, too, love NT7 and I don't intend to part with NT7 anytime soon.
The transition sucks, but you need to plan for it and get started.

Don't underestimate NinjaTrader 8.
The future of NinjaTrader, LLC is NinjaTrader 8. Full stop. End of story.
There is no going back.

On a bright note, NinjaTrader management has promised to support NT7 for years to come.

This is also the reason why you see less usage on the NT8 compared to NT7.

Ridiculously untrue.
Third-party NT8 products abound.
Many companies who previously sold NT7 products now only their sell NT8 version.
The world is moving to NT8 all around you, haven't you felt the ground shifting?

[bltdavid]

Also, the necessity to refactor almost 100% of the code makes something which is impossible for me.

Why do you say "impossible"?

By "impossible" do you mean "extremely difficult" or "extremely expensive"?

If you're not a programmer, there are many consultants who could help you.

When migrating to NT8, nothing is truly impossible.

Have faith, my friend.

[nikolaalx]

Why do you say "impossible"?

By "impossible" do you mean "extremely difficult" or "extremely expensive"?

If you're not a programmer, there are many consultants who could help you.

When migrating to NT8, nothing is truly impossible.

Have faith, my friend.

Well it took me a 2-year painful learning curve to get to a point where I am able to work efficiently with NT7. I don't have the time resource to go through that with NT8. Having the flexibility to design -> code -> test -> implement -> repeat as a sequence is what I find unique as I am sometimes doing 12-hour work sprints...and using the time when I am not awake to do heavy backtesting.

When working with a consultant, the whole cycle breaks and it just doesn't work for me. What normally takes days of work, turn into weeks or months of work...and the costs are also insane.

Anyway, I guess there is no way to optimize the efficiency of NT7.

How about any hacks to do that?

[nikolaalx]

Searching for more information, I found this interesting post which mentions Process Lasso,
which sounds like it could be useful, esp for NT7.

They have both Free/Pro versions -- Pro version is $26 for single PC, or $36 for all PCs
in your house.

I've never used it myself (heck, I'd never even heard of it), but it seems like a worthwhile
experiment.

So you think that it could force the NT7 to a higher affinity? I double-checked and by default it has all threads enabled. Not sure what higher optimization it could achieve.

I will give it a try though, thanks.

[bltdavid]

This might be helpful,

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/80949-here-s-a-script-to-convert-indicators-and-strategies

[bltdavid]

Well it took me a 2-year painful learning curve to get to a point where I am able to work efficiently with NT7.

I hear you.

Painful as the learning process will be, even if you do it part-time,
I'm suspect you could do your NT8 migration in well under 6 months.

Probably much faster.
It may feel slow, but your knowledge will start steam-rolling.
Besides, much hard-won NT7 knowledge is still useful for NT8.

On the plus side, there is now a huge amount of working NT8
code to learn from. Thankfully, much more now than before,
I mean, I'm kinda glad I've waited ...

I intend to start my own 6-months of "hell" soon ...

Good luck!