Faster Optimization

[waterhorse]

Hi there,

Is there a way to make the optimization goes faster? Can I hook up multiple computers in parallel to perform the optimization?

[Trader.Jon]

FAST processor ... http://www.youtube.com/watch?v=hQBDn...eature=related

[waterhorse]

FAST processor ... http://www.youtube.com/watch?v=hQBDn...eature=related

Thanks for the reply John. I have a quad 3GHz and the optimization takes about 6 hours. Faster CPU is such a brute force method. I think that Ninjatrader must move into 21st century solution with mass optimization calculations using multiple remote processor cores.

[jjmmg]

I'm workin on a parallel optimization, but before implementing any solution (CUDA, parallel libraries for .NET, mesagging, etc) I've found a problem I need to solve (applies to the messaging solution, the one I've decide to used). How can I insert the performance of an iteration executed in another computer into the main computer, for NT takes it into account. The other problem was how to get the performance result of an iteration, but that's already done.

More info in: http://www.ninjatrader.com/support/f...ad.php?t=38098

[J_o_s]

Hi there,

Is there a way to make the optimization goes faster? Can I hook up multiple computers in parallel to perform the optimization?

Perhaps you can use the genetic optimizer?

[Luger]

Perhaps you can use the genetic optimizer?

The issue with the genetic optimizer is that it may find local minimum and maximums and not the true max and min. Although much faster, it may provide sub-optimal results. However, I do use them when I've constructed something rather complicated and want a quick performance review, 15 minutes instead of 15 hours. It has its uses, just not for final optimizations in my opinion.

Also, if the strategy being genetically optimized contains "if" statements then it is forced to only use one CPU core. IF what I read somewhere else on this forum is correct.

[Trader.Jon]

The issue with the genetic optimizer is that it may find local minimum and maximums and not the true max and min. Although much faster, it may provide sub-optimal results. However, I do use them when I've constructed something rather complicated and want a quick performance review, 15 minutes instead of 15 hours. It has its uses, just not for final optimizations in my opinion.

Also, if the strategy being genetically optimized contains "if" statements then it is forced to only use one CPU core. IF what I read somewhere else on this forum is correct.

I have used the PHGenetic and it definitely can fill all 8 cores on my dual opteron. The strategies definitely have LOTS of ' if()' .. I replace enums/bools with 1/0 to make best use of the GO tool.

Like you I do wish NT could have facility to use GPU to help with optimization, as that would assist with default optimization like you want to do.

GO can get 'stuck' on local minimums and maximums and miss the 'higher elevations' but I feel thru multiple testing of variations in the logic of the strategy that 90% of the time the answer is in the ballpark ... I still havent progressed to the point to use View360,

http://www.ninjatrader.com/support/forum/showthread.php?t=28261&highlight=View360

but that is an option or at least to ouput each iteration to test again in Excel etc

TJ

[darckeen]

I dont think you could back test with a GPU and if you did the strategies would have to be extremely simpe. GPUs are good at doing lots of independent calculations and bad at doing linear control flow branching, which is what most strategies are.

[Trader.Jon]

I am not a scientist, but I am aware that the use of GPU in Genetic Algorithims has been proven to be extraordinary improvement in throuhput:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.93.1862&rep=rep1&type=pdf

Needless to say there is a need to have programming that can take advantage of that power.

Jon

[darckeen]

The issue isn't the genetic algorithm it's the fitness function.

Anyhows, to speed up optimization disable trade history and order management.

[Trader.Jon]

The issue isn't the genetic algorithm it's the fitness function.

Anyhows, to speed up optimization disable trade history and order management.

darckeen,

Thanks for the tip: can you provide a more explicit example so I can fathom what you are referring too please!?

Tia,
Jon

[darckeen]

A fitness function is how a genetic algorithm rates each population member, i.e. backtesting.

Backtesting is serially correlated, and GPUs are bad at serially correlated calculations.

[Justin300]

The fitness function is the criterion you are trying to "optimize on ...". For example, optimize on "max net profit" or "max sharpe ratio". There are many fitness functions not currently included in NinjaTrader, such as System Quality Number (SQN), Upside Potential Ratio, and Sortino Ratio. Basically, the fitness function is the criterion used to evaluate the quality of a trading system. Optimizing on different fitness functions will produce different "optimal" parameters for a trading system.

darckeen is absolutely right about GPUs. They excel at calculations involving vectors, like multiplying one list of numbers by another list. But they are not good at calculations involving loops where information needs to be processed sequentially, like in trading systems. For example, trading data needs to be processed sequentially because you need to know where one trade exits before another can be initiated.

[Trader.Jon]

Justin300,

I always thought that Genetic Optimization and Neural Nets were sampling types of calculations, (which was supported by the links to existing GPU uses) and GPU are useful for those algorithims

Hope I didnt mislead too many people