What is the most important performance standard when optimizing?

[Greenstream]

I'm optimizing on max profitability, but comparing the resulting Total net profit to other resulting Total net profit. I'm also comparing the performance standard "probability" against other probability results to see what seems more likely. What I'm wondering is how important is probability comparison. For example, if I have two results, one with a Total net profit of $2,000 with a probability of 10% and another with a Total net profit of $200 and a 35% probability; which result is most important? If there's 10% prob of $2,000, put it in another way, there's a 90% probability that it will be less than $2,000. I understand everyone has their own performance standards that are important to them, so each answer I get may be different - based on personal preference. Also, what is the most important performance standard to maximize profit?

[NinjaTrader_Jim]

Welcome to the forums Greenstream!

Our Support staff won't be able to suggest specific statistics that you should follow as this borders on trading advise, but you may reference our statistics for these metrics at the link below.

https://ninjatrader.com/support/helpGuides/nt8/optimization_fitness_metrics.htm

We'll leave the thread open for any community members that wish to share their insight.

[rtwave]

Greenstream,


i have been creating strategies for a long time.


i don't find any of the most widely used metrics very useful. i always optimize for net profit and prefer strategies that have a consistent diagonal upward trajectory created by numerous trades.


the most important thing you have to do is to keep at least 20% of data out of sample to run all your optimization processes and then incorporate that data to evaluate how your strategies would have performed for real. i have posted about this on other public fora and can send you the links via private message.

[linuxguru]

Greenstream, et al.

I have also noticed that none of the individual OptimizationFinessess are ideal (if there is even such a thing). However, I have found that Max Strength appears better than most. If you examine the source, you can see that it takes into account profit factor, R^2, number of profitable trades and filters out non-profitable runs as well as runs that make zero trades (which causes profit factor to be equal to 99 which is misleading). You might want to take a look at it.

Also, I have started with this and added a few more tests to create fitness tests that help narrow down factors that I care most about. If you know how to code, you might consider doing the same.

[rtwave]

i have given some thought to the idea of creating a new evaluation metric.


the closer a performance graph was to this trajectory below, the better.




this metric would give higher scores to strategies that consistently built up profits like in the image above and would punish big drawdowns (there could be a lot of ways to do this. maybe deduct the sum of applying some power to all trades that lose significantly more than the average winning trade).


anyway, i'm really busy at the moment setting up two different automated trading systems and also have some other pending projects, but this is something that i will tackle when i have a chance.

[NT-Roland]

Hi Greenstream,

I couldn't agree more to rtwave as to the importance of out of sample tests.

Your computer will almost always be able to "find" something if you expose all historical data to optimizations. Completely useless.

Unfortunately, NinjaTrader does not (yet?) seem to offer the functionality to do the following in ONE go:
1) Automatically calculate the start and end of optimization and control period based on user setting (e.g. 75% vs. 25%),
1) Run an optimization for the 75% period and automatically check these settings in the (unknown) control period of 25%,
2) Show the results for the optimization period and the control period in one strategy analyzer window side by side, with
3) Display of Analysis in compare mode (CumNetProfit, CumNetDrawDown, etc.) for optimization vs. control period.
Thus, you will have to do this manually (or use other software).

As to optimization "fitness". I wouldn't want to choose only one of them. Here are the good news.
There is no need to limit yourself in this respect. You are free to create your own (custom) optimization fitness in NinjaScript.
By doing so, you can combine several parameters you consider important and weigh them as you like in ONE (custom) fitness which best suits your individual taste.

NT-Roland

[rtwave]

NT-Roland,



man, this very same request you make of the people at nt and some others, i have made multiple times through different channels.


it is crucial for those of us who do use nt and have to bear with these serious limitations to push for the most important improvements to the platform.


i have created some threads in the suggestions forum, please check them out and state your support for these improvements there:

https://ninjatrader.com/support/forum/forum/suggestions-and-feedback/suggestions-and-feedback-aa/1094592-there-is-a-lot-that-could-be-improved-with-nt-7-and-8-s-optimization-engines

[linuxguru]

rtwave, et al.

Rsquared measures how consistent the price follows a line from the beginning to the end. However, there is no guarantee that it will be a positive slope. Max Strength uses Rsquared, but also takes into account if profit factor is at least 1.0 and that you had at least one trade.

[Greenstream]

Hi folks. I now have two bots. I cannot seem to optimize on the second bot. Is it likely that I cannot do so because the creator of the bot left in old information from the first bot in the second bot? Or, why is this happening? (NOTE: I know the creator left the old conditions in the bot because I opened the script as if I was going to edit it, but I don't know enough about deleting things without messing anything up. I am more than will to do this with some guidance if someone has the know-how.)

What it does is start out like it's going to optimize; says it will take 13 minutes to run, then runs about two seconds and stops without giving any idea results. Thoughts?

[linuxguru]

If you open the output window, you might get a hint as to what the problem is. It or one of the indicators it is using is probably erroring and stopping the strategy.

Feel free to PM me if you would like more help and I'll see what I can do.

[Greenstream]

Asked a question that I deleted

[linuxguru]

It appears that Ninjatrader's data servers are down this weekend, at least for me. When I try to test a strategy, it thinks about it for a few seconds and then stops with no results. Not exactly the best feedback.

Apparently they have been doing maintenance on their servers over the last few weekends.

[NinjaTrader_ChelseaB]

Hello linuxguru,

Typically there is maintenance on the weekend with most brokerage servers on Friday night and / or Saturday.

Most brokerages will have the maintenance completed Sunday before market open. You will need to restart NinjaTrader if NinjaTrader is open during this time.

Backtesting with the Strategy Analyzer only requires historical data. Do you have the historical data needed for the test cached and showing on the Tools > Historical Data window > on the Edit tab in the Historical section?

[linuxguru]

Thank you for your answer. I didn't have the data cached, but I managed without it. It is unfortunate that the system is down when it is the best time to backtest and work on systems. However, being an IT guy myself, I can see the other side of the coin. I believe it would be more professional to post something to the news page when the system will be down, but if that is often, then I guess there's no need for weekly posts.

Thanks.

[MojoJojo]

Although I don't have that much experience with strategies yet, I do find net profit and consistency the best variables just like rtwave mentioned. Other than that number of trades is interesting and max draw down is an indicator of consistency.

Theoretically a strategy that trades less is preferable to one with a larger number of trades with a similar profit, but the latter makes the result more likely to be not to be just curve fitted.

Exceptions could be assuming 1:1 RR, a strategy might have a high win rate but because of one extreme event max draw down is bad. Of course such an event should be handled but the approach might actually be more promising than what the results indicate.

However there is one other thing to optimize for, back testing speed, slow performance will lead to fewer cases tested and a tedious process.
After ditching tick based indicators, identifying StdDev in my case as the major culprit and replacing it with an approximation, iteration time has improved a lot.

Unfortunately often the results over a short time span are promising but they fail when testing over a larger sample size.
Dunno yet if it's a case of fooled by randomness or having to model macro market conditions better.