optimization engine crashing.

[rtwave]

people with nt,



i had had this issue before and had already reported it to nt.


i have been running optimizations on different instruments. and i have been hitting some limit of how much data nt can process. here are some examples:


this is the nq contract, minute bars from 202001 to date:





the process crashed and results are incomplete.


nq contract, minute bars from 202004 to date.





much better results.


nq contract, minute bars from 202005 to date.




even better credibility with these results.



now, nq contract, tick data from 202001 to date.




abject failure.


nq contract, tick data from 202006 to date.




this does work.



and i have had similar results when working with nasdaq data.


strategies can be optimized without problem on twtr data, both minute and tick.

however, when working with aapl data, minute data will work fine going back as far as 202001, but tick data will be too much and will cause crashes as those illustrated above. i don't have the time to create screengrabs for each occurrence, but it is quite evident that there is a limit of how much data nt can process.


at the moment i can solve this by reducing the number of months to work with, but this is not ideal as an optimization is more robust as more data is available to process. for the instruments with very little data i will have to corroborate my results with a much crappier platform i also use that is severely limited.



very well, this is the situation, nt can easily replicate and investigate these results with the instruments i mentioned and sample strategies that i have provided. regards.

[NinjaTrader_Kate]

Hello,

Thank you for your post.

Are you running optimizations using the Default Optimizer option on your strategy?

The NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible, and it is advisable not to trigger an optimization during a time where you would need to be using the PC. How much data you can successfully perform an optimization on would be somewhat dependent on your PC resources. However, there are a number of ways in which you may be able to reduce your CPU and memory usage during an optimization run.

I can't quite tell how many parameters you are optimizing your strategy on, but strategies with a relatively large number of parameters for optimization can multiply the impact on your PC resources.

When working with strategies with a large number of parameters, avoid using "1" as the increment value for the parameters, to avoid forcing the optimizer algorithm to run the maximum number of permutations. Changing the increment value to as little as "2" can cut the number of permutations in half.

Also, you may want to try testing using the Genetic algorithm instead of the Default. The default algorithm tests every single combination of parameters within the optimization range to find the best possible result. For more complicated optimizations this can take some time to complete, and depending on the amount of time you're iterating over you could see the optimization not complete as you've seen. The Genetic algorithm, on the other hand, can be more useful for more extensive optimizations, because it doesn't test every combination - instead, it works on something similar to evolution via natural selection.

Genetic optimization will find the approximate optimal solutions, but since it doesn't test every possible combination it's not guaranteed that the optimization would be the absolute best.

Here's a link to our help guide that goes over Genetic optimization:

https://ninjatrader.com/support/help..._algorithm.htm

If this is a strategy you have coded yourself, there is also the IsInstantiatedOnEachOptimizationIteration property you can set in State == State.Configure.

The default behavior of the Optimizer is to re-create the strategy for each optimization run. However, this process requires more time and computer resources to return results, which could impact the amount of time it takes to run an optimization and whether it's able to be completed.

When this property is set to false, the strategy is re-used to save time and computer resources. When this is set to false, internal properties are reset to default values after each iteration, but it is possible that user-defined properties and other custom resources may carry their state over from the previous iteration into a new backtest run. To take advantage of performance optimizations, developers may need to reset class level variables in the strategy otherwise unexpected results can occur.

You can find out more information regarding this property in our help guide link below:

https://ninjatrader.com/support/help...niteration.htm

Please let us know if we may be of further assistance to you.

[rtwave]

thanks.



i do always use the genetic optimizer.


[now, changing an increment from 1 to 2 will indeed have a linear effect on one single variable, however, if this is done for multiple variables simultaneously this effect will compound and the total combinations will decrease approximately by the inverse of a power function. jaja. this is just about terminology but i want to be very proper.]



nt support had already mentioned to me several months ago the possibility of creating versions of my strategies that would not instantiate for every single iteration. however, there are no easy to work with samples for such an undertaking on nt's documentation so i have never been able to create one such strategy. perhaps i will look into this as soon as i have a chance.



i had run into this kind of incidents before, so i have already emailed Jim a strategy similar to the ones i use and explained how to replicate these crashes. it would be nice if nt could take a look at these occurrences.



very well, regards.

[rtwave]

people with nt,



i reported these incidents weeks ago and these crashes continued frequently even on strategies which i had coded a while ago and did not have issues previously.


i was asking the platform to optimize data from a starting date and always up to the latest date, but i stopped receiving futures data since the end of january, so the platform did not have any data to work with after that month.


i am now running optimizations from a starting date and only to the end of january. this has been a big difference, i have been able to keep crashes like these in check for the most part.




however, i can confirm that all my strategies which are coded to liquidate all positions at the end of the day (or at the end of fridays only) and then restore (or not restore) these positions after the next session has started, do frequently cause the platform to crash when running an optimization process. a strategy will have no problems if it does not include commands to liquidate - restore positions at the close - open of the session, but then if i add that code, the strategies will become very problematic to optimize. also, including code to use a stop loss order and then move it up to a breakeven stop after a position has been made to move in my position will also overwhelm the platform. so, these issues can be considered to be abundantly confirmed.

[NinjaTrader_Kate]

Hello rtwave,

Thank you for your reply.

We would need to see a reduced test script that can reliably reproduce the crashing when optimizing for us to review and potentially give to our development team to help them resolve the issue if it is determined there is a bug causing the problem. If you can provide a script for us to reproduce this with, we'd be happy to test on our end.

Thanks in advance; I look forward to assisting you further.

[rtwave]

i had already reported this kind of crashes to nt some time ago. i tried to document these incidents as best as possible and exchanged several emails with Jim at nt. i sent nt a sample strategy and descriptions of the conditions which cause the crashes. the last opinion of the people at nt about these situations was that i was optimizing over a wide range of parameters and over a lot of data. well, if nt knows what are the limitations of their platform in terms of parameters and data, they should make them well known so that no one would waste any time trying to get the platform to do what it could never possibly accomplish.



and the exit on session close setting is a good illustration of the situations i'm talking about. i can work with some of my strategies without problem but if i turn this setting on, then the platform will be almost guaranteed to crash eventually.

[NinjaTrader_Kate]

Hello rtwave,

Thank you for your reply.

I've located Jim's last reply to you from last year, and it appears he stated the strategy you provided for him was overly complex for us to be able to take to our developers:

This particular test case is proving to be very difficult to narrow down to a specific problem as there are still numerous factors involved.

If there is an issue with the platform, we will be able to illustrate it with the SampleMACrossover strategy. In the example provided, I can see a case where the Genetic Optimization does not compete, but there are still too many factors involved where a specific issue is not know.

The factors involved are:

1. Your strategy logic
2. Massive amounts of tick data involved
3. Data Series optimized.
4. Using Genetic Optimizer
5. Using Slippage

These factors will need to be eliminated if we are to create a test case that we can use to file a report. I suggest:

1. Use the SampleMACrossover strategy
2. Use Daily data or minute data so we do not have to download gigabytes of data to run a test on tick data. Keep the amount of data to test at a minimum so it is just enough to demonstrate the issue
3. Test if the issue is seen with Optimize Data Series checked/unchecked.
4. Test with Slippage set to 0 and again set to a value like 25.

Using the SampleMACrossover strategy with a small set of data and eliminating the factors above would be the best way to come up with something that can be investigated further. As of now it is not known what a specific issue is. We only have a a very large Genetic optimization over a very large amount of data with several factors involved.

To have us investigate a matter with the Strategy Analyzer, we will need a simple test that clearly describes a specific issue. For example, we would need something like:

Testing SampleMACrossover with these settings gives us a good result. When I change X (single item) we can see the issue first hand.

We look forward to assisting.

I would concur with Jim here, as stated, in order for us to be able to have the development team look further into this we need a simple script that reproduces. If you can reproduce with Sample MA Crossover or a simplified version of your own script, we'd be happy to try to reproduce.

Please let us know if we may be of further assistance to you.