Start Strategy Analyzer using NinjaScript

[staycool3_a]

Hello.

This might be a tough question... I think.

My strategy has about 10 variables that I did an optimization on. I just got done with some in-sample optimization. I ran through about 300k iteration. I printed the values that I was interested in to the output window for each iteration. Did the analysis I wanted outside of NT. Now I’ve got about 1k combination of the 10 variables that I’d like to do out-of-sample testing on.

Is there a way to update the set of my variables and run the strategy analyzed using ninjascript? Or is the only option to run each of the combination one-by-one?

I was thinking I could set individual combination of my variables within a bool. The bool at the end of each strategy analyzer conclusion would kick off the next bool and restart the strategy analyzer for the same date range.

So:

If bool a=true
{
variable a=1;
Variable b=2;
Variable c=3;

-Somehow run strategy analyzer for 2012-2018.

-If today is last day of strategy analyzer run, then,

set bool b = true
}

If bool b= true
{ run strategy analyzer for the same date range as above with new variables
....
}


I think this shouldn’t be hard as long as there is a way to kick off strategy analyzer using ninjascript and then trapping it in a loop that goes from bool to bool

Thoughts?

[NinjaTrader_Jesse]

Hello staycool3_a,

Thank you for the question.

In this situation, there is currently no way to programmatically invoke the analyzer as this is a manual tool. I can put in a feature request for this.

As far as setting defaults you could do that. If you have a list of parameters that were created outside the platform you could have the strategy parse that for some defaults. This wouldnt allow for minimum or maximum values to be set for the optimization but you could set the initial value.

I look forward to being of further assistance.

[staycool3_a]

Hello staycool3_a,

Thank you for the question.

In this situation, there is currently no way to programmatically invoke the analyzer as this is a manual tool. I can put in a feature request for this.

As far as setting defaults you could do that. If you have a list of parameters that were created outside the platform you could have the strategy parse that for some defaults. This wouldnt allow for minimum or maximum values to be set for the optimization but you could set the initial value.

I look forward to being of further assistance.

Yea that would be huge if there is a way to do this. B/c these optimization engines that are available to people crunch millions of data points and give us the "top 10/20/100." That's just not enough. Because

1) there is still no way to test all the top results individually with ease
2) you do need to find some sort of pattern/statistical edge from the optimized variable and then rank them on in-sample data based on some measure
3) then test as much as possible.. all the combinations on out of the sample data to see how the rankings compare to in-sample testing.
4) then picking the consistent ranked for further testing

there's gotta be a way.. ugh

Isn't there a way to add variables to the strategy separated it by semi-colons and have the optimizer test them all.. similar to how the optimizer would do it

var 1 = 1;2;3;7;8;1.2

[NinjaTrader_Jesse]

Hello staycool3_a,

Thank you for the reply. I will include these details in the feature request.

I have two requests in now:

SFT-1051 for optimizer attributes.

SFT-2193 for automating the strategy analyzer.


Right now there is no property or process that would be suggested to use a delimited list as you noted. I will put this in the request as well or a way through code to define the optimization values.

Because the optimizer is a manual tool it would require you to manually select the script to run its defaults. The properties the script has would be displayed in the UI with their values but there is currently no attribute which delegates the range of the optimization that could be applied to each property. I will put this in as a request as well for better control over the range of optimization.

I look forward to being of further assistance.

[staycool3_a]

Jesse, I tested about 109 combinations today and my fingers are hurting.

I have a clever idea that might suffice but it depends on the workings of walk forward testing methods in NT

What I wanted was this from the beginning:

Test all possible combinations. Do some testing outside of NT and then pick top 100k for out of sample testing in NT. Which we can’t do as we discussed.

Instead, is it possible to instruct NT to test all possible values during in-sample and then test the same combination during out of sample instead of taking the “best” from the in sample?

As it stands, how many combinations are tested on out of sample data? The help guide just says it takes the “best” from in-sample based on optimization measure.

Is it possible to instruct NT to test all combinations in out of sample?

If not, bc NT is taking the “best” combinations from in-sample based on optimization fitness function... what do you think happens if I create a optimization variable that only returns zero. How does NT decide between “best” in that scenario. Would it then be forced to test all combinations during out of sample data as well?

I know you might be wondering, why doesn’t this guy just run the optimizer on all data if he’s wanting to test the same combinations on the out of sample data. Answer: I’m trying to look for consistency. And I’m interested in seeing how every single combination that was used during in sample testing... how that individual combination performed during out of sample testing. My definition of consistency: I do not want to see any of the top 20 combinations flip-flopping from the bottom 20th percentile. I want to visually see the bottom performing.. stay and bottom and top performing.. stay on the top. Printing these should be easy bc I can hard code the in and out of sample dates and printing the values from my strategy and print the performance measures I’m interested in. If only there’s a way to test all the in-sample combinations SEED in out-of sample data...

[NinjaTrader_Jesse]

Hello staycool3_a,

In this case, this is kind of "what you see is what you get" item because the optimizer is a manual tool. The optimizer is basically going to be able to do what you can see and use in the GUI and nothing outside of that.

In regard to testing all possible values, there is no way to change the way the walk forward works or reports in general so if you are seeing a case that you want to complete or view data for but cannot that would be the something we can feature request.

The walk forward will only show the test periods along with the found settings for that period. You can find how many total iterations or permutations were run next to the "Run" button, however, this does not specify the total for in sample and out of sample which is something I could feature request if that is needed.

For the question surrounding a 0 value, this is something you can test if you feel this situation will come up for you but I would not have a direct answer for this hypothetical without testing it first.




Please let me know if I may be of further assistance.

[staycool3_a]

Ok. Thanks.

Can you clarify the below?

Scenario 1)

Optimizer runs 1000 iteration. Takes the top 5 based on performance during in sample and then tests them in out of sample.

Scenario 2)

Optimizer runs 1000 iteration. Always take the the settings that rank 1. And then tests that during out of sample.

Scenario 3)

Optimizer runs 1000 iteration for both in and out of sample. Then just displays the top performer during out of sample

Which one is NT doing? I’m just asking for clarification bc this is not in the help guide.

Also, even though I selected the “keep Best # result” to 5....it always ONLY shows 1 result for the walk forward. Why?

[staycool3_a]

Ok. Thanks.

Can you clarify the below?

Scenario 1)

Optimizer runs 1000 iteration. Takes the top 5 based on performance during in sample and then tests them in out of sample.

Scenario 2)

Optimizer runs 1000 iteration. Always take the the settings that rank 1. And then tests that during out of sample.

Scenario 3)

Optimizer runs 1000 iteration for both in and out of sample. Then just displays the top performer during out of sample

Which one is NT doing? I’m just asking for clarification bc this is not in the help guide.

Also, even though I selected the “keep Best # result” to 5....it always ONLY shows 1 result for the walk forward. Why?

Well i figured out how to see the other ones. ALl you have to do is right click and click on view optiomization to find the "keep X results"

b/c it keeps other settings as well, i have a hunch that it runs different settings on in sample and then test them all on out of sample and displays the results. if that's the scenario, then its perfect b/c i can print values at different times.

id appreciate it if you can confirm which scenario is being utilized in NT

[NinjaTrader_Jesse]

Hello staycool3_a,

I will need to see if product management can provide more information here as the logic for this tool is internal. For items that are not documented, we would generally need to put in a feature request for that to be added to the help guide. If I have further information supplied on this question I will reply back with that information.

I look forward to being of further assistance.

[staycool3_a]

Hello staycool3_a,

I will need to see if product management can provide more information here as the logic for this tool is internal. For items that are not documented, we would generally need to put in a feature request for that to be added to the help guide. If I have further information supplied on this question I will reply back with that information.

I look forward to being of further assistance.

Jesse, I actually will no longer need the answer to my questions.

I ended up doing this:

b/c my whole goal is to run my in-sample optimized variables in an out of sample data... and b/c my in-sample combinations are about 1000 combinations and manually testing them was a pain. I did this:

i create a dummy variable
then i added this to my strategy:

if dummy variable = 1
{
variable 1 = x
variable 2 = b
and so on..
}

else if dummy variable = 2
{
variable 1 = c
variable 2 = s
and so on
}

all the way to if dummy variable = 1000. Then i just ran the optimization on my dummy variable for a 1000 iteration... and printed ending values for each iteration!

[CB001]

Jesse, I actually will no longer need the answer to my questions.

I ended up doing this:

[...]

all the way to if dummy variable = 1000. Then i just ran the optimization on my dummy variable for a 1000 iteration... and printed ending values for each iteration!

Hi staycool3_a, has your solution proved reliable? I'm looking to do a very similar routine in order to automate the backtesting process within NT8