B8 Optimization Issue for Multi Instrument Strategy

[wuileng]

I have a multi instrument strategy which trades the primary instrument and hedges the position with an opposite position in the secondary instrument, SPY. This strategy uses no indicators.

I've been trying to run an optimization <DOW 30>. However, based on my output logging it seems that the optimizer is trying to load SPY as the primary instrument for a bulk of the optimization runs.

This is what it should look like:

Startup [SPYWeekly-BAC] ##### Starting up strategy.
Startup [SPYWeekly-BAC] Classes Setup -> Setup complete.
Startup [SPYWeekly-BAC] Logging Setup -> No summary logging requested.
Startup [SPYWeekly-BAC] Instrument List Setup -> Adding BAC.
Startup [SPYWeekly-BAC] Instrument List Setup -> Adding SPY.
Startup [SPYWeekly-BAC] Instrument List Setup -> Setup complete.

This is what the problematic output looks like:

 Startup [SPYWeekly-SPY] ##### Starting up strategy.
 Startup [SPYWeekly-SPY] Classes Setup -> Setup complete.
 Startup [SPYWeekly-SPY] Logging Setup -> No summary logging requested.
 Startup [SPYWeekly-SPY] Instrument List Setup -> Adding SPY.
 Startup [SPYWeekly-SPY] Instrument List Setup -> Setup complete.

My logging has a header in the format of [StrategyName-PrimaryInstrument] so you can see something is wrong. The results are erroneous and they also change every time the optimization is re-run (i.e. unstable). See attached screenshot for results. The results should be different but they aren't since SPY is being used as the primary instrument for all runs.

When running the optimization on single instruments (non-basket) the results are still unstable and change every time it is re-run. However, it doesn't seem to be loading SPY as the primary instrument.

When the secondary series is retained, but the hedging logic removed (i.e. SPY is added to the strategy but is not accessed/traded), the problem still remains.

When the secondary series is removed, the basket optimization works fine, and provides stable/replicable results. Backtesting is also fine and stable, even with the secondary series added.

If it matters, I have IsInstantiatedOnEachOptimizationIteration = true;

[NinjaTrader_ChelseaB]

Hello wuileng,

Thank you for reporting this.

What operating system are you using to test NinjaTrader 8?

[wuileng]

I'm using Windows 7 64 bit.

[NinjaTrader_ChelseaB]

Hi wuileng,

I created a simple strategy to test placing orders to the SPY as a secondary series.
I made a video to demonstrate testing this script.
http://screencast.com/t/ItoSLV6fCb

I would like to you test this script and ensure that the secondary series of SPY is added correctly and prints in the output window and also that the short trades marked secondaryEntry are only going to the SPY instrument.

[wuileng]

Made two simple changes to your script:

1. Changed EnterLong/EnterShort quantities to trade AVariable.
2. Added the following, to see what the primary instrument of the strategy is:

else if (State == State.Historical)
{
Print("Primary instrument = " + Instrument.MasterInstrument.Name);
}

http://screencast.com/t/9c8rCvSB7E

http://screencast.com/t/Ty7t0DJteW

In my first video, you'll see me perform a:
1. Basket optimization on DOW 30 - note the results/primary instrument issue.
2. Regular optimization on GS - I repeat this a few times so you can see the instability (see the top result drop from $15.75 to $10.25 with AVariable = 5), though at least it adds the correct primary instrument.
3. Regular backtest on GS - this works fine, with AVariable = 5 we should get $15.75.

In my second video, I edit the script to remove the secondary series. Optimization works perfect.

[NinjaTrader_ChelseaB]

Hello wuileng,

I want to be very clear about the behavior you are reporting.

Is the issue specifically that the Instrument.FullName is returning the secondary instrument?

In otherwords, the trades themselves are not what we are focusing on, is this correct?

[wuileng]

No, the main issue here is of course that the optimization doesn't work. I say it doesn't work because it is trading the wrong instrument in the case of basket optimizaitons (as demonstrated in my screencaps), and also that the runs are volatile for single instrument optimizations (different results for the same optimization when using the regular non-genetic optimizer).

I could care less about the reporting if the results were stable and valid.

I merely entered the Instrument.Name stuff to demonstrate that the strategy is not adding the desired instruments, and instead, just adding and trading the secondary insturment unintentionally.

[NinjaTrader_ChelseaB]

Hello wuileng,

I understand and I am able to reproduce the behavior using the original script by testing with an instrument list of only two instruments.

In the results, the second tested instrument in the instrument list had all trades placed to the secondary instrument with the name primaryEntry, though the code of the script does not allow for this.

I have reported the behavior to our development department. Once I have a tracking ID for this I will post in this thread.

Thank you for your working finding and reporting this

[wuileng]

Thanks for your help.

Please also note that the instability in single instrument optimizations (with a secondary data series) is a major issue which needs to be resolved as well.

[NinjaTrader_ChelseaB]

Hello wuileng,

I am unable to reproduce this same exact behavior when optimizing a single instrument.

May I confirm that with the original script I have provided, that you are able to reproduce this behavior when testing an individual instrument?

[wuileng]

I don't think you'll see any instability because your original script does not have results that depend on the inputs. This is why I made that change in mine, so that AVariable becomes the number of shares traded.

In my first video, you'll see this reproduced from about 1:05-1:25. Look closely at the results, and you'll notice they change on the last run. I find that this problem becomes more severe when running more complex strategies with more optimized parameters.

[NinjaTrader_ChelseaB]

Hello wuileng,

Are you 100% certain that the behavior is the same where the incorrect instrument is used for trades?

Are you reporting different behavior?

What is the behavior you are reporting?

[wuileng]

2. Regular optimization on GS - I repeat this a few times so you can see the instability (see the top result drop from $15.75 to $10.25 with AVariable = 5), though at least it adds the correct primary instrument.

No, from the screencap and my previous post, the problem with single instrument optimizations (when a secondary series is present) is that the results are unstable. This means that the same optimization can provide different results when run over and over, even though there are no stochastic/random processes involved in the strategy.

This is a problem because you don't know if the optimization results are valid and can be trusted.

It is not the same behaviour as the basket test because in this case it does add the correct primary instrument. However, I have strong reasons to believe that the problem is in some way related to the other because it only pops up when adding a secondary data series (as far as I can tell).

[rmt81]

This seems very similar to the issue reported in following thread

See http://ninjatrader.com/support/forum...ad.php?t=82148. The development team should probably consider them together.

I agree with the OPs that not being able to rely on optimization results from strategies with multiple instruments is a serious problem.

[NinjaTrader_ChelseaB]

Hi wuileng,

Just to clarify, when you mention that the results are unstable, you mean specifically that when an optimization is run and this reproduces results.

I am not seeing a screencap / screenshot included with any post other than the first post. However, there are two videos that are included with your post #5. I am not seeing clearly in either video that the results are changing each time the optimization is run.

Both videos appear to show optimizations with an instrument list selected. Neither video shows a single instrument being tested. In post #9 you mention the behavior happens with a single instrument.

The second video shows optimizations being run three times. The results do not change from either optimization.

I've attached two screenshots from your video. The first screenshot is after the first optimization run. The second screenshot is from the 3rd optimization run.

I am not able to clearly see what is being reported in these videos or what to look for so that I may reproduce the behavior and create a bug report.



rmt81,

The behavior where the secondary series places trades to the wrong instrument would not be the same behavior where after an optimization is run, running any backtest from the optimization results changes the results of that backtest as reported in item# NTEIGHT-9368. This should be reported separately and I have reported this.

Yes, optimization results should be reliable. In this Beta if we find that the results are incorrect, we report these as bug reports as soon as we are able to reproduce the behavior.

At this time, I am unclear of the behavior experienced by wuileng so that I may reproduce this.

Last, we have not been able to reproduce the behavior of NTEIGHT-9368 on Windows 7. So far this can be reproduced with Windows 10 and Server 2016.

If we are able to reproduce with Windows 7, I will want evidence of this, then I will need to reproduce consistently on my end, and then update the bug report that this can be reproduced with Windows 7.