8.0.2.0 - Huge Backtesting / Strategy Analyzer Bug

[wuileng]

So, in 8.0.2, within Strategy Analyzer - whenever you run a backtest on a single instrument strategy (I've tried with minute and daily bars), it seems like everything is actually executed twice.

Try running the strategy I attached on minute bars and see if you get the following output. In 8.0.1, you'd only see one instance of the output, rather than two.

11/28/2016 10:00:00 AM
11/29/2016 10:00:00 AM
FINISHED RUNNING
11/28/2016 10:00:00 AM
11/29/2016 10:00:00 AM
FINISHED RUNNING

When I ran a variant of the attached strategy on daily bars, this is what I got. Again, two instances:

12/5/2016 4:00:00 PM
12/6/2016 4:00:00 PM
FINISHED RUNNING
12/5/2016 4:00:00 PM
12/6/2016 4:00:00 PM
FINISHED RUNNING

My impression is many users will not find a problem here since the Strategy Analyzer window still displays the correct results for the backtest. However, having everything executed twice is a major problem for me (and potentially other users) because I use custom methods to output data to help me analyze performance (e.g. intra-trade drawdowns which NT doesn't do natively). Furthermore, when backtesting more complex strategies over longer timeframes, the outputs/execution become interleaved meaning that you can't isolate one instance of the output results easily.

In addition, when I ran a backtest on a multi-instrument strategy, I now see new output being shown as if the secondary bar series is being called (despite having a BarsInProgress filter). In the example below, the secondary bar series, VIX should not be seen at all, at wasn't in 8.0.1.0.

[SampleStrat3333-VIX] Session Open: December 1, 2016 @ 09:30:00 --------------------
[SampleStrat3333-SPY] Session Open: December 1, 2016 @ 09:30:00 --------------------
[SampleStrat3333-SPY] Session Open: December 2, 2016 @ 09:30:00 --------------------
[SampleStrat3333-SPY] Session Open: December 5, 2016 @ 09:30:00 --------------------
[SampleStrat3333-VIX] Session Open: December 2, 2016 @ 09:30:00 --------------------
[SampleStrat3333-VIX] Session Open: December 5, 2016 @ 09:30:00 --------------------
[SampleStrat3333-VIX] Session Open: December 6, 2016 @ 09:30:00 --------------------
[SampleStrat3333-SPY] Session Open: December 6, 2016 @ 09:30:00 --------------------

Needless to say, I'm pretty disappointed by the fact that NT8 is out of beta with all these new and existing bugs being reintroduced, and problems like freezes/crashing due to announcements making it technically unusable for reliable live trading.

[NinjaTrader_JessicaP]

Hello wuileng, and thank you for your report. I noticed that many of your settings were not done through code in the State == State.SetDefaults section. Would it be possible for you to send us a picture of the strategy configurations settings you used through the GUI? I am attaching an example.

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.
Click here for instructions
Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screenshot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.
Click here for detailed instruction

[wuileng]

Sure - please see the attachments.

One is the GUI screenshot you requested, and the second is the coded settings I used for one of my strategies I also tested (i.e. SampleStrat3333 in my previous post).

[NinjaTrader_JessicaP]

Thank you for your report.


This behavior was confirmed on our end. We will be investigating this further. Please keep an eye on the NinjaTrader 8 Release Notes page for updates and bugfixes.

http://ninjatrader.com/support/helpGuides/nt8/en-us/release_notes.htm

Tracking ID: NTEIGHT-10886

[tradertakeo]

Hell Jessica,

With 8.0.3.0 a report is generated again when choosing "Display Trades", which causes printing to Output multiple times.

1. Compile the strategy attached.
2. Run a backtest wtih the strategy.
3. Choose "Display Trades"
4. Go back to "Display Summary"
5. Choose "Display Trades" again.
6. Repeat # 4 and #5.

[tradertakeo]

Also, I found out that print statement executed multiple times when running a multiple instrument strategy backtest.

1. Compile the strategy attached.
2. Choose "Display Trades".
3. Run a backtest wtih the strategy.

[NinjaTrader_JessicaP]

Hello tradertakeo, and thank you for your report.


This behavior was confirmed on our end. We will be investigating this further. Please keep an eye on the NinjaTrader 8 Release Notes page for updates and bugfixes.

http://ninjatrader.com/support/helpGuides/nt8/en-us/release_notes.htm

Tracking ID: NTEIGHT-11021