Strategy Analyzer in 8.0.6.1 - Inconsistent Data Bug

[maker]

While backtesting a strategy with the Strategy Analyzer in 8.0.6.1 I am seeing inconsistent data if I change the date range from:

• 2016-12-31 to 2017-05-10

to:

• 2016-12-30 to 2017-05-10

When I do this, the data picks up the last trading day of 2016 and I see slightly different bars in the chart. I then tried to load this date range on a plain Chart and it presented even different bars, but was consistent for either of these date ranges.

Something is causing the data to be inconsistent in the Strategy Analyzer, which is making it next to impossible to backtest my strategy. I see these bar differences in one instance/session of NT8 and can reproduce the differences after closing and opening again.

I am using a demo account, but the data should be cached locally and be the same on all three views.
The market is MGC.

Attached are three screenshots all loading different bars.

[NinjaTrader_AlanP]

Hello maker,

Because you are using tick based charts, which the difference of 1 tick can build completely different candles, to try and reproduce a tick based chart to look exactly like like another with a different start day would be difficult, thus it would be expected you'd get different results. Because there are many ticks per second, you’d have to tweak the start time down to the millisecond, which is not possible in NinjaTrader.

I’ve provided a couple links to the helpguide which help explain this point.
Discrepancies: RealTime vs Backtest:

http://ninjatrader.com/support/helpGuides/nt7/?discrepancies_real_time_vs_bac.htm

How bars are built:

http://ninjatrader.com/support/helpGuides/nt7/?how_bars_are_built.htm

Please let us know if you need further assistance.

Regarding running a backtest with locally stored data please see below.

To have backtest run on locally stored data, go to Control Center>Tools>Options and selecting the Data Tab. You’ll then uncheck the box next to “Get Data from Server (if Available)”. This will force NinjaTrader ot use its locally stored cached data and to download data only when that data is not already available.

Please let us know fi you need further assistance.

[CMillz]

Alan,

Question on your response, regarding:

" to try and reproduce a tick based chart to look exactly like like another with a different start day would be difficult, thus it would be expected you'd get different results. "

How does this apply to time-based charts? My understanding is the backtested (not real-time obviously) results should show the results no matter the start date. Correct?

[maker]

Thanks for the response Alan. I thought this might be the reason, but it was unexpected to see it. I guess it's much more pronounced in the small size and low volume of the E-micros. I have not seen such a difference in results when testing E-minis. I guess it's safe to say any strategy that relies on bar formation should not trade tick charts on E-micros or low volume markets?

If start time effects the bar formation, then why do two charts, one with "Days to load" set to 132 and one with "Days to load" set to 75 have identical bars when looking at the same day on each Chart? Do they load backwards from the "End date"? I did a test with a different "End Date" and all three Charts draw the exact same matching bars for a given day.

What makes the Strategy Analyzer inconsistent, but the Chart window always the same?

[NinjaTrader_AlanP]

Hello Cmillz,

This would not impact time based charts the way it would tick charts.

Hello maker,

Pricing a strategy off a less traded/wider market such as the emirco rather than the bigs could present issues.

Regarding the days to load issue, earlier you had said you used two different start dates and were getting differently built bars,

2016-12-31 to 2017-05-10
to:
2016-12-30 to 2017-05-10

When I do this, the data picks up the last trading day of 2016 and I see slightly different bars in the chart. I then tried to load this date range on a plain Chart and it presented even different bars, but was consistent for either of these date ranges.

Are you now using time based charts?

I look forward to your reply.

[maker]

Alan,

Those two date ranges you quoted were in the Strategy Analyzer window. I was comparing 2 Strategy Analyzer charts and 1 standard Chart. All three were different. However, if I compare 2 or more standard Charts, they are always the same. Even with completely different "Days to load" and/or "End date". The Strategy Analyzer is what is inconsistent, not the standard Chart.

What I was referring to in my last post was that on the standard Chart window (New > Chart), setting the "Days to load" and/or "End date" to different setting always results in the same bars. The Chart window is fine and always shows the same bars for tick data. The Strategy Analyzer window's chart and underlying tick data bars change based on the date range. This ends up effecting the strategy performance numbers.

Why is there a difference between the bars in the Strategy Analyzer window and its chart depending on the date range, but no difference in the standard Chart window when you use different dates?

You said above:

Because you are using tick based charts, which the difference of 1 tick can build completely different candles, to try and reproduce a tick based chart to look exactly like like another with a different start day would be difficult, thus it would be expected you'd get different results. Because there are many ticks per second, you’d have to tweak the start time down to the millisecond, which is not possible in NinjaTrader.

If that is the case, then why does the logic in the standard Chart window work fine and always show consistent bars? The logic in the Strategy Analyzer must be different for some reason and it show inconsistent bars when you use different dates.

[NinjaTrader_AlanP]

Hello maker,

If you use Custom Range for the Chart rather than Days to Load, and enter the Start/End date which matches the strategy analyzer, do you get two charts that look the same?

I have provided a screen shot of setting this up.

I look forward to your reply.

[maker]

Alan,

I tried using a "Custom Range" on the standard Chart and it does match the bars in the Strategy Analyzer with the same range. Also, using different "Custom Ranges" on two standard Charts gives me different bars. So that is how to get the standard Chart to match the inconsistent bars I was seeing in the Strategy Analyzer.

My obvious followup question is... is there a way to get the Strategy Analyzer to use "Days to Load" like a standard Chart? That would allow me to backtest with consistent bars across different date ranges. What makes "Days to Load" draw consistent bars, but "Custom Range" different?

Thanks

[NinjaTrader_AlanP]

Hello maker,

You could hover your mouse over the first bar on the chart in both the SA and the Chart and middle mouse button click and confirm they are completed at the same time. Are they? Bars are built from the left to the right, so if they are starting at the exact same time they should be the same.

Can you confirm you do not have Break at EOD checked in the SA or chart?

I look forward to your reply.

[maker]

Alan, yes they both show the same time for the very first bar in the chart. I have been unchecking "Break at EOD" on both SA and the Chart.

[NinjaTrader_AlanP]

Hello maker,

You should use custom date range if you'd like the charts to match. When using charts built from tick data, unless the two start on the exact same tick/time, you could expect differences. Trying to match the start tick/time of a chart using Days to load with a SA using a custom range, would be very difficult.

Please let us know if you need further assistance.

[maker]

Is it possible for the Strategy Analyzer to support "Days to Load"?

[NinjaTrader_AlanP]

Hello maker,

Not at this time, but I will submit a feature request on your behalf for strategy analyzer to support days to load.

This feature is being tracked under SFT-2281.

Please let us know if you need further assistance.