Backtesting Errors

[sysimp]

Guys,

I think that we have some bugs in the results summary when running a strategy backtest.

I am attempting to move over from NT6 to NT6.5, and I believe that I have catered for any errors in my own code or systems by:
- checking the RELEASE NOTES and making any code changes
- testing the same strategy in NT6, using exactly the same operating system, and with no extra loaded software which might be interfering in some way.

I have attached images showing firstly the successful results obtained in NT6, and then the errors that are being shown in NT6.5.

It seems to me, that as the strategy is starting with a profit of -750%, I really can't see how this can be due to my strategy setup.

It seems to be a performance reporting and charting problem, as a quick review of my chart shows that in general, the trades are pretty similar. The only trades that are being affected badly, are those that require a check of performance data (eg. p&l) before the trade is allowed to proceed.

Please review, as it is creating quite a lot of "uncertainty" (not to mention wasted time) for me...

[NinjaTrader_JoshP]

Interesting results you have there. As a procedure to debug what is happening please try this:

1. Ensure you have the exact same data set in NT6 as you do in NT6.5.1000.1 (make sure you are on the production release instead of the beta for 6.5)
2. Backtest the strategy SampleMACrossOver in NT6
3. Backtest the strategy SampleMACrossOver in NT6.5
4. You should have the exact same results.
5. If 4 is true then I suggest going into your NT6.5 strategy and adding some print functions at the beginning of your strategy to see how it managed to get -750% profit at start.

[sysimp]

Interesting results you have there. As a procedure to debug what is happening please try this:

1. Ensure you have the exact same data set in NT6 as you do in NT6.5.1000.1 (make sure you are on the production release instead of the beta for 6.5)

2. Backtest the strategy SampleMACrossOver in NT6
3. Backtest the strategy SampleMACrossOver in NT6.5

4. You should have the exact same results.

5. If 4 is true then I suggest going into your NT6.5 strategy and adding some print functions at the beginning of your strategy to see how it managed to get -750% profit at start.

Thank you for your reply Josh. Here are my answers:

1. Yes, exactly the same data set is being used, and I have re-installed both NT6.5 AND NT6. I was very cognisant to ensure that I had the latest builds, and to ensure that I had removed any old "hangovers".

2 and 3. Done - results attached

4. Yes, exactly the same thing is happening.

5. As exactly the same thing is happening to MACrossover, I think that this needs to be handled by a Ninja debugging team.

Perhaps one way that it could start at -750% is if a start up variable is not being handled correctly; I haven't changed any input data in the change over from NT6 to NT6.5 (ie. the starting capital is still $100k, etc.). As I have already spent "days" working through this problem to ensure that it isn't on my side, I don't think that I should be spending any more time until this is fixed.

Thank you so far.

[sysimp]

OK Guys,

I've found the source of the error for you. The issue is that "someone" has buggered up the commissions calculations which thus affects all the performance results... I wonder how far throughout the code this bug is replicated?

So, the quickest way for a user is to set COMMISSIONS = FALSE when backtesting in NT6.5, until this error is fixed. (Not great for accurate financial analysis...)

cheers,
sysimp

[NinjaTrader_Ray]

Thanks, we will look into it.

[NinjaTrader_JoshP]

Interesting sysimp. Could you please inform me of the commission structure you have setup for your instrument in your tests so I can try to reproduce on my end? Thanks.

Edit: Also it would be helpful if you could provide me exact testing parameters you used too. e.g. instrument, time period, order handling, fill type, etc.

[andrgm]

The same backtest problem

My systems also give different results on NT 6.5 and NT 6

However MACrossOver strategy works well. I think the problem lies with the stop orders.

Another thing: The graph of the draw down on NT 6.5 shows erroneous results.

[NinjaTrader_JoshP]

Hi andrgm,

Thanks for the report. Could you also post a simple as possible strategy that demonstrates the issue?

In regards to the drawdown issue. Could you please describe what exactly is erroneous and how you figure? Thanks for your help in these issues.

[andrgm]

Strategy Analyzer problem

When I use orders:

EnterLong (DefaultQuantity, "");

Or

EnterShort (DefaultQuantity, "");

NT6 and NT65 backtest are equal.

But when on the same system using stop orders:

EnterLongStop (Bars.CurrentAsk + TickSize, "GHS Cross Entry");

NT6 and NT65 backtest results are different.

Here's two pictures of Drawdown of the same system in NT 6 and NT 6.5.

Sincerely,
Andres

[NinjaTrader_JoshP]

The strategy has errors as seen in the error logs in the Control Center for both NT6 and NT6.5. Potential differences in error handling is what could lead to the different results you are seeing.

Also, to get bid and ask prices we do not support Bars.CurrentAsk. We recommend you use GetCurrentAsk() instead.

[andrgm]

I have used "Bars.CurrentAsk" as shown in the example of the "Help Guide" for EnterLongStop() and EnterShortStop ().

But using other forms of entry as:

EnterLongStop (Close [0] + TickSize, "GHS Cross Entry");

OR

EnterLongStop (GetCurrentAsk () + TickSize, "GHS Cross Entry");

(...)

The problem remains unresolved.

Is there a different backtest fill method for Stop and Limit orders on NT 6.0 and NT 6.5?

I found this problem in several markets and systems, using always the same paramter vaules, commissions, slippage and time frames.

Here's an example.

[NinjaTrader_Ray]

Will update the DOC. Thanks for pointing it out. I do believe there is a different logic for sequence in how orders are filled between 6 and 6.5. If memory serves me correctly, market orders are always filled before stop/limit orders now. Before it was based on the the order in which the orders were submitted. The filling logic is the same though.