Can you accidently trick a strategy analyzer backtest?

[sampras010]

I have a strategy that when run in the strategy analyzer creates too good to be true returns. It's a multi-instrument pairs trading strategy, and does not appear to be functioning as planned. The plan was to simply go long one instrument and short the other once the difference between ones bid and the other's ask is wide enough, and exit on the opposite condition. I know it's not working as planned because the order pairs aren't going in simultaneously. I can post the strategy and/or backtest if someone can figure out how it's winning so easily, and if these results are actually possible live, as is or not.

[NinjaTrader_AlanP]

Hello sampras010,

Thank you for your note.

What time series are you running the strategy on?

In the analyzer results, if you look at trades and examine the times, are the trades being taken intraday or perhaps on the opening only?

Are you including a secondary series of type tick or second to get granular fills as described at the following link?

https://ninjatrader.com/support/forum/showthread.php?t=6652

I look forward to your reply.

[sampras010]

The trades are taken intraday, and I now see my mistake in choosing 1 tick range bars for the main time series and the secondary series. What I'm curious about is how nearly every pairs trade is successful using 1 range bars. I wonder if I'm fooling the backtest, or if waiting for each range bar to close is a viable strategy. I've also now tried 1 tick bars and the orders generally go in at the same time, but something weird is still happening that seems to be making the strategy work better than expected. Until my flatten everything time each day, the first series entry/exit seems to be at the beginning of a second, and the second series entry/exit is no closer than the end of that second. I just can't tell without the secondary series chart in the strategy analyzer if somehow waiting a little to enter with the second series after the bid/ask condition is met is making these particular backtests work in a replicable way.

[NinjaTrader_AlanP]

Hello sampras010,

Are you by chance using limit orders to exit your positions in your strategy?

If you enable "fill limit orders on touch" does this reduce profits to the point of expectation?

I look forward to your reply.

[sampras010]

I'm not using limit orders, but I guess a better question would be, is there any scenario where better than actual results would be achieved by the backtest traversing backwards filling the entries and exits in opposite order, but then listing the results in logical order? I should say this because my strategy uses Crude oil and the crude oil mini, I know the spread between the two almost never open up as wide as the bid ask condition I set. That's why I thought the backtest would fail. But somehow the strategy keeps buying low and selling high with non-simultaneous entry. 100s of pairs trades with almost no losses. Could it be that I put the bars for the main series in the script. Or in other words used series 1 and 2, not 0 and 1? Also, no trades are taken when live.

[NinjaTrader_AlanP]

Hello sampras010,

With all the variables of how the strategy could be set up, I would need to test on my end.

If you'd like to load the strategy and provide a screen shot of the strategy analyzer parameters as you have them set up, I will test.

To export a NinjaScript from NinjaTrader 8 do the following:
From the Control Center window select Tools -> Export -> NinjaScript...
Click Add>Select the indicator>OK>Export.
Then attach that file you saved; under My Docs>NT8>Bin>Custom>Select the downloaded .zip file.

I look forward to your reply.

[NinjaTrader_AlanP]

Hello sampras010,

Your strategy is accessing GetCurrentBid(), which in state.historical, will be set to the close price. To test a strategy which uses the bid/ask, you would want to see note 1 at the following link,

https://ninjatrader.com/support/helpGuides/nt8/en-us/?getcurrentbid.htm

When using backtesting the strategy will run exactly the same as if CalculateOnBarClose is set to true. This is true whenever the strategy runs on historical type data. The reason for this is because the only info known historically is the Bar open, High, Low and close and not all of the ticks that make up the bar. You can also get extremely misleading results if you are using a bar that changes shape such as renko bar type as the finished bar does not show the true range of price movement.

Please see Discrepancies: Real-Time vs Backtest section of our helpguide.

http://ninjatrader.com/support/helpGuides/nt7/?discrepancies_real_time_vs_bac.htm

With your strategy it might be best to test your strategy using market replay.

Please let us know if you need further assistance.

[sampras010]

Ok thanks, so does using getcurrentbid/ask while in state.historical mean that the backtest could end up traversing backwards picking perfect closing prices that meet the condition before they happened?

[gauri]

Hello,
I have a strategy that when run in the strategy analyzer creates too good to be true returns. It's a multi-instrument pairs trading strategy, and does not appear to be functioning as planned.
Thank you

[NinjaTrader_AlanP]

Hello sampras010,

Without testing the strategy I wouldn't know for sure, but I would speculate those entry times are after a bar closes in each series, and that juicy profit you are seeing is the arb between a lag of the two bar closes. And not the strategy analyzer picking perfect entry times.

Please let us know if you need further assistance.

[NinjaTrader_AlanP]

Hello gauri,

Thank you for your note.

Are you using limit orders for exits?

Have you gone through the trade tab and investigated the fills and fill times?

What is it which leads you to believe the results are inflated?

I look forward to your reply.