Replay Problems

[moon_121]

I am having problems with the Market Replay. I get hugely different results in backtesting, replay and running a strategy live with IB.

Using your SampleMACrossOver as an example, I get the following results:
Time: 2/7/12 - 4/7/12
Parameter 34/68
EURUSD
1 Minute
Backtest: 13380 € Profit
Replay: 23.019€ Realized Loss (420€ unrealized Loss)

I know that there are differences between backtesting and replay. But turning a 13.380 profit into a 23.019 loss seems more than a bit odd.

Can you reproduce these results on your side? It seems that my Market Replay is behaving much stranger now, than it used to do. I already tried to reset the DB and reload all the historical data. It does not make a difference.
Do you have any explanation?
By the way: If I run a strategy live with IB, it is much closer to the backtesting than to the replay results. What is wrong?

[NinjaTrader_Joydeep]

Hello moon_121,
To assist you further may I know are you running the strategy in Market Replay with Calculate on bar close set to true or false.

I look forward to assisting you further.

[moon_121]

"On bar close" is set to "true". It gets even worse if I try it with false.

[NinjaTrader_Joydeep]

Hello moon_121,
Unfortunately backtest, replay and live results can vary.

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?discrepancies_real_time_vs_bac.htm

Backtest is calculated at the end of the bar and thus if you are running the strategy with Calculation on bar close set to false then the results can be even farther.

Also please note, there can be data discrepancies between IB and Market Replay since IB uses snapshot data to construct the bars.

[moon_121]

I knew I would get this generic reply. It is the same answer NT gives everybody who has problems with backtesting, replay or setting up a strategy with Renko.
Perhaps you could explain a slight difference between backtesting and replay with this answer. But the complete opposite? With SMA-cross? With 1 Minute charts? It can't get any dumber than that. We are not talking Ticks or Renkos here.
At least, the backtest should be approximately the same like the replay.
And by the way: I am using Kinetick as market data provider. And as I said: Everything is set to "on bar close: true".
Here is my challenge: Why don't you show me an example, where your backtesting resembles a market replay. Can it be done? Or is one of them complete rubbish?

[NinjaTrader_Joydeep]

Hello moon_121,
Backtesting by default does not use bid/ask data to execute the orders. This is not the case while you are live. This can make a considerable difference. You can further enhance your code and add the bid/ask series and submit the orders in the bid/ask series while backtesting your strategy.

We are aware of the demands of trades regarding this discrepancies and it is already in the our feedback/enhancement list and we hope to enhance the backtest feature further in our next major release.

[moon_121]

I am glad to hear that you are working on a better backtesting.
But:
I can't believe that the bid/ask difference in a market replay makes all the difference. My SMAcross-Example executed only 6 trades during 3 days. With bid/ask diffrences it should perhaps have a 1000€ difference to the backtest. But I am getting 26000 € difference. I think that is a bit to much to explain by a bid/ask spread.
Could you at least tell me, if you are getting the same replay and backtesting results on your machines? I suspect that something is wrong with my db.

[NinjaTrader_Joydeep]

Hello moon_121,
Can you send a screenshot of the exact backtest settings so that I can test it at my end.

I suppose you are testing from 2nd July - 4th July 2012 (and not from 7th February - 7th April 2012).

I look forward to assisting you further.

[moon_121]

I tried to replicate the results with an other opimization run in backtest and got some different results.
In the screenshot, you see the Strategy Analyzer Window on the left hand side. On the right side are the results I got earlier and saved in Excel. (Disregard the PointAndFigure results. This would only make everything more complicated.)

[NinjaTrader_Joydeep]

Hello moon_121,
Unfortunately the screenshot did not upload. Can you please reload it again.

Edit: Never mind got it. Thanks.

[NinjaTrader_Joydeep]

Hello moon_121,
It may be the case that new data are being loaded when you are backtesting/optimizing the strategy again. Forex session starts in the evening and ends in the following day. So todays data will be also included if you set the end date to 4th July '12.