Live sim vs. backtest discrepancy (COBC)

[LightWeight]

I am running into an issue with live testing vs. backtesting regarding strategy entry and exit. In backtesting of course CalculateOnBarClose = true, and all entries and exits have a time stamp at the bar open or close, trades are not made intra-bar.

My expectation then is that while running a live sim on the same strategy with COBC = true, entries would be at the same time as the backtest time stamp, as well as exits based on the strategy signal changing (not stop loss or target being hit). However, in live sim trading all of my strategies ALWAYS enter trade BEFORE the backtest time stamps -- it's like COBC is set to false or completely ignored, and the strategies jump in immediately on signal instead of waiting until the close as the backtest does.

The real problem appears to be on exit though. While my entries vary slightly in price, they are usually within a tick (live sim vs backtest), but the exits can vary wildly. Again, it seems as if the live strategy is using COBC = false, so as soon as the signal reverses or indicates exit, the strategy hops out/exits intra-bar. The backtest meanwhile waits until bar close, which often means that the exit signal has disappeared or changed, and thus it does not exit the strategy. I've attached screen shots of performance in back test vs. live sim for the same strategy, and you can clearly see the difference between the two, as well as that the live sim entries are not at all on bar close and always preceed the backtest time stamps. The result in performance was that the loss in live sim trading was over TWICE as bad as backtesting would indicate it should be.

Let me also point out the following to be clear:

Strategy code is COBC = true
Dialogue/GUI box setting is COBC = true
All strategies use market orders for entry
All strategies have limit and stop orders for target and profit, but also use market orders for exit if indicator signals change (which should be calculated upon, and thus exit upon, bar close)
In spite of all this, live sim strategies consistently enter and exit intra-bar, thus giving different results than the live backtest, almost universally for the worse.

So my question is, why are the strategies not entering and exiting at the same time as the backtest, and how can I get the live strategies to operate in the same fashion as the backtest in terms of timing? I am aware that Ninja backtesting can only calculate and execute on bar close, and that is fine -- I can make a profitable strategy in backtesting using that limitation. The problem is getting live performance to operate in the same manner. Everything I have read indicates that simply setting COBC to true will function the same, and yet it is not.

Regards,

Andrew

[NinjaTrader_PatrickH]

Hello Andrew,

Thank you for your post.

The item here is that while in backtesting even the order fills occur on the close of the bar, meaning that NinjaTrader calculates where a fill would have occurred based on Open, High, Low and Close of that bar and factoring in the settings for FIll Type and Slippage.

When running a strategy on the real market, even though we set CalculateOnBarClose to True our fills occur intra-bar and at whatever price the market is able to fill us at for the respective orders and order types.

For more information on discrepancies between real-time and backtesting please visit the following link: http://www.ninjatrader.com/support/h...ime_vs_bac.htm

[LightWeight]

Hello Andrew,

Thank you for your post.

The item here is that while in backtesting even the order fills occur on the close of the bar, meaning that NinjaTrader calculates where a fill would have occurred based on Open, High, Low and Close of that bar and factoring in the settings for FIll Type and Slippage.

When running a strategy on the real market, even though we set CalculateOnBarClose to True our fills occur intra-bar and at whatever price the market is able to fill us at for the respective orders and order types.

For more information on discrepancies between real-time and backtesting please visit the following link: http://www.ninjatrader.com/support/h...ime_vs_bac.htm

Hi Patrick,

Okay, I think I may have figured some of this out then, but I'm not sure. First of all, if the time stamp for a backtest strategy entry is 7:45:00, would I be correct in assuming that the signal was actually calculated at 7:44:00, but that backtest is using the close price (and time) of the signal bar for entry? My assumption was that if it calculated the signal for entry as occurring on the bar close at 7:43:59/7:44:00, that it would then enter at the same spot...but it seems now that it actually enters on the close (or close price) of the bar immediately after the signal is calculated, which looks like a minute delay.

Second, I went back and looked at my time stamps for entry and then a 1 second chart, and I realized the entry problem -- because these are sim orders, then even though the are programmed for market trades, they can't actually fill until a real world trade takes place. In a market like Silver and in the evening, that may take 40 seconds or more into the one minute candle before a trade happens and a time stamp and price can be recorded in the simulator. So if we use what I'm guessing above -- that the entry signal was calculated at the close of the previous bar -- then COBC is in fact working properly and the signal was on the PREVIOUS bar, the time delay is simply due to the nature of sim requiring real world trades and the delay that causes, whereas in live trading obviously a market order would be filled right away, and I would see time stamp entries right around the bar open.

I think that solves my confusion about the entry time discrepancy, but the exit conditions are a little hazy. Again, as for the time delay I have to assume that's the same reason -- sim waiting for real world trades before it can record a transaction. The confusing bit is that sometimes the backtest exits much later, which makes me thing that because backtest is exiting on the close, it is first calculating once again if the signal still exists BEFORE it will take itself out, and naturally sometimes the signal will have disappeared, so the strategy will actually not exit, while in live trading, because it did not wait until the bar close to exit, it would have already triggered.

If so, then if I wanted my exits to mirror backtest results more closely it seems like I would have to code the strategy to wait until the first tick of the next bar before exiting, and hope/assume that it is calculating whether the signal still exists before doing so. This is all conjecture but it's the closest I've gotten so far....at worst it seems like if I want live trading to more closely match backtesting then I need to code the strategy to wait until the first tick of the next bar to trade.

[NinjaTrader_PatrickH]

Hello Andrew,

Thank you for your follow up on this matter.

Your conclusions here are correct. It is possible to just use FirstTickOfBar before placing orders, but this is already accomplished when using CalculatOnBarClose = True. It would require waiting for the signal and then waiting for the the FirstTickOfBar 1 bar later.

However, instead of trying to force the real-time fills to assimilate to the backtest results you can simulate intra-bar fills in backtesting with the example at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

[LTWCI]

I had a situation this morning where a strategy bought an ES contract in sim but the same strategy didn't buy the ES contract live. This has happened for three trades now. The live strategy eventually did buy independent of the same strategy in sim--the sim strategy didn't buy that time. This is the first time I've ever seen this happen. I double checked both strategies in the edit box and could find no discrepancies. Calculate on bar closed is true for both live and sim. Any ideas as to how or why this happened? I lost money today because of this.

[LightWeight]

Hello Andrew,

Thank you for your follow up on this matter.

Your conclusions here are correct. It is possible to just use FirstTickOfBar before placing orders, but this is already accomplished when using CalculatOnBarClose = True. It would require waiting for the signal and then waiting for the the FirstTickOfBar 1 bar later.

However, instead of trying to force the real-time fills to assimilate to the backtest results you can simulate intra-bar fills in backtesting with the example at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

Hi Patrick,

That doesn't seem correct. All of my strategies are using COBC = true on a 1 minute chart, and they always go in earlier than the backtest time stamp. For example, say the signal pops up at 7:44:59, or at the close of the 7:44 bar, then the live strategy will execute a market order at 7:45. However because this is sim it won't fill until a trade takes place in the real world, which means the entry times are random -- but they always occur sometime during the next minute. Backtest however will record the same entry as occurring at 7:46:00, ALWAYS at the end of the bar that came after the signal. If what you are saying is correct, then I would expect that the live sim would always enter AFTER the backtest time stamp, not before it. But that doesn't seem to be how it works -- backtest takes the close of bar entry, while in live trading it will have entered somewhere prior (and in live money trading, because it is using a market order, it would have entered immediately on the open, thus the live entry and backtest entry would be a full minute apart).

That is why I have been trying to figure out how to force the live strategy to delay execution until the end of the bar, to try and match the backtest timestamp, because the backtest delay (trading on close) nearly always gives a more favorable entry and exit. This makes sense because the strategies are counter trend, so usually price will still be moving the original direction a minute later. This difference of even a couple of ticks per side adds up to be quite a bit over time and many trades, giving you a false sense that the strategy is profitable, meanwhile when it is trading live it is getting in an out up to a full minute earlier than the backtest is calculating it would.

I am trying to experiment right now with different versions of the strategy to try and mirror the next-bar/close timestamp of backtest, if you're aware of how I can do that it would be helpful. Again, simply putting COBC = true does not do it -- it will immediately execute DURING the bar that came after the signal, while backtest always waits for the price of the CLOSE of that same bar. I hope that makes sense.

[NinjaTrader_PatrickH]

Hello Andrew,

Thank you for your response.

Not quite what I was trying to convey there. COBC = True will always calculate on bar close, we agree on that. In real-time trading the order is then filled at the nest available price, but in backtesting this would not be until the next bar closes which we also agree on.

What I am detailing is there really is no way to force real-time trading to fill on a specific bar close. We could however count the bars since the calculation determined that an order needs to be placed and then then place the order once the next bar closes. For example:

if(myCondition)
{
barCount = CurrentBar;
placeOrder = true; // bool to only check when myCondition returns true.
}
if(barCount + 1 == CurrentBar && placeOrder)
{
// place your order.
placeOrder = false;
}