Comparing backtest with live trading performance

[kbellare]

I'm have trouble comparing backtest with Sim101 live trading performance (for futures strategies) because ..
1) It's hard to compare aggregate performance summaries by specific dates because backtest date/time starts the previous date (e.g. backtest from Sept 1-Sept1 has trades from Aug 31st 5pm to Sept 1 midnight CST) but Sim101 typically from Sim101 (typically futures session template boundaries)

2) Comparing trade-by-trade is tedious (and pointless as we don't expect the individual trades between the 2 systems to perfectly match) because the trade start times for backtests are based on local PC time whereas Sim101/live trades are based on the Exchange time - lots of Excel manipulation required.

Has anyone had a different better/faster approach to comparing performance between the two? This process is important for troubleshooting because i'm seeing a huge difference in performance between backtests/walk-forward testing and Sim101/live performance (much worse than backtests).

I realize that the Sim101/live perf wouldn't be the same for various reasons, but the whole premise behind auto-trading is to develop strategies using walk-forward and backtesting and deploy these for live trading, so we should expect the two to have at least comparable performance, for auto-trading methodology to work.

Kiran

[NinjaTrader_ChelseaB]

Hello Kiran,

I am not aware of a faster way to combine these other than by hand.

(You might take the performance information and trade information and write these to file when OnTermination is called. Not sure if that would be a solution but its something to think about.)

However, this thread will remain open for any community members that would like to assist.

[misova]

I think you do not need to make all that pairing trades to make significant progress
and get the idea what causes the difference.

LIVE account represents the reality.
SIM account represents the simulation.

Point is to understand the idea, where SIM processing behaves differently from LIVE processing.
To get that idea, you probably will need to analyze only first trade that behaves differently.

If you have good logging, you should find the difference in processing relatively easily.
Follow the execution/processing path, and look for the point, where is the difference.

[sledge]

Has anyone had a different better/faster approach to comparing performance between the two? This process is important for troubleshooting because i'm seeing a huge difference in performance between backtests/walk-forward testing and Sim101/live performance (much worse than backtests).

I realize that the Sim101/live perf wouldn't be the same for various reasons, but the whole premise behind auto-trading is to develop strategies using walk-forward and backtesting and deploy these for live trading, so we should expect the two to have at least comparable performance, for auto-trading methodology to work.

Kiran

Are you using 1-tick dataseries cheat in your backtests?

And have you used Market Replay instead of Sim/Live to test many days and figure out what's going wrong?