Backtest results profitable over a period that in real life trading was all losses

[RunnrX]

Hi,

I've carefully developed something that backtested as a profitable trading system and with great happiness I've run it forward for almost 30 days now. The live outcome is much different than even a backtest of that same period with a few notable discrepancies:

1) The backtester over the past 30 day period shows a good profit whereas the actual live trades over the past 30 days are a steady loss using the same strategy

2) On my charts, winning trades are being shown with dotted red lines in real time but if I refresh the chart and the strategy goes into historic redraw, the lines turn green

3) I still have issues in the backtester where entry points are being labeled as "exit" in some cases and it almost seems to be reversing the trade's P&L, making losers get counted as winners

Its the first two points that I'm most concerned about, and I'd be happy if anyone who knows about these kinds of things would like to talk about it in more depth. I'm willing to share the strategy code and an .xls of the historical live trades and backtest trades to compare if anyone wants to work through this with me. The potential benefit of doing so is that we might be able to work out what the cause of the discrepancy is and get this system to work properly, and you'd have access to that code.

Thanks!

[sledge]

With regards to #1, that is to be expected.
.
Have you run your strategy in market replay? That'll be a more true test, and even that can be a little optimistic with fills in your favor.

[NinjaTrader_Matthew]

There are a number of reasons that a strategy may not take orders in real-time the same as you see when you reload the strategy historically.

Please review our Help Guide article on Discrepancies: Real-Time vs Backtest

http://www.ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

Once you have read this material, please tell me a few things about your strategy to help determine why that could be occurring:


* What type of orders do your strategies use? (Market, limit, stop, etc)
* What time interval are you trading on? (5 minute, 150 tick, etc)
* What state is the strategy in from the Strategy tab of the Control Center at this time? Is it "Green" or "Yellow"?
* Does the chart data or indicator values if you right click on the chart and select 'reload all historical data"?



I look forward to assisting you further.

[RunnrX]

Thanks for helping out guys, here's what I have so far:

I've carefully looked at how charting and orders work in backtesting and I designed the strategy to work around those issues, so I should have resolved most of those discrepancies as far as I can tell. Here's the actual trading logic:

The chart has two bars objects, the first one is a 5 minute candlestick of USD/JPY and the second is a 3 Line Break chart with 5 minute period of the same currency pair

If there is no open position, it looks at the line break chart at the close of each period to see if there has been a new green bar appearing after a red one (in that case use market to go long) or if a red bar has closed after a green one (use market to go short). In this case as far as I can see it the backtesting should agree, since I'm just using the close of a prior bar and entering market on the open of the next. Slippage isn't a concern because 2 pips or whatever isn't a significant impact on the outcome of the strategy.

Once the long or short position is in effect, the strategy looks at the close of the candlestick chart bars to see if they have closed below (if long) or above (if short) a trailing stop type indicator line that follows price trend but gives it some moving space. If they close beyond the trailing line, then a market exit order is issued at the close of that bar.

When I look at it in backtesting I look at both the candle and line break chart side by side and I think "ok most trades are executing properly" except for some irregular occurrences where it will call a short entry "short exit" and the losing short exit "short entry" and consider a losing trade a winning trade which I've had happen at other times when using line break charts.. but besides that because it shouldn't be occurring enough to offset my real results by much, I'm finding that running the backtest produces a steadily climbing cumulative net profit graph whereas the live trading one is simply a string of losses and nothing like the backtest at all.

The odd thing is when I look at the live trading on the chart, it appears to be executing properly as well so the discrepancy is in there somewhere and my eyes are getting too tired to look through every trade and try to find what seems to be a needle in a haystack cause

The strategy colour state is yellow until it syncs up with a new trade and then it's green so that's just a short period of time and not enough to make this big of an impact on the overall results

When I reload chart data the bars and indicator values appear to remain exactly the same as before

Thanks for going over this with me!

[NinjaTrader_JC]

Hello RunnrX,

Running a backtest on a LineBreak may produce different results that one that is being built with live data on charts due to the LineBreak chart being a time-independent, which has its strengths and weaknesses just as any other chart type.

Considering that in running a backtests will only have access to the Open, High, Low, Close of a bar there can be a lot of price action in that bar that may effect your strategy when running with Real-time data that backtests will not account for.

That being said, to have your results be more like the real-time results there are two options. The first would be to use the Market Replay Connection that will act like you are getting real-time data from the data that you have either recorded/downloaded.

The other would be use add an intrabar granularity to get more accurate results like when you are running in real-time by adding a 1 minute or 1 tick time frame like you are doing with the 5 Minute LineBreak chart. We actually have a reference Strategy that you may view for an example of this.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

Let me know if you have any questions

[RunnrX]

Just to post a resolution to this thread, while you have also said good facts about the discrepancies, I have done further testing to show that even in backtests the strategy itself isn't longterm profitable, causing me to want to discard it for other reasons and go back to the planning stage - thank you for helping with this question though!

[stephen314]

RunnrX,

Your post is very interesting. I desperately need Ninja Trader to be able to back test based on preselected intraday tick data. Are you telling me that there is no reliable way of verifying the effectiveness of a day trading program so that when you go live you don't lose your shirt?

[koganam]

RunnrX,

Your post is very interesting. I desperately need Ninja Trader to be able to back test based on preselected intraday tick data. Are you telling me that there is no reliable way of verifying the effectiveness of a day trading program so that when you go live you don't lose your shirt?

You can do reliable Backtests with NinjaTrader. It is a matter of setting up your Strategy correctly. Remember that profitable Backtest results do not mean that one will necessarily make profits with the system when one goes live.

[stephen314]

koganam,

Thank you for your post and your offering of assistance. Are you saying that back test reliability and real time reliability are not guaranteed to be the same due only to the fact that the data being analyzed in real time is different than the data that was analyzed during the back test?

Steve

[koganam]

koganam,

Thank you for your post and your offering of assistance. Are you saying that back test reliability and real time reliability are not guaranteed to be the same due only to the fact that the data being analyzed in real time is different than the data that was analyzed during the back test?

Steve

No. I am saying that Backtesting is inherently different from live trading, though the different data sets are obviously part of the reason.

This thread has a somewhat fuller discussion of the issue,. http://www.ninjatrader.com/support/f...light=backtest

If you search on "Backtest" and my username, you will see a whole bunch of other threads which discuss limitations of backtesting, on which I commented.

[stephen314]

Hi,

Thanks for your quick response.
Without wading through all of the past comments, there is only one thing, then, that I need to understand about the inconsistencies of back testing vs. real time.
If we could imagine back testing the month of July of this year we would have in our pockets the profit information for 23 trading days according to Ninja's Playback feature. Since our instrument is @ES# and nothing else has changed in our strategy or any other variable connected to profit, we now also have the profits that we actually made for those same 23 days during real time. Each of these 23 profit numbers could then be constructed into a histogram showing the mean and standard deviation of the real time vs the playback. My question to you is which multiple choice answer best describes the differences we would expect to have: a) Playback population has the same mean value as the realtime but is simply more variable. b) Realtime is more variable than Playback but the mean value is the same c) Realtime gives us statistically significant higher profits than Playback d) Playback gives us statistically significant higher values than Realtime.
If your answer is "d", then there must be a special cause associated with the difference that needs to be understood. Either the Playback does not include all of the dynamics of Realtime or there is something going on during Realtime that is driving down the profits.
If your Playback or Market Replay feature cannot duplicate realtime performance then it has little value.
If we then lack the ability to replicate Ninja performance by using the same data, then there is an unknown that exists that presumably would make the trader uncomfortable unless the differences could be explained or better quantified.
The data will tell the story.

Steve

[NinjaTrader_JC]

Hello Steve,

If you are referring to mean value as price data then it would be similar to "B" but it is a bit more complicated than that. Using a the backtest you will only be able to process your strategy on Historical Data which can be like a snap shot of what is happening which is not accounting for each incoming tick of data like real-time data would.

The results will typical be fairly accurate but not have all of the real-time price action of the real-time data. You can add intrabar granularity to be able to see all of the price action just like you would in real-time but this would require some additional custom coding to be able to accomplish this.

We have a reference that you may view that demonstrates this at the following link so you can add a 1 tick granularity to see all of the data like it would be in real-time.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

Market Replay will simulate this so you do not have to add additional code to your strategy.

Happy to be of further assistance.

[koganam]

Hi,

Thanks for your quick response.
Without wading through all of the past comments, there is only one thing, then, that I need to understand about the inconsistencies of back testing vs. real time.
If we could imagine back testing the month of July of this year we would have in our pockets the profit information for 23 trading days according to Ninja's Playback feature. Since our instrument is @ES# and nothing else has changed in our strategy or any other variable connected to profit, we now also have the profits that we actually made for those same 23 days during real time. Each of these 23 profit numbers could then be constructed into a histogram showing the mean and standard deviation of the real time vs the playback. My question to you is which multiple choice answer best describes the differences we would expect to have: a) Playback population has the same mean value as the realtime but is simply more variable. b) Realtime is more variable than Playback but the mean value is the same c) Realtime gives us statistically significant higher profits than Playback d) Playback gives us statistically significant higher values than Realtime.
If your answer is "d", then there must be a special cause associated with the difference that needs to be understood. Either the Playback does not include all of the dynamics of Realtime or there is something going on during Realtime that is driving down the profits.
If your Playback or Market Replay feature cannot duplicate realtime performance then it has little value.
If we then lack the ability to replicate Ninja performance by using the same data, then there is an unknown that exists that presumably would make the trader uncomfortable unless the differences could be explained or better quantified.
The data will tell the story.

Steve

That would depend on how your Strategy is coded. If your strategy makes orders only on bar close, and has simple determinate entries and exits based on price or well-behaved indicators, I should expect little variation, if any, between a Backtest and live results over the exact same period.

[Baruch]

Hi,

Thanks for your quick response.
Without wading through all of the past comments, there is only one thing, then, that I need to understand about the inconsistencies of back testing vs. real time.
If we could imagine back testing the month of July of this year we would have in our pockets the profit information for 23 trading days according to Ninja's Playback feature. Since our instrument is @ES# and nothing else has changed in our strategy or any other variable connected to profit, we now also have the profits that we actually made for those same 23 days during real time. Each of these 23 profit numbers could then be constructed into a histogram showing the mean and standard deviation of the real time vs the playback. My question to you is which multiple choice answer best describes the differences we would expect to have: a) Playback population has the same mean value as the realtime but is simply more variable. b) Realtime is more variable than Playback but the mean value is the same c) Realtime gives us statistically significant higher profits than Playback d) Playback gives us statistically significant higher values than Realtime.
If your answer is "d", then there must be a special cause associated with the difference that needs to be understood. Either the Playback does not include all of the dynamics of Realtime or there is something going on during Realtime that is driving down the profits.
If your Playback or Market Replay feature cannot duplicate realtime performance then it has little value.
If we then lack the ability to replicate Ninja performance by using the same data, then there is an unknown that exists that presumably would make the trader uncomfortable unless the differences could be explained or better quantified.
The data will tell the story.

Steve

GIGO. If you know what I mean.

[stephen314]

Gentlemen,

Let me go back to RunnrX and thank him for posting this thing in the first place since it appears to be a common theme in several of the forums I have visited.
Koganam, thank you for your comments and what would appear to be a slight shift in your thinking.
I would like to mention JC's comments regarding how the data would be interpreted in one time frame compared to another. For any given trading day, the price data is a perfect and unequivocal representation of all of the action out in the marketplace. This includes fear, greed, bad decisions, good decisions, special announcements and every aspect of human behavior and every aspect of equity flow. So once we collect the data during that live day it becomes an unchanging road map to the human equation for that day. Now, just because we do a playback on the data at a later time doesn't mean that we have to change the granularity to match changing environments. If I play back the data one month later and the big bomb drops, the data will still reflect the environment at the time it was created. the fact that you are adding and subtracting ticks to see what the data should really look like is ludicrous.
The data is what it is and the profits should be statistically indistinguishable.

Steve