Backtesting and computer problems

[coolmoss]

Hello,

I'm trying to assist a client who is having major issues obtaining consistent backtest results. The user does extensive work running backtests on tick data for a variety of stock and futures. Most frequently these backtests are in the range of 10 - 20 days of data. What happens is the user will start noticing double trades (two trades where it should only be one), then resets the db.

Obviously, this is rather worrisome from the point of view of the user. But what is really frustrating is that when the db is reset, every strategy is deleted from every chart. And the strategies have upwards of 50 - 70 parameters, so many hours are lost each day just resetting up charts.

Also, the following odd behavior is noticed when running performance reports. Let's say today is Wednesday. A performance report is run for the past 10 days. In particular, we note Tuesdays performance and trades. Come Thursday morning, we run another 10 day report. The performance and trades for Tuesday, between the two reports, do not match; and obviously they should.

You should know that on this computer there has been a corrupt database, foreign key errors, needs to reset the db on an almost daily basis, gets out of memory errors, freezes, etc. This is a pretty newish Win7 computer with 6 gig of ram. The running software typically includes IB TWS, Ninja 7, a browser and excel. I sometimes work on this computer via TeamViewer.

I'm a bit perplexed as how to start resolving these issues. Thanks in advance for your help.

[NinjaTrader_AdamP]

coolmoss,

There are known discrepancies between live trading and backtesting.

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?discrepancies_real_time_vs_bac.htm

There are some reference samples for resolving some of these, such as the "backtesting with intra-bar granularity" reference sample.\

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

As far as the database reset, strategies should not be accessing the database through supported code, and also shouldn't have the ability to reset the database. Is this occurring or does the client reset the database?

As far as the performance report, how are they generating these reports? Are they using historical trades that were placed live, or is it based on historical backtested trades?

There is a slight difference here, as you can see the performance results of a strategies historical trades that were performed in in a "backtest" mode, as well as "real-time" performance (which may also include historical trades that were placed live).

Essentially, when a strategy is attached to a chart it backtests on historical data so that it can determine what state it should be in at the most current bar, as well as to determine its "historical" performance. It will then perform all trades in real time from that point forward. You can see the results for the strategy by selecting either "historical", "real time and historical" or "real-time" when you right click the strategy in the "Strategies" tab and go to > "Strategy Performance".

[coolmoss]

Hi Adam,

I think you might have misunderstood me. The differences aren't coming up comparing real time to historical. It's comparing historical to historical. I will question my client to make sure I'm getting the story straight on this.

The database is being reset by us, because we frequently and continuously keep getting obvious errors in the backtesting report. It's gotten so bad that we sometimes need to reset the db more than once in a day.

Please see my attachement below. If you scan down the trades, you'll see a series where the bars count is like 30,000 bars, and one is even negative 30,000. I've brought this up before, but never really got a clear resolution.

On the chart itself, these trades are not plotted and do not show.

[NinjaTrader_Bertrand]

coolmoss, I've been chasing this negative bars issue for a bit, is this easily reproducible for you or would it appear spurious only? If you can please send me a note to support at ninjatrader dot com to my Attn, so we can look into a reproducible setup we can forward to our dev team for investigation / fixing.

Thanks for your support in this matter,

[coolmoss]

Bertrand,

It's steadily gotten worse. It's to the point that after resetting the db, the performance reports start going wacky within 10 to 15 minutes.

Just to reiterate: the user is now fairly frequently getting Ninjatrader produced messages stating that NT needs to shut down since the pc has run out of memory. Also, virtually everything he does in NT gets the little whirling Win7 icon (showing the computer is working) going for extended periods of time.

[NinjaTrader_Bertrand]

Thanks coolmoss, I understand you discussed with with Matthew on the phone, please let me know if you would still see this issue occurring after the SQL changes implemented.