Error messages on historical data

[heech]

I'm actually writing about the *lack* of error messages.

I've noticed that when historical data isn't available, I don't seem to get any visible errors/warnings on a backtest. It just runs, and comes back with a nonsensical response (usually showing no trades).

That's the benign case. I had a less benign case, where I originally ran the backtest with the "wrong" start-date (and historical data was fetched). I later adjusted the backtest by one day to the correct start-date I had in mind... and the analyzer never re-fetched historical data starting from the new date. It just skipped the first date, and only used the historical data on hand.

I've now figured out to regularly close the Analyzer, and make sure I'm truly connected from the control center.

But is there something I can keep an eye on where warnings/alerts about unavailable historical data is being generated? Any rules of thumb on how/when I need to actively reload historical data?

[NinjaTrader_Bertrand]

Hi heech,

All error messages are usually logged in the NinjaTrader Control Center (Log tab).

Thanks for the suggestions to enhance them, I will forward these to our development team.

Have a great day!

[heech]

Ah, great. I don't see anything in the log related to missing historical data... but I don't really remember at what point today the events happened. I might have even closed the control center at some point. I'll keep an eye on it next time it happens.

[heech]

I'm still not seeing relevant error messages, but historical data seems to continue to be broken. It drives me nuts... my algorithm is very deterministic, and my experience with back-testing on past platforms (TradeStation + Wealth Lab) has always been deterministic. I expect to see the same results when I re-run my tests with the same parameters.

With Ninjatrader, I'm not getting that, and it's driving me nuts. The only explanation I can come up with is historical data is either wrong, or misaligned, or... something. I believe I read elsewhere you guys are relying on .NET to cache historical data in some way... if that's the case, it's a huge mistake. From what I've seen so far, this is by far the biggest Achillies' heel in the whole platform. You guys need to do something to rectify this... implement your own caching/storage of historical data preferably. Or, if nothing else, tell me how to get consistent and deterministic behavior. Should I, for example, close Strategy Analyzer and delete historical data every run?

How useful is back-testing if I only have 75% confidence that the code is seeing the correct data?

[NinjaTrader_JoshP]

heech,

No, NT has always loaded data into a database. If you are receiving different results it is simply because you are changing the database. This can happen if you keep reloading your data from your data provider as there are instances in which data can be different even though it comes from the same provider.

If you are "missing" data as in bars are missing from your data set all you need to do is reload your data set to get a fresh one from your data provider. Tools->Historical Data->Reload.

[NinjaTrader_JoshP]

heech,

We do not know what your cumProfit/config.StrikePrice is. You will need to debug it on your end. We assure you there is nothing wrong with the Strategy Analyzer and the database. Many of our users backtest every day and do not experience any discrepancies. To confirm this for yourself run a backtest of SampleMACrossOver. The results will not change considering your data and parameters remain constant.

[heech]

Josh,

I don't really understand how bars can be missing, but there isn't a warning or error. Here's the behavior I'm seeing... I happen to have a repeatable problem right now, so... easy to demonstrate.

My code says:

... onBarUpdate() ...

if (CurrentBar == 0) // First time in...
{
Print("First bar:" + Time[0]);

I have set my backtest to start with 08/19. When I run with most symbols in my list, I correctly come up with:
First bar:8/19/2008 7:00:00 AM

When I run with a specific symbol (AMR in this case), I end up with this output:
First bar:8/20/2008 7:00:00 AM

In other words, the first day of bars are simply not there... and there's no warning, no indication that I can see.. I wouldn't know this was happen *unless* I explicitly look at the chart and search for missing bars. I've also explicitly closed/reopened NinjaTrader several times, but the same problem persists.

For what its worth, I might have "created" the problem by originally running the backtest from 08/20 (hours and hours ago). But I've since adjusted the parameter to 08/19 and tried to run repeatedly (and successfully on other instruments in the same list)... so I'm baffled what's going on here.

Now, following the steps you suggested (going into tools->historical data->reload) did work. It made the problem go away... but it's still a huge issue, because I don't know how to use this thing. I have instrument lists with 50+ symbols. Do I go through and hit reload manually on every run? Is there some way to reload the entire instrument list?

Most importantly, is there no way to detect the presence of missing bars in the first place, without looking over everything with a fine-toothed comb?

[NinjaTrader_JoshP]

heech,

You are not missing bars. If you can pull up a chart of it then you have the data. Just disconnect from your data feed and ensure you are loading from the database. Also remember that NinjaTrader strategies won't even start calculating till at least 20 bars have gone by. This is a default setting under the parameter "Min. bars required".

[heech]

Josh,

If you can pull up a chart of it then you have the data. Just disconnect from your data feed and ensure you are loading from the database. Also remember that NinjaTrader strategies won't even start calculating till at least 20 bars have gone by. This is a default setting under the parameter "Min. bars required".

This is the key point. I *can't* pull up a chart of it.. not of the missing data.

Going again with the example previously. I have the backtest set to start on 08/19. I run the strategy... no warnings, no errors, everything looks fine. I look at the chart, and the first bar of the chart is 08/20. I modify the code to dump out Time[0] on the first bar, and the first date going into OnBarUpdate() is also from 08/20.

I close everything, rebuild everything, use Historical->Reload... and now when I have backtest set to start on 08/19, I actually get data on 08/19.

[NinjaTrader_JoshP]

heech,

If you can't pull up the data it means you don't have the data. Just reload the data so you have it. Then you will have it in your database. If you reload and you still don't have the data it simply means your data provider is not giving it to you.

NinjaTrader works like a radio receiver in terms of data. There is no funny business going on. If its there it is there. If its not it is not.

When you backtest for sure it will not give you a warning. It will load what it can. If its not there it will load what is there.

[heech]

When you backtest for sure it will not give you a warning. It will load what it can. If its not there it will load what is there.

Josh,

I recognize that's what it's doing. (And I also recognize partly what causes the condition... and it's *not* the provider. It only happens when I begin to tweak the start/stop dates on back-testing.)

What I want you and the NT development team to realize is that this is really, really, really poor behavior. It makes back-testing incredibly unreliable, when results are transparently changing on the basis of background behavior that's not knowable by the user.

That's the first question I asked. I realized reloading it solves it, but when I'm dealing with 100 symbols... when do you I suggest I reload? Out of paranoia, do a reload every 5 minutes, every time I run a backtest? (And as far as I can tell, there's no way to reload an entire instrument list.. I have to select each instrument within the list individually.) We get warnings when there's a "compressed data mismatch" (see other thread)... why aren't we warned that bars from XXX to YYY are missing from the database?

[NinjaTrader_JoshP]

heech,

I understand your frustration, but we do not have any other report like yours. If your data set remains constant your backtests will also remain constant provided you have the same parameters.

You don't need to reload ever. Just load once. Ensure you have the data set you want. Once you have the desired set you can completely disconnect from your data feed and backtest from there.

[heech]

Not sure I understand your attitude. The change required from your side would be trivial from a programmatic point of view.

I guess for my sake, I can only hope many other people start tripping into this latent, hidden, but very obnoxious bug in your software.

[NinjaTrader_JoshP]

heech,

Perhaps we are having a misunderstanding. I am only trying to convey that if your data set and parameters are the same your backtest results will also remain the same. There should be no bug here. Unless you are able to give us a reproducible scenario where your data set and parameters are the same and your results are coming up different we would not know how to address your issue.