Retrieve Historical Data oldest date

[samlam]

Greetings,

My problem is simple, I backtest a stategy that use hundreds of symbols. Unfortunatly, from what I read and experiments, Ninjatrader have a "feature" that prevent the backtest from computing itself if only one symbol is missing. Furthermore, it seem that, if all symbol do have some data, the backtest will only work from the newest starting point.

For example: I want to back test from 2001 to 2009 (using Daily/Months bars) I only get result starting form 2008 to 2009 because some symbol(s) somewhere do not have 2001->2008 data (which is perfectly understandable)

This is all fine and good, I can understand the reasons behind this, however, I failed to discover where I could get the information on which symbol(s) has missing data. (So I can remove them and backtest with the other at least)

Last weeks I have checked every single symbol using the chart windows and noted all the symbols who didn't had data to 2001. I then removed them from my lists and obviously, since I'm human, I missed some it seem.

I failed to figure out a way to automatise something that would tell me the oldest day of each symbols on my list (since NT trunk valid data). So if there is somewhere I can get which symbol(s) are the bottleneck or an idea to automatise the retrieval of the oldest dates of each symbol I'd be grateful.

Thank you in advance.

[NinjaTrader_JoshP]

samlam,

Thank you for the suggestion. We will add it to our list of future considerations.

[samlam]

Thanks for your help.

Although I did a little more exploration this morning in the database without much luck. However I stumbled upon something very interesting in the trace log of Ninjatrader regarding this issue.

Basically here is a partial solution to my problem. Basically, you need to run the backtest twice, I believe. The "first" backtest is (depending of your system/instrument list) is done until you can all the data has been received. In another words when NT stop crashing due to out of memory or when the backtest worked (because sometimes I have to start a backtest 6-7 times before it actually work, dunno if it's an NT or IQFeed problem).

So once you managed to have a valid backtest, you do a second backtest (obviously with the same start date). Once this is done, you go in the trace file (.../MyDocuments/NinjaTrader 6.5/trace) and in the current day trace file you will see line like:
2009-05-15 16:52:55:718 (IQFeed) Data.Bars.GetBarsNow: instrument='SPN' from='2001-01-01' to='2009-05-15' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-05-15 16:53:16:543 (IQFeed) Data.Bars.GetBarsNow: instrument='SPW' from='2009-05-14' to='2009-05-15' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=Daily splitAdjusted=False dividendAdjusted=False bars=0

As you can notice, the from of both of them are quite different. The one with '2001-01-01' (My requested start date) is an incomplete symbol. The second one only needed to request the current day so it has a from of the previous day (I did my back test last Friday (2009-05-15), although my requested end date was '2009-05-09').

So, you don't have the oldest date for SPN anywhere, but you know it's incomplete. If I open a chart I do have all the date for SPW, but SPN as only from 2001-05-15.

I verified quite a few symbol from the trace log and I haven't found one discrepency so far.

[NinjaTrader_JoshP]

samlam,

NinjaTrader loads data whenever one of these are breached: http://www.ninjatrader-support.com/H...HistoricalData

[samlam]

I am totally aware of that and it's also because one these conditions is breached on one of the symbol I use (from a list of hundreds) that I cannot backtest properly.

As I excludes symbol that breach these conditions from my list I can backtest further and further. The problem is finding which one it is.

[NinjaTrader_JoshP]

Unfortunately you will just have to find the one you don't have data for manually right now. Strategies will start calculating the moment all bars objects have data for a given timestamp.

[samlam]

Here is a little ugly code snippet example to retrieve the list of incomplete symbols:

            string tracepath = @"C:\Documents and Settings\UserName\My Documents\NinjaTrader 6.5\trace\trace.20090520.txt";
            string outputpath = @"C:\Documents and Settings\UserName\My Documents\NinjaTrader 6.5\trace\exceptionlist_generated.txt";
            string tmppath = string.Concat(tracepath, ".tmp");
            string searchstring = @"from='2001-01-01'";
            string startsymbol = @"instrument='";
            string endsymbol = @"'";
            int firstpos = 0;
            int length = 0;

            StringBuilder strb = new StringBuilder();
            string[] lines = { };

            File.Copy(tracepath, tmppath);
            lines = File.ReadAllLines(tmppath);
            File.Delete(tmppath);

            foreach (string line in lines)
            {
                if (line.IndexOf(searchstring) != -1)
                {
                    firstpos = line.IndexOf(startsymbol) + startsymbol.Length;
                    length = line.IndexOf(endsymbol, firstpos) - firstpos;

                    strb.AppendLine(string.Format("{0},", line.Substring(firstpos, length)));
                }
            }

            File.WriteAllText(outputpath, strb.ToString());