Bug in either Help or HistoricData import

[Moog]

Hello,

I wrote a small conversion program in C# to convert my txt files into the ; delimited no OI and with a 6 digit timestamp, ie so I can import historic data. As I want to check out the flexibility in the backtesting features.

The error reads in the log:

Unexpected number of fields in line, should be 6 or 3.

The help files are showing 7 fields, for 1 min intra day data, whats the true situation, bug or help file issue? Im gonna mess with it, ie remove the volume from the intra day and see if that cures it. Whats the score with the daily fields, what should they be?

Please let me know.

Also,

I dont know whether it is possible yet, as Im still trying to import the data but does the backtesting suite support analysis from the portfolio and or sector level? If not then I think this addition will make for a truly wonderful and powerful overview. Im also hoping that position algorithms can be set and incorporated into the backtest feature (Im presuming so)?

I wrote my own intra-day testing harness in C++ which took a few years and I believe if you crack the portfolio and sector analysis, youll make my own work redundant to some extents.

I'm laughing...



Cheers

[Moog]

imported post

Update:

Intra day will not import, Input string not in correct format, Ive done this with vol included error see last post, and vol excluded, this error.



Daily imported however if volume is zero, illegal volume reported.

Removed the 3 zero volume days and it successfully imported. Cant seem to access the data though? Where is it? Charting and Strategy analyser with much tooing and froing was unsuccessful at reading the imported data stream?

Comments?



Cheers

[NinjaTrader_Ray]

imported post

Hi Moog,

Minute format in the Help Guide is 6 fields:

yyyyMMdd HHmmss;open price;high price;low price;close price;volume

Tick format in the Help Guide is 3 fields:

20061107 000431;1383.00;1

Zero volume for sure is a problem since it's invalid. Should we allow zero volume to be imported?

Portfolio, risk and money management is currently not supported however they are all on our list for future development.

Imported data is stored in a local database. To test, just open a chart for the data that you imported and see if it comes up.

Ray

[Moog]

imported post

Hello Ray,

Important that you let the zero volume through, otherwse relentless import dramas, testers would tend to handle that anyhow their sidewith pre-trade volume/tick filters, it was good seeing the warnings however with the line numbers, good idea may be to print the line numbers to a separate txt log file, that will have benefits on any external re-read correction edits the tester may wish to engage in.

My intra day database goes back to the mid 70s and as thats pre-computer tec there are times when volume/ticks may be iffy on the intra day and even patches where it is non existent.Also volume filtering would tend to be picked up off the daily stream, which would most likely bethe secondary stream inany main strategy testing intra day.

The issue with letting volume be illegal is that on the intra-day data, just because there were no up or down ticks doesnt necessarily mean you wouldnt have got a trade, also the dataerrors are expounded by the further back in time scenario.

Also I think from a backtesting standpoint of say 5 min data over 30 years on a single instrument using single contract trading (ie looking fornatural edgestats) thevolumeisnt that important. Whereas from a edge honing exercise around the bid offer based over say the past 12 months on the electronic markets its more significant there. I think, allow it and filter for it in the script. Obviously in any real time or sim tradingyou wouldntwant that tobe the case.

On another note, A good idea would be to develop a portfolio instance and a sector instance based upon instrument symbol list pointers. So the user clicks on the instruments to be part of the backtest portfolio on the GUI, that creates the list pointers and as the backtest runs each instrument is looped, on a per systemper timestamp basisand posts out to the port and the sector. Powerful.

I wasnt too concerned about moneymanagement as hopefully that can be tended to in the script editor however one would need to introduce position objectsholding unit objects, that wayyou can use FIFO and LIFO looping to scale in and more importantly to scale out of positions without upsetting the position flow logic. Then all you need to do is post the number of contracts purchased/sold in the position object and increment and decrement as the objects are created or destroyed. Hope this is not too boring but its how I tended to mine if its of any help.

As I know I said in a private email but I think needs to be said publically, in my humble opinionNinja has the potential to be an awesome piece of software. The leap from v5 to 6 was a real eyeopener when I looked. Congrats.

Over the data import, I er somehow saw a ; between the date and timestamp that wasnt there, funny how your eyes play tricks, I normally have the txt files comma separated and theres a comma between the two in mine, where are my glasses...?

Let me just removethe ; and adjust the timestampsand try and import some 1 min...

OK, it worked however, I still cant see the data in the charts, even importing 100 bars or so. I have setup the instruent database but what am I missing? Agood idea is top allow the user to setup the instrument database via a dialog before the import process occurs and to maybe attach or to be able to clone to an existing symbol in the data dictionary thereby reducing the drama time. Another idea would be to include non electronic instruments in the data dictionary so that backtesting routines can be run. The NYBOT has morphed in one formonto NYMEX now, theres no liquidity yet but that wont last forever.

Have you run any tests benchmarks on the time it takes to run say a 3 min backtest on 30 years of data on a single market? I would be interested to know as I cant get this working as yet my side.

Lastly, I think a powerful aspect ofNinja is the potential ability of hooking up with the contra-to bid ask routines in your script, Im hoping that its robust to a large extent inside there and not too buggy. The Guis were perfect and intuitive on the DOM I loved the bid ask exposure boxes each side of the spread, I hope that can be implemented programatically with minimum fuss. The methods seem to be there I looking forward to delving in...

Regards

Moog

[NinjaTrader_Ray]

imported post

Hi Moog,

Thanks for your thoughts, they are appreciated.

Volume - Currentlywe zero value volume records are valid for indexes but not for other instruments. I understand your points but for now we will leave it as is and force users to clean up their data. This could change as we get more feedback.

I truly appreciate your compliments on the evolution of NT5 to NT6. A lot of work has went into this and we are just getting started. We are still making advancement in our charting and have just spent the last few days really tweaking our internal order handling routines to be industrial strength. Although some exchanges limit order changes to 5 per seconds, the next version of NT can handle 50+ client side changes per second. Although this is not practical, if someone really wanted to push the pedal to the medal with tick by tick order submission, changes and cancellations, NT can handle that with ease.

I just ran our sample MA stategy against 2 years historical 1 minute bars. Completed in about 4 seconds. I have run more complex strategiesthrough optimization side by side identical TS strategies and in some cases we ran about 10x faster than TS and others as slow as 3x faster

Regarding historical data import, send me a private mail to ray at ninjatrader dot com and I will send you a test import file with further insructions. I can't post the file here since there are size limitations.

You can access bid/ask prices via the strategy but accessing depth is not officially supported. Once you get through some basic understanding of NinjaScript, send me private mail and I can point you in the direction on how to attach market depth event handlers and aggregate it on your own.

Ray

[Moog]

imported post

Thanks Ray,



Its importing successfully but still no data for historic, all the sim and TS data is working perfect except for the imported stuff. Ill email you privately, cheers