Import Historical Data (ASCII)

[Aspirant]

Hi Guys,

We have about 10 years of minute by minute data I'm attempting to import into Ninja, but nothing seems to work. Here's what I've done.

Firstly, the data was converted to the appropriate format. Next I went through and set all the txt file names to correlate to "Interactive Brokers" symbols as that is the broker we're using and those symbols are mapped into the "Instrument Manager".

Next I went to the "Historical Data Manager > Import" tab.
In the "Format" list box I chose "NinjaTrader".
In the "Data type" list box I chose "Last".
In the "Time zone..." list box I chose "UTC"
... and I also set to generate Day bars...

I then hit the "Start Import" button and all the data imports fine terminating with the pop-up "Historical data imported successfully." Checking the Log it just states the volume is invalid (=0) and was modified. I presume this didn't cause a critical failure as it says it was replaced and the previous message indicated a successful import.

Also, if I go to the "Historical Data Manager > Edit" tab I can see both of instruments there (AUD & HSI) and can review the data imported, however, the AUD data is truncated to two decimal places (even though the 6A/AUD instrument is set to 4 decimal places).

I then went to "Instrument Manager" and created a new "Instrument List" I called "Test." To this I added the two "Futures" instruments I imported as follows.

I typed in "Australian Dollar" and symbol Name "6A". I went to "Edit" this symbol and confirmed all the fields including that it had "AUD" as a mapped symbol under "Interactive Brokers". I then set the expiry as ##-## for a continuous contract and clicked the Left Arrow to add it to the Instrument List.

I then did the same for the HSI which has the same NT symbol as Interactive Brokers and clicked "Okay".

I then went "File > New > Chart" and from the "Data Series" window opened my Instrument List ("Test"), picked the HSI ##-## symbol clicked "New" to display 5 days for 1 minute data (as is the default)... and I got nothin'.

If the data symbols weren't mapped the data would not have imported... but then there does not appear to be any way of telling NT that the symbols it's importing are Stock or Futures. So I also added the AUD and HSI to my "Instrument List" as stocks, but these didn't plot either.

So, then I thought the txt files might need "##-##" suffixes, so I changed the file names from AUD.txt and HSI.txt to AUD ##-##.txt and HSI ##-##.txt. This time the HSI imported fine, but the AUD fell over with the log saying that the "Instrument is not supported". Anyway, as HSI ##-##.txt imported I added that to my "Instrument List" but that still wouldn't plot.

Ideas? I've wasted so many hours on this it's not funny.


Please help.

[NinjaTrader_Bertrand]

Aspirant, welcome to our forums.

Since you did not include an expiry for the futures contract (the distinguishing element between stock and future) it was imported as a stock on your first run and hence you saw the digits being truncated unfortunately, as stocks generally have a ticksize of 0.01

If you check the imported data via the Historical Data Manager for your custom created futures symbols, does that show the data imported correctly and also having correct timestamps?

I could only imagine the timestamps being off because of a local format and thus when you request the chart the needed range is simply not available.

If you can email me the files zipped to support at ninjatrader dot com I would be glad to take a look and give them a run here on our end for you.

Thanks,

[oosthout]

Aspirant,

The symbol labels IB uses may differ from those applied by the Exchange/Ninja Trader. For instances 6E in Ninja is EUR in IB. If you rename the data files, the labels should be identical to those use by the Instrument Manager.

[Aspirant]

Importing ASCII data

Hi Guys,

Thanks for the responses.

As stated, I recognized that AUD and HSI may have been mapped as stocks, but when I applied the AUD and HSI stock symbols to the Instrument List they didn't chart (in 1m) either.

I have tried naming the AUD & HSI futures files (note data finishes Mar-11) as follows:

1. 6A.txt
2. HSI.txt.
3. AUD.txt.
4. 6A ##-##.txt
5. HSI ##-##.txt
6. AUD ##-##.txt
7. 6A 03-11.txt
8. AUD 03-11.txt
9. HSI 03-11.txt
10. 6A03-11.txt
11. AUD03-11.txt
12. HSI03-11.txt
13. 6A##-##.txt
14. AUD##-##.txt
15. HSI##-##.txt

1. didn't import because 6A is an unsupported instrument (not a stock I guess).
2. & 3. imported to 2 decimal places because NT thought they were stocks. They didn't plot 1m data, but did plot daily. Note: the stock symbols use a 24/7 Session Template.
4. imported 1m and daily to four decimal places but only plots in daily data not 1m.
5. imported 1m data but "Historical Data Manager" did not create Daily data despite the check box being checked. Therefore it would not plot daily, but would not plot 1m either.
6. didn't import because the "Instrument is not supported."
7-9 had the same issues as 4-6.
10-15 don't work because there is no space between the symbol and the expiry. That's fine, but 6 & 8 don't import either, even though there is a valid expiry date on the file.


Following is the data.

http://dl.dropbox.com/u/17138407/6A%20%23%23-%23%23.txt

http://dl.dropbox.com/u/17138407/HSI%20%23%23-%23%23.txt

Bottom-line, I'm still nowhere and I look forward to your assistance as I have a lot more symbols to import.


Cheers.

[NinjaTrader_Jason]

Hello Aspirant,

You will need to make sure the following instruments are added in the Instrument Manager before importing.

6A ##-##
HSI ##-##

Name the .txt files:

6A ##-##.Last
HSI ##-##.Last

Make sure you select the proper time zone. Subsequently check if data is imported successfully.

I tested a portion of the HSI data and it imported without error. I am not sure what time zone is used for the data at the links you provided. I used UTC time zone. When I compare the data, the time stamps are off. There is a difference of 5 hours. Please select the proper time zone when you import it and check if data is imported correctly.

[Aspirant]

Importing ASCII data

Hi Jason,

You suggested that I add the following instruments to the "Instrument Manager".

1. 6A ##-##
2. HSI ##-##

I presume you mean by 'add' you are referring to the "Master Instrument" field in the "Instrument Editor" window of "Instrument Manager".


If that is the case,NT only allows numbers and characters in this field (i.e. not #). If you are referring to something else, please elaborate.


That aside, I'd rather not create a new symbols for all these Instruments that are already defined in the Instrument Manager with all the vendor maps I need, pPlease help!





Time zone... Yes, for some stupid reason NT seems to want me to set a single time-zone for the import instead of it being a field in the actual instrument.


HSI is the Hang-Seng Index, this is +8, 6A is the CME Globex USD/AUD contract, therefore the time-stamp should be -6.


These setting seem to not make any difference to the successful plotting of the intra-day data.


How did you get it to plot specifically?


I'm using v7.0.1000.8 in Win7 SP1 32bit.

[Aspirant]

Importing Historical ASCII data

Hi Jason,

You suggested that I add the following instruments to the "Instrument Manager".

1. 6A ##-##
2. HSI ##-##

I presume you mean by 'add' you are referring to the "Master Instrument" field in the "Instrument Editor" window of "Instrument Manager".


If that is the case,NT only allows numbers and characters in this field (i.e. not #). If you are referring to something else, please elaborate.


That aside, I'd rather not create a new symbols for all these Instruments that are already defined in the Instrument Manager with all the vendor maps I need, pPlease help!





Time zone... Yes, for some stupid reason NT seems to want me to set a single time-zone for the import instead of it being a field in the actual instrument.


HSI is the Hang-Seng Index, this is +8, 6A is the CME Globex USD/AUD contract, therefore the time-stamp should be -6.


These setting seem to not make any difference to the successful plotting of the intra-day data.


How did you get it to plot specifically?


I'm using v7.0.1000.8 in Win7 SP1 32bit.


P.S. Every time I hit "Submit Reply" to post to the forum it tells me I'm logged out and I have to go back in, recreate my composition (from my back-up thankfully) then post again.

[Aspirant]

Importing ASCII data

Hi Guys,

Any other ideas? This is going on Day 3 for me now.

Do I need to create a new Instrument symbol for each ASCII time-series I want to import, even though all the symbols are already defined the Instrument Manager?

If so, what name do I give them and why is this different to what I've done?

Please help.

P.S. When I mentioned that I could get daily data on the HSI & 6A symbols, it was actually only 2 months, however, when I go to weekly or monthly I seem to be able to see the years of history. What I need for now is the intra-day data, and I can't get any of it to plot even though I can see it all in the Historical Data Manager.

P.P.S. I tried using the 6A ##-#.Last.txt and HSI ##-#.Last.txt file names. Now NT seems to be creating the daily data from the 1min for HSI which it wasn't before (it was doing it for the AUD).

[NinjaTrader_Brett]

Hello,

I tested your file. worked perfectly for me that you put on the drop box.

The 6A is the instrument the ##-## is the expiry. The 6A ##-## is already set up in NinjaTrader therefor no need to check what Jason mentioned unless something has been deleted.

Steps I took is imported your 6A ##-## file as it was already correctly labeled and imported it.

Waited for the import to report is was a success. Verified in the Edit tab of the instrument manager that the data was there. Is is there for you for this instrument?

If it is go ahead and pull up a chart, it will pull up black that is because its trying to display data from today 5 days back with default settings. You needed to right click data series and change the data load settings to custom range or specify enough bars back to load.

To clarify further the ##-## contract is available for all futures instruments and tells NinjaTrader you are importing the continous contract. You will want to use the continuous contract for data that goes back this far.

Let me know if any questions.

[Aspirant]

Imported ASCII data - not plotting, but can run Strategies...

Hi Brett,
If I run one of my strategies over the data I can get the 1m data to plot in the Strategy Analyzer, but until I do that it will not chart in either the Strategy Analyzer or File > New > Chart.

May have something to do with the data ending in March 2011. Either way, not impressed, but at least I can run the data in the Strategy Analyzer.

I'll import the rest of my data as XX ##-##.txt and see how we go. Just mapping symbols to the NT symbols instead of the IB symbols I originally set.
Cheers,

[NinjaTrader_Brett]

Sounds good.

Finally as soon as the data is imported you can File->new->Chart it.

-Brett

[atrader123]

Problem with importing monthly historical data

I am not able to import historical monthly data into ninjatrader. Following is the error in ninjatrader log - "MEAFE: Import field separator could not be identified".

As per instructions in the help section, following is the data file I created. Data has only closing prices. Can you please suggest on how to resolve this issue?

File name: MEAFE.Last.txt
Content sample:
19691231 000000;100
19700130 000000;98.624
19700227 000000;96.097
19700331 000000;97.34
19700430 000000;88.966
19700529 000000;84.28
19700630 000000;85.85
19700731 000000;89.003

Regards

[NinjaTrader_Jason]

Hello atrader123,

You can only import data in the format listed at the link below under 'Understanding import file and data formats'.

http://www.ninjatrader.com/support/helpGuides/nt7/importing.htm

Unfortunately, it does not include monthly data. We only support tick, minute and daily data.

Please note that you can build monthly charts from historical daily data you import. Weekly, monthly and yearly charts are build by historical daily data.

[atrader123]

Thanks. Looks like I need to find another alternative like R or Multi charts. I have a lifetime license to Ninjatrader for couple of years now. But I am finding myself more and more reasons to switch and I suspect that might be case with other traders as well.

I can list more but to give an idea following are some that one cannot do in ninjatrader but probably needed by many traders -
(a) Cannot plugin ATM strategy with custom code to trigger after placing a trade discretionary..

(b) Cannot do any portfolio level back testing nor at least have any place holders to plugin pre and post code before strategy executes on a watch list.

(c) Cannot export automatically results of strategy on multiple markets. Why not atlteast expose the APIs the UI uses to display the strategy results?

(d) Cannot do any ranking functionality on a watchlist. There is whole class of strategies that would need this functionality.

(e) Doesn't have built indicators for some common stuff like Relative Strength, Volatility stops etc etc. If one has to pay to consultants for basic stuff then why do I need ninjatrader?

(f) No easy way to import and limited functionality for data imports.

(h) No way to plugin any custom metrics into strategy summary results except manually exporting to excel and doing the computations.

It would be useful if Ninjatrader provides a roadmap of what is coming next, when and which type of traders the product is planning to target in future and quicker releases. On side note, the post is not a reflection of support.

[NinjaTrader_Brett]

Hello,

Thanks for the post and the honesty, we are always looking for this type of feedback.

Unfortunately I am unable at this time to provide any road map of future possible features while we are in development for our next major release. We do not have an expected release date at this time however what I can say is they we have a lot of good stuff and user suggestion being added into our next major release of NinjaTrader. Just like from NinjaTrader 6.5 to NinjaTrader 7 we added many features we will continue to do the same for future major releases.

The features you are discussing are ones that are on the suggestion list for development to review and I will have your requests added.

Thanks for the comment on support as well, we do our best.

-Brett