Strange problem importing data from txt.files for generic instrument

[koalboy]

Hi guys,

I'm on NT 7.1000.6

I want to do following:

• Plotting Values of COT-Report on a Chart.
• Each Instrument (in total 46) I want to track has 4 to 7 subvalues (OpenInterest, CommercialNetPosition, NonComsNetPosition, Spread% etc...) so all together I have around 550 Datapoints each week (the COT-Report is published weekly).
• Because I'm not a programming professional, I took the "dirty" way and created an Instrument for each Subvalue (550 new Instruments added via "symbol import".
• To plot historical Data (from 1995-2011) I downloaded the original files from CFTC website and processed them with excel VBA to make txt.files out of it, which can be imported by Ninjatrader.

here is some example of the naming logic -> they represent the Symbol-Name for adding to the InstrumentManager and with the extension ".txt" the are named for the data holding txt-files for importing:

"COT_[Symbol]_[Type of Report]_[Subvalue]"


COT_MXN_TFF_5LEVMon, COT_MXN_TFF_6OthRep, COT_MXN_TFF_7NonRep, COT_NDQ_COToV_1OI, COT_NDQ_COToV_2Spd, COT_NDQ_COToV_3Coms, COT_NDQ_COToV_4NonComs, COT_NDQ_COToV_5NonRep, COT_NDQ_TFF_1Oi, COT_NDQ_TFF_2Spd, COT_NDQ_TFF_3Dealr, COT_NDQ_TFF_4Assmgr, COT_NDQ_TFF_5LEVMon, COT_NDQ_TFF_6OthRep, COT_NDQ_TFF_7NonRep, COT_NDQmin_COToV_1OI, COT_NDQmin_COToV_2Spd, COT_NDQmin_COToV_3Coms, COT_NDQmin_COToV_4NonComs, COT_NDQmin_COToV_5NonRep, COT_NDQmin_TFF_1Oi, COT_NDQmin_TFF_2Spd, COT_NDQmin_TFF_3Dealr, COT_NDQmin_TFF_4Assmgr, COT_NDQmin_TFF_5LEVMon, COT_NDQmin_TFF_6OthRep, COT_NDQmin_TFF_7NonRep, COT_NG_CDY_1OI, COT_NG_CDY_2Spd, COT_NG_CDY_3Prod, COT_NG_CDY_4Swap, COT_NG_CDY_5MoMg, COT_NG_CDY_6OthRep, COT_NG_CDY_7NonRep,

Everything works fine in the first place. All my 550 "instruments" show values in the Historical Data Manager/Edit Section. Also I can do the plotting on a chart like I wanted... BUT:

On the Charts there were strange periods of unchanged Open Interest for several weeks -> I took a look in the original Data from CFTC and also in the txt.files for importing -> the values in the database and hence plotted on the chart were not right!

I looked up the values in the "Edit" Seciton of Historical Data Manager and they were wrong, but not all data for a single instrument (only like est. 20%) and most instruments don't show wrong data at all.

As the naming logic of my 550 instruments makes their filenames quite similar I tried some debuggin (I erased all 550 generic instruments from the sdf-database (MyDocuments/Ninjatrader 7/db/***.sdf) before each attempt):

Trying to check the naming problem I changed the naming logic to much more different names for the instruments but it didn't help (which is logic to me, as every instrument gets an unique ID in the sql-database).

After deleting all the instruments in the database again I tried to create only a single instrument and import data for it -> the problem still remains. The Data I imported from the text file shows - to some extent - not the same values in the Edit-Section of Historical Data Manager.


I attached "screenshot1" showing contemporarily:

- the Data Manager / Edit Section with the values imported from
- txt file -> look at the red circles -> NT didn't import the right values
- Chart with databox -> daily -> showing the data of Data manager (what else)

I also attached the txt.file to import. The specs for generating the proprietary instrument I used for the "only importing one single instrument"-attempt are (screenshot2):

MasterInstrument: USTECHMIN_COTOV_1OI
Instrument type: Stock
CCy: usDollar
TickSize 0,01
Point Value 1
Session Template: Default 24/7
merge policy: useglobal settings
Excanges: "Bm" and "Box"
Symbol Map: only in row Collective2 "asdf"

I'm running out of options here, absolutely no clue why it's not showing the right data...
Please help!!!

thx

[NinjaTrader_ChristopherJ]

Hello koalboy,

Thank you for your post.

Within the Instrument Editor please enable the Default Exchange as well. Next please add the instrument to your instrument with Default selected. Next open a new chart with the dates set to correlate with the dates imported.

Please let me know if the issue persists after this procedure.

[koalboy]

Hi Christopher,

thx for the fast reply.

your suggestion didn't work, same problem still occurs...

I tell you what I did:
as I told you in my first thread, I was erasing all generic instruments from the sdf-Database before trying to work with only one instrument -> so only one of the generic instruments was left in the instrument manager and also only one set of imported data for this instrument.

Following your instructions I deleted the instrument from my own List "COTData" and then readding it with choosing "default" for the exchange value. Afterwards I opened a chart with value "1875 days back" from todays date. Still the wrong data is shown. I checked in Edit Section of Historical Data Manager and of course the data were still "wrong" as well.

The I deleted the instrument in the instrument manager, created a new one with checking also "default" when using the exchange-selection. Then I added this new instrument to the "default" InstrumentList and imported the data from the textfile (same like attached in first post). Again the data shown in the edit section of Historical Data Manager showed not the same values like in the text file...

The must be some kind of bug when importing the data via the text file. I wonder if this only occurs on my end or if you have the same problem if you try it on your machine?

thx

[NinjaTrader_ChristopherJ]

So I may attempt to test on my end can you tell me the specific symbol used for the .txt you attached. Is it COT_oV_1OI?

The file must be saved with the same name as the instrument you are attempting to import.

For example if you are importing data for AAPL the file needs to be named AAPL.txt.

[koalboy]

according to the name of the attached txt.file in post #1 name of the symbol to add should be USTECHMIN_COTOV_1OI ...

I'm curious!

[NinjaTrader_ChristopherJ]

when reviewing the Daily data it appears that the Open, High, Low and Close are all the same price with volume of 1 as seen in the attached screen shot.

Is this correct or should be values be different?

[koalboy]

hi christopher,

yes, the format is correct. Please take a look at "screenshot1.pdf" file on the first post attachment.

For Date Range 24.03.2009 to 09.06.2009 you see contemporarily the values of the text-file and the values in the Edit-Section of the Historical Data Manager and also a chart with this values - all three items show the date range from march to june 2009.

And on my machine the values of the text file are not identical with the values in the Edit-Section (and hence on the chart) after importing them.

Does this behaviour also occur on your system?

thx

[NinjaTrader_ChristopherJ]

I have imported the .txt file you have attached to your original post and have attached a screen shot of the chart that is populated with this data.

[koalboy]

Hi Christopher,

I attached a Screenshot (Screenshot3.pdf) with a direct comparison of the values in the text file (which should show up on the chart) and the chart which is produced by NT on your end (your attachment of the last post)... I marked some of the line values in the txt.file which are plotted wrong...

As you can see, the chart shows not the values given in the text file, so the problem occurs also on your end, not only on mine...

Do you have any explanation for this wrong importing behaviour of NT?

[NinjaTrader_ChristopherJ]

Please right mouse click within your chart-->Data Series-->Days to load-->Set to 1000-->Click Ok.

Please let me know if you then see the data correctly as referenced in my attached screen shot.

[koalboy]

I get the same chart like you -> the wrong data for the period checked in Screenshot3.pdf (attached above) now is shown right BUT:

now there are "wrong" values at other places in the chart on this 1000 days setup -> everytime when the pirce-value stays flat for a longer period of time (march/april 2010 on your chart 4 weeks same value -> not complying with values of txt.file; or August/September 2009 or April/May/June 2009) it is not right. -> the price doesn't stay flat and this can be proofed when looking into the data of the import-txt-file...

I'm desperate...

[NinjaTrader_ChristopherJ]

I will need to do additional testing on my end to see why these are plotted at the same price. I will follow up as soon as I gather additional information.

[blackhawk]

I have found the solution. All the process is correct but you need to open Instrument Manager >> Instrument Editor and change "Merge Policy" to "DoNotMerge". You add the instrument to Instrument List again and your chart will be fine.