COT Report

[zweistein]

Hello,

I have started using NT only about 6 month ago, but I find it so fantastic that I sometimes cannot stop developping stuff for it.

NT is great and the C# language is so powerful and flexible that up to now I always managed to do what I want in NT.


Since I had never parsed http , here a code for reading the

COT data from the

http://www.cftc.gov/dea/futures/

website.

The Cot indicator presented will

- read current or archive COT Report directly from the cftc.gov web site.
- parse all this into a Commitment class
- Print in an Indicator the Long/Short/Spread Positions for Commercial/NonCommercial or NonReportable participants.

- limitiations:
- some parts of the code unfinished.
- DateTime is working for GMT+1, as hours are hardcoded into.
for EST some hour offsets are needed

- load time is long as the indicator will download (from within OnBarUpdate) all reports needed from cftc.gov



I would like to offer my source code here under the Limited GNU Public Licence,hence you can use the code for yourself but you may not distribute it further without notice to me, please through this forum.

Commercial use of is prohibited.


Here a screenshot:
COT offers OpenInterest for each participant group, so please select

Contract-USTreasuryBonds and OpenInterest-Short
and try out.


Actually there is quite much thinking in this code and I put in here in the hope to get a nice general discussion out of it, and I also hope to get good ideas and good tips from others

Best regards


Andreas (zweistein)

[MicroAl]

Cot Report

Many thanks for sharing!

[Elliott Wave]

THANK YOU!

I've been hoping someone would tackle the HTML parsing and create a proper COT indicator.

Thank you for also making it free, open and asking others to contribute. I never really liked the idea of paying a subscription for an indicator that only changes 4 times a month...

[Elliott Wave]

I noticed the indicator at this point only works for 3 contract types so I tried to add some others I think people will be interested in (currencies, S&P, E-Mini etc).

"CANADIAN DOLLAR - CHICAGO MERCANTILE EXCHANGE",
"SWISS FRANC - CHICAGO MERCANTILE EXCHANGE",
"BRITISH POUND STERLING - CHICAGO MERCANTILE EXCHANGE",
"JAPANESE YEN - CHICAGO MERCANTILE EXCHANGE",
"EURO FX - CHICAGO MERCANTILE EXCHANGE",
"3-MONTH EURODOLLARS - CHICAGO MERCANTILE EXCHANGE",
"S&P 500 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE",
"E-MINI S&P 500 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE",
"NASDAQ-100 STOCK INDEX - CHICAGO MERCANTILE EXCHANGE",
"NASDAQ-100 STOCK INDEX (MINI) - CHICAGO MERCANTILE EXCHANGE",
"DOW JONES INDUSTRIAL AVERAGE - CHICAGO BOARD OF TRADE",
"WHEAT - CHICAGO BOARD OF TRADE",
"2-YEAR U.S. TREASURY NOTES - CHICAGO BOARD OF TRADE",
"10-YEAR U.S. TREASURY NOTES - CHICAGO BOARD OF TRADE"

At this point though it appears only one exchange can be supported at a time, so I changed it from the "Chicago Board of Trade" to the "Chicago Mercantile Exchange".

The bad news is the original three supported contracts (Wheat, 2yr and 10yr treasury notes) won't work, but the good news is hopefully everything else will.

I assume there are more currency and S&P traders using NT than Wheat traders anyway.

At this point I think we need some people to test it, particularly people with plently of historical data (I don't have much).

In terms of coding I think a good thing to work on would be supporting all the exchanges. A good way to do it might be to choose the exchange by parsing after the "-" in the contract name...

Another good idea I think would be to have the option to plot the ratio between the Commercials/Non-Commercials etc, instead of just the number of contracts.

Thanks again Andreas!

[zweistein]

Hello elliot wave,

I might think of the Commercial/Non Commercial ratio. This is easy to add.
One just has to define an enum

publicenum OpenInterestDisplayType {
Contracts,
Percent,
Commercial_NonCommercial_Ratio
}

as a starting point. A public property of the enum to set from the dialog,
and from within the code check the enum variable and act accordingly.

more exchanges is easy also, you might have seen that I set already the basics for the implementation.

I would just put all the contract from the other exchanges also in the
enum ContractType {
Wheat,
....
...
InterestRateSwaps5,

MilkClassIII <----- first index of next exchange ,so I would do

COTReportHelper.Exchange exchange=COTReportHelper.Exchange.ChicagoBoardofTr ade;

if((int) ct >=(int)ContractType.MilkClassIII) exchange=COTReportHelper.Exchange.ChicagoMercantil eExchange;

if(int) ct >=(int) ContractType.WhatEverContractGoesHere)
exchange=COTReportHelper.Exchange.WhateverExchange BelongsToIt;

...
and so forth

string url=COTHelpers.COTUrl(d, ReportType.Futures,exchange );



Actually there are quite a few unfinished parts in the code, and much fine tuning is needed, also to minimize downloads from cftc.gov

I hope at least that the code supplied is a useable basis for a properly finished indicator.

extending to currency and Index data should be just a matter of extending the hardcoded enums and strings

Andreas

[zweistein]

Hello again,


so attached a modified version in compiled form. No new source code.

Also in compiled form under Limited GNU public license, no commercial use!


Main changes:
- CBOT and CME supported
- downloaded data is cached in the COTReport subdirectory
- http downloads minimized
- many tiny fixes
- Commercial/NonCommercial Ratio added.
- should work from any timezone.


Good luck

Andreas

P.S. removed zip (as it was unfinished), finished code on my web site

[NinjaTrader_Bertrand]

Thanks for sharing Andreas!

[cunparis]

- load time is long as the indicator will download (from within OnBarUpdate) all reports needed from cftc.gov

Just an idea: I download the data in excel and generate "look up" code that I paste into a strategy. That way there is no lookup. I do this in tradestation or I'd just post it. But it's easy to do. I even thought of writing code to parse the csv/excel file but so far just pasting the lookup generated by excel is easier.

[zweistein]

Actually I like doing things properly and the COTReport represents my philosophy quite well. No messing around with Excel sheets and data conversions. Data straight from the source and userfriendly to look at.

Download for the report is long only at first load. Then all the data is cashed in the local COTReport directory.

The second time you use the indicator there is no more delay as the data comes from the local xml file.
Then the code will download each friday at 15.31 EST time the latest report, so this might cause a lock of NT for about 5 seconds. (I might change load time to after market close)

If still too unresponsive one could use a workerthread for loading the data in the background. I do this in one of my indicators, but here I wanted to open a discussion on proper implementation and to concentrate on the COT Report stuff.

Actually all this COTReport idea started 3 days ago because the cleaning woman comes always Mondays from 12.00 to 16.00 MEZ and I was relaxed already after a medium successful trading morning. I usually leave mu office for the time the cleaning woman is there. So I sat with my laptop in a public library and I started looking into COTReport. One night, and another night of concentration, and now I don't have to think about the COTReport anymore.
If I need it, I just use it without hassle. For me I prefer to spend 3 days once and then have an open mind for other stuff instead of beeing tied all the time to excel conversion, parsing, etc. which would then distract me constantly.
Just my point of view and for me this works out

Best regards

Andreas

[cunparis]

I didn't realize that you cached it locally so I think it's a great idea.

I actually like updating the excel because I take time to think about the latest numbers and study the changes.

[zweistein]

@cunparis,


actually looking at the numbers and thinking the data over is a very good idea.

[jthom]

Download for the report is long only at first load. Then all the data is cashed in the local COTReport directory.

How long would that be? My system is hanging forever??
Great work btw.

[zweistein]

download dipends on internet speed etc. ,

so lets, say 10s per report . Multiplied by the weeks you have in your bars.

So for one year is 52 reports is 520 seconds.

It is more problematic if something goes wrong with the download an an exception is thrown.
Then the code will retry a download after the number of bars corresponding to 15 minutes have elapsed. (I changed that value, so version 1.0.2 might be different) .

Let's say you have 5 minute bars and for some reason an exception is thrown. Then every 3 bars an exception (which takes 0,1s to elaborate) for 20000 bars of data... so it makes 6600 exceptions with at worst 0,1 s to elaborate makes 600s = 10 minutes.

All this only only as long data is incomplete and while OnBarUpdate is working on historical data.


Once loading has finished and OnBarUpdate is working on real-time data there is no more delay.

Please check your log and report back

Please check your CORReport directory to see if .xml files have been saved!



Regards

[jthom]

So just to clarify, if trading on ES, would you use a different contract than Emini S&P in cot report to compare too? or use it?

[zweistein]

goodwil payment

hello to everyone!

do you remember my motivation for publishing COTReport?
i was intereted in a discussion on interesting implemenations
of indicators and strategies.
well there is no discussion.
so please think,why should i spend my time to develop an indicator
to your liking?
i can't find a valid motivation!
instead i propose that eveyone interested in cotreport tells me how much
he would pay for having it fully functioning.
if i see sufficient interest i will publish a much improved version
and i would then dipend on your goodwill concerning a paypal payment.
have a look at the source code and you will see that it is a complex
piece of software already and to finish will be 2-3 times the time
aleady spent.
with only a small amount of money you get the fruits of quite an effort
in deveopment.
i 'll get back in a week
andreas