Why Doesn't Open[0] = Close[1]?

[aventeren]

Howdy--

Super basic question:

On a daily chart, why would Open[0] != Close[1] (i.e., the candle opens and closes are not matching up)?

Thanks,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

The close of a session can sometimes be different because of overnight trading.

If you are using a daily bar and Kinetick's data (or another data provider that uses RTH hours for daily bars), then the daily bar will be in RTH hours (Regular Trading Hours). These will not include the data from the ETH hours during the time the RTH session is closed. This can cause large gaps between the close of previous session and the open of the current.

If you open a 1440 minute chart using the session template <use instrument settings> are you still seeing a difference between the open and prior close?

What instrument are you viewing?

[aventeren]

I'm looking at ES and my data provider is Continuum.

I've tried a daily ES bar with the Default 24/7 session template and then the custom session template that I created to mirror the CME trading hours for the ES instrument (per the CME website).

I've tried with both Continuum and Kinetick Free-End-Of-Day data streams.

When switching between data providers and session templates, I've "Reloaded All Historical Data".

Is there a way to have daily bars match up (i.e., open[0] = close[1], which means that all ETH trades are displayed)?

Thanks,

Aventeren

[Harry]

For a daily bar you will never expect that the open of the new session equals the close of the prior session. As you mentioned ES let us go through the details.

CME

The official data published by CME for the exchange refers to

daily open = open at 5:00 PM CT
daily close = volume-weighted average price of all trades that printed between 3:14:30 and 3:15:00 PM CT

Why do you think that an average price from 3:15 PM should equal the open price at 5:00 PM ?

DTN/IQ or Kinetick

Daily prices are identical with those published by the exchange ( = quality data feed).

Rithmic or CQG

These data providers use the regular open at 8:30 AM CT and the regular close at 3:15 PM CT for the daily data. Of course you may not expect that the daily open at 8:30 AM has anything to do with the close of the regular session of the prior day at 3:15 PM.

Interactive Brokers

Some brokers simply build there daily bars from intraday data (= cheap data feed). In that case you get the daily open at 5:00 PM and the daily close at 4:15 PM next day (prior to the technical break). Of course ES rarely reopens at 5:00 PM at the last traded price from 4:15 PM.

Intraday data

Even for intraday data your reasing that the open of the next bar should conincide with the close of the prior bar is false. The first trade of a new bar may print at the ask and thus at a higher price if the last trade of the prior bar occured at the bid. The frist trade of a new bar may print at the bid and thus at a lower price if the last trade of the prior bar occured at the ask. The bid/ask spread can be one or more ticks. Also it it possible that the bid/ask spread moves between two trades, when some of the orders sitting in the order book are removed without being executed.

The whole idea that a new bar should open at the same price as the prior bar would only be true if prices cannot move between two ticks. But why would you want to trade, if the market cannot move?

[NinjaTrader_ChelseaB]

Hello aventeren,

Harry is correct here. With some instruments (including the ES) there is a settlement price that is a fair price calculated by the exchange.

You can find out how this is calculated from information on the CMEGroup website linked below (settle procedure).
http://www.cmegroup.com/trading/equi...fications.html

(Click Quotes to see a chart, click Settlements to view the settlement prices)

[aventeren]

DataGapChecker001

Howdy--

So I built a little indicator (attached) that prints month and year changes on a chart, plus notes if the month and year is off (ie, June, August--no July, error; 2006, 2008--no 2007, error). The errors are printed on the chart, plus to the Output Window.

I also looked at the absolute difference between Open[0] and Close[1] to see how far off my data is (in ticks). For instance, if I define an 8 tick error tolerance between the Open[0] and Close[1] (ie, due to the Settlement Procedure that Harry mentions), and the difference is instead -22 ticks, I may want to investigate why (ie, is this gap acceptable or is something off that I need to try and fix?).

So when I open up a daily ES 06-14 chart with 10,000 days and with "MergeBackAdjusted" as the Merge Policy with a session template that includes both RTH and ETH, my DataGapChecker001 indicator reports that there were 198 instances where the absolute difference between Open[0] and Close[1] was greater than 8 ticks. Here are the results from the Output Window.

1: Price Gap Problem on Bar Number 64 on 3/10/2005 6:30:00 AM; Difference between Open[0] and Close[1] is 21 ticks
2: Price Gap Problem on Bar Number 128 on 6/9/2005 6:30:00 AM; Difference between Open[0] and Close[1] is 21 ticks
3: Price Gap Problem on Bar Number 193 on 9/8/2005 6:30:00 AM; Difference between Open[0] and Close[1] is 25 ticks
...
196: Price Gap Problem on Bar Number 2394 on 5/26/2014 6:30:00 AM; Difference between Open[0] and Close[1] is 21 ticks
197: Price Gap Problem on Bar Number 2397 on 5/29/2014 6:30:00 AM; Difference between Open[0] and Close[1] is 11 ticks
198: Price Gap Problem on Bar Number 2399 on 6/2/2014 6:30:00 AM; Difference between Open[0] and Close[1] is 13 ticks

So I'm trying to figure out if the data that I have downloaded from BMT is totally off and I need to re-build my dataset or if these types of differences between Open[0] and Close[1] are normal. My gut says that they are not normal, but I'd like a second set of eyes on this.

My ultimate goal is to make sure that I have quality data upon which my historical backtesting, optimization and walk forward analysis (per Pardo) are compiled on (ie, Garbage in-Garbage out).

Thanks for your help,

Aventeren

[aventeren]

For a daily bar you will never expect that the open of the new session equals the close of the prior session. As you mentioned ES let us go through the details.

CME

The official data published by CME for the exchange refers to

daily open = open at 5:00 PM CT
daily close = volume-weighted average price of all trades that printed between 3:14:30 and 3:15:00 PM CT

Why do you think that an average price from 3:15 PM should equal the open price at 5:00 PM ?

DTN/IQ or Kinetick

Daily prices are identical with those published by the exchange ( = quality data feed).

Rithmic or CQG

These data providers use the regular open at 8:30 AM CT and the regular close at 3:15 PM CT for the daily data. Of course you may not expect that the daily open at 8:30 AM has anything to do with the close of the regular session of the prior day at 3:15 PM.

Interactive Brokers

Some brokers simply build there daily bars from intraday data (= cheap data feed). In that case you get the daily open at 5:00 PM and the daily close at 4:15 PM next day (prior to the technical break). Of course ES rarely reopens at 5:00 PM at the last traded price from 4:15 PM.

Intraday data

Even for intraday data your reasing that the open of the next bar should conincide with the close of the prior bar is false. The first trade of a new bar may print at the ask and thus at a higher price if the last trade of the prior bar occured at the bid. The frist trade of a new bar may print at the bid and thus at a lower price if the last trade of the prior bar occured at the ask. The bid/ask spread can be one or more ticks. Also it it possible that the bid/ask spread moves between two trades, when some of the orders sitting in the order book are removed without being executed.

The whole idea that a new bar should open at the same price as the prior bar would only be true if prices cannot move between two ticks. But why would you want to trade, if the market cannot move?

Harry, I've been thinking more about your response (which was thorough, as usual, and for this I am grateful), and given the differences in Settlement Prices, what would you expect to be an acceptable range between Open[0] and Close[1] in ticks? For instance, what would you expect to be acceptable differences on the ES, CL, 6E and 6J?

I built a little indicator to point out gaps that are outside an acceptable tick range (it's in this thread), and perhaps I could run your "acceptable differences" and see what gaps remain.

My goal here is to have a solid data set that I can backtest. At this point, given that I have cobbled together my data from BMT, I'm not quite confident enough that significant data problems exist to obfuscate my backtest results.

Thanks for your help and input--it's always appreciated.

All best,

Aventeren

[Harry]

Harry, I've been thinking more about your response (which was thorough, as usual, and for this I am grateful), and given the differences in Settlement Prices, what would you expect to be an acceptable range between Open[0] and Close[1] in ticks? For instance, what would you expect to be acceptable differences on the ES, CL, 6E and 6J?

Any difference is acceptable. The electronic exchange is closed for about an hour
(technical break from 5:00 PM to 6:00 PM EST for 6E and 6J, and 5:15 PM to 6:00 PM EST for ES and CL) during the week, and it is closed for two entire days during the week-end.

Whenever the exchange is closed, the news that hit the market during the break may move price in such a way that there is a significant gap.

Also you need to take into account that other markets are not closed. For example, the currency pair EURUSD is traded during the technical break for CME. If EURUSD gains 20 pips during the period from 5:00 PM to 6:00 PM EST, this means that 6E will typically gap up about 20 pips when the market opens again.

I do not know where you got that funny idea that a market should reopen at the same price level at which it closed. Just accept that you will always see gaps when a market closes and reopens later. Gaps are ok. A price gap during the technical break is no error.