OnMarketDepth and e.Position

[alex.nt]

Hi,

I am using replay data for ES and it appears that in OnMarketDepth I receive an two events for Sell side of the book:

Remove at position 0
Insert at position 10

This means that order book does not maintain the integrity, i.e. e.Position defines an order only, but not the actual position.

Please confirm.

[koganam]

Hi,

I am using replay data for ES and it appears that in OnMarketDepth I receive an two events for Sell side of the book:

Remove at position 0
Insert at position 10

This means that order book does not maintain the integrity, i.e. e.Position defines an order only, but not the actual position.

Please confirm.

It is implemented as a Queue. That is the correct description of a queue: out at the front; in at the back. FIFO.

[alex.nt]

Sorry, but this one does not count

The thing is that when an item is removed then the size becomes 9, which means it would not be possible to insert at position 10th because there is no value at position 9 after the removal (0....8 = 9 items)

[koganam]

Sorry, but this one does not count

The thing is that when an item is removed then the size becomes 9, which means it would not be possible to insert at position 10th because there is no value at position 9 after the removal (0....8 = 9 items)

Actually if there are 10 items in the queue, then they span 0 to 9. A queue is sized dynamically, just as a real-life queue (such as a checkout line) is dynamic. One can always add at the end of the queue, and the number would be one bigger than the current count.

So, an item is removed at 0, leaving 9 items, which span 0 to 8. The next will be inserted at index position 9, which is the tenth position. Are you sure you saw an addition at index 10? How did you see that? Maybe a little code, so that I can verify it?

[alex.nt]

Yes, I would also expect it to be at position 9 but it comes at 10. This does not make any sense. It is too dramatic to be true

The code is really simple, pretty much this:

case Operation.Insert:
Print(string.Format("Inserting {0} at {1}", bidAsk.ToString(), e.Position));
break;

In fact I used a list to insert, update and remove items there - and got the crash exactly due to the mentioned issue. It is ES 12-12 of 2012-Nov-02.

I will try to give the complete script tomorrow.

[alex.nt]

Bumping.

Anyone could shed a light?

Here is the log from my application:
12:12:52.984 AM: Starting processing threads
12:12:53.068 AM: InitializeMarketDepth ... Success.
12:12:53.199 AM: ProcessMarketDepthInsertEvent: Ask book size = 0, item: 0[Ask] @ 1422 V:33, T:10:00 AM
12:12:53.201 AM: Adding new row at 1422.00...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 1, item: 1[Ask] @ 1422.25 V:81, T:10:00 AM
12:12:53.211 AM: Adding new row at 1422.25...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 2, item: 2[Ask] @ 1422.5 V:78, T:10:00 AM
12:12:53.211 AM: Adding new row at 1422.50...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 3, item: 3[Ask] @ 1422.75 V:101, T:10:00 AM
12:12:53.211 AM: Adding new row at 1422.75...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 4, item: 4[Ask] @ 1423 V:97, T:10:00 AM
12:12:53.211 AM: Adding new row at 1423.00...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 5, item: 5[Ask] @ 1423.25 V:147, T:10:00 AM
12:12:53.211 AM: Adding new row at 1423.25...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 6, item: 6[Ask] @ 1423.5 V:145, T:10:00 AM
12:12:53.211 AM: Adding new row at 1423.50...
12:12:53.211 AM: ProcessMarketDepthInsertEvent: Ask book size = 7, item: 7[Ask] @ 1423.75 V:148, T:10:00 AM
12:12:53.212 AM: Adding new row at 1423.75...
12:12:53.212 AM: ProcessMarketDepthInsertEvent: Ask book size = 8, item: 8[Ask] @ 1424 V:127, T:10:00 AM
12:12:53.212 AM: Adding new row at 1424.00...
12:12:53.212 AM: ProcessMarketDepthInsertEvent: Ask book size = 9, item: 9[Ask] @ 1424.25 V:306, T:10:00 AM
12:12:53.212 AM: Adding new row at 1424.25...
12:12:53.212 AM: ProcessMarketDepthInsertEvent: Bid book size = 0, item: 0[Bid] @ 1421.75 V:160, T:10:00 AM
12:12:53.212 AM: Adding new row at 1421.75...
12:12:53.212 AM: ProcessMarketDepthInsertEvent: Bid book size = 1, item: 1[Bid] @ 1421.5 V:127, T:10:00 AM
12:12:53.212 AM: Adding new row at 1421.50...
12:12:53.212 AM: ProcessMarketDepthInsertEvent: Bid book size = 2, item: 2[Bid] @ 1421.25 V:201, T:10:00 AM
12:12:53.212 AM: Adding new row at 1421.25...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 3, item: 3[Bid] @ 1421 V:424, T:10:00 AM
12:12:53.215 AM: Adding new row at 1421.00...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 4, item: 4[Bid] @ 1420.75 V:300, T:10:00 AM
12:12:53.215 AM: Adding new row at 1420.75...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 5, item: 5[Bid] @ 1420.5 V:303, T:10:00 AM
12:12:53.215 AM: Adding new row at 1420.50...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 6, item: 6[Bid] @ 1420.25 V:352, T:10:00 AM
12:12:53.215 AM: Adding new row at 1420.25...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 7, item: 7[Bid] @ 1420 V:692, T:10:00 AM
12:12:53.215 AM: Adding new row at 1420.00...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 8, item: 8[Bid] @ 1419.75 V:259, T:10:00 AM
12:12:53.215 AM: Adding new row at 1419.75...
12:12:53.215 AM: ProcessMarketDepthInsertEvent: Bid book size = 9, item: 9[Bid] @ 1419.5 V:277, T:10:00 AM
12:12:53.215 AM: Adding new row at 1419.50...
12:13:10.706 AM: ProcessMarketDepthRemoveEvent: Ask book size = 9, removed 1422 at 0
12:13:10.707 AM: ProcessMarketDepthInsertEvent: Bid book size = 10, item: 0[Bid] @ 1422 V:24, T:10:01 AM
12:13:10.707 AM: ProcessMarketDepthInsertEvent: Ask book size = 9, item: 10[Ask] @ 1424.5 V:518, T:10:01 AM
12:13:10.707 AM: ProcessMarketDepthInsertEvent: Attempt to insert an element that is not adjacent
12:13:10.707 AM: ProcessMarketDepthRemoveEvent: Bid book size = 10, removed 1419.5 at 10
12:13:18.258 AM: ProcessMarketDepthRemoveEvent: Ask book size = 9, removed 1422.25 at 0
12:13:18.258 AM: ProcessMarketDepthInsertEvent: Bid book size = 10, item: 0[Bid] @ 1422.25 V:31, T:10:02 AM
12:13:18.258 AM: ProcessMarketDepthInsertEvent: Ask book size = 9, item: 10[Ask] @ 1424.75 V:741, T:10:02 AM
12:13:18.258 AM: ProcessMarketDepthInsertEvent: Attempt to insert an element that is not adjacent
12:13:18.258 AM: ProcessMarketDepthRemoveEvent: Bid book size = 10, removed 1419.75 at 10
12:15:08.230 AM: Stopping processing threads

The interesting part is this:
12:13:18.258 AM: ProcessMarketDepthRemoveEvent: Ask book size = 9, removed 1422.25 at 0

Now we have 0..8 (9 in total) for Asks

Next item in log shows that we receive Operation.Insert with Position=10
12:13:18.258 AM: ProcessMarketDepthInsertEvent: Ask book size = 9, item: 10[Ask] @ 1424.75 V:741, T:10:02 AM

However, our Ask book's size is 9! "item: 10[Ask] @ 1424.75" is printed using MarketDepthEventArgs.

12:13:18.258 AM: ProcessMarketDepthInsertEvent: Attempt to insert an element that is not adjacent

Any idea?

May it relate to specific broker?

[NinjaTrader_Lance]

Hello,

Could you please post the script or simplified version of that replicates the error you are seeing so that we may test on our end?

[alex.nt]

It is not a script. The script is simple, it just logs the OnMarketDepth.
It is data.

Here is the script:
private int myInput0 = 1; // Default setting for MyInput0
private enum BidAsk
{
Bid,
Ask
}


private Dictionary<BidAsk, List<double>> m_book = new Dictionary<BidAsk, List<double>>();

/// <summary>
/// This method is used to configure the indicator and is called once before any bar data is loaded.
/// </summary>
protected override void Initialize()
{
Add(new Plot(Color.FromKnownColor(KnownColor.Orange), PlotStyle.Line, "Plot0"));
Overlay = false;

m_book.Add(BidAsk.Bid, new List<double>());
m_book.Add(BidAsk.Ask, new List<double>());
}

/// <summary>
/// Called on each bar update event (incoming tick)
/// </summary>
protected override void OnBarUpdate()
{
// Use this method for calculating your indicator values. Assign a value to each
// plot below by replacing 'Close[0]' with your own formula.
Plot0.Set(Close[0]);
}

protected override void OnMarketDepth(MarketDepthEventArgs e)
{
BidAsk ba = e.MarketDataType == MarketDataType.Ask ? BidAsk.Ask : BidAsk.Bid;

switch (e.Operation)
{
case Operation.Insert:
Print(string.Format("Book[{0}] size: {1}, inserting at position {2}"
, ba.ToString(), m_book[ba].Count, e.Position));
m_book[ba].Insert(e.Position, e.Price);
break;
case Operation.Remove:
Print(string.Format("Book[{0}] size: {1}, removing at position {2}"
, ba.ToString(), m_book[ba].Count, e.Position));
m_book[ba].RemoveAt(e.Position);
break;
case Operation.Update:
m_book[ba][e.Position] = e.Price;
break;
}
}


Here is the output:

Book[Ask] size: 0, inserting at position 0
Book[Ask] size: 1, inserting at position 1
Book[Ask] size: 2, inserting at position 2
Book[Ask] size: 3, inserting at position 3
Book[Ask] size: 4, inserting at position 4
Book[Ask] size: 5, inserting at position 5
Book[Ask] size: 6, inserting at position 6
Book[Ask] size: 7, inserting at position 7
Book[Ask] size: 8, inserting at position 8
Book[Ask] size: 9, inserting at position 9
Book[Bid] size: 0, inserting at position 0
Book[Bid] size: 1, inserting at position 1
Book[Bid] size: 2, inserting at position 2
Book[Bid] size: 3, inserting at position 3
Book[Bid] size: 4, inserting at position 4
Book[Bid] size: 5, inserting at position 5
Book[Bid] size: 6, inserting at position 6
Book[Bid] size: 7, inserting at position 7
Book[Bid] size: 8, inserting at position 8
Book[Bid] size: 9, inserting at position 9
Book[Ask] size: 10, removing at position 0
Book[Bid] size: 10, inserting at position 0
Book[Ask] size: 9, inserting at position 10
Error on calling 'OnMarketDepth' method for indicator 'TestMe': You are accessing an index with a value that is invalid since its out of range. I.E. accessing a series [barsAgo] with a value of 5 when there are only 4 bars on the chart.

[BigWaveDave]

"BidAsk ba = e.MarketDataType == MarketDataType.Ask ? BidAsk.Ask : BidAsk.Bid;"

whoa... what's this?

You've got your own enum for BidAsk... then you make an assumption about the values of MarketDataType.Ask and MarketDataType.Bid.

I think this may be your problem.

This should probably read...

BidAsk ba = e.MarketDataType.Ask ? BidAsk.Ask : BidAsk.Bid;

Add... On second thought... that's not right either... it looks ok....

Your code looks ok... I dunno.

[BigWaveDave]

Here's a suggestion though.

Why not simply rework the Insert case to call List.Add()?

Add will always add to the end of the list. You shouldn't need the Position value for inserts anyway, only for updates...

The docs don't make if very clear what the behavior actually is...

Now, if updates have incorrect Position values then you've got another issue...

[alex.nt]

Unfortunately, we may not use List.Add as book items can be added/removed from both ends.

Moreover, it is not stated anywhere that book items may not be removed/inserted from/in arbitrary positions.

I could use this in my case but I am afraid it is not a solution as I had clearly had book integrity failure.

[alex.nt]

After further investigations it appears that:

a) Book size (e.MarketDepth.Bid/Ask.Count) fluctuates from 9 to 11
b) When book size is 9 then the next insert always occurs at position 10, which is wrong. Internally it is appended to the book so everything remains alive.

So I guess the best way to handle this issue is the same way NinjaTrader handles it: just append the item to the end of the book if the index of insertion is out of range.