Market Replay fill logic bug

[NinjaTrader_ChelseaB]

Hello,

That was my mistake. The tick replay is part of NinjaTrader 8 and adds intra-bar granularity to your script automatically.

If you are asking about NinjaTrader 7, OnMarketData will not work historically at all (it will not trigger).


So if you are adding a 1 tick series for the ask and a 1 tick series for the bid, the logic would then no longer use OnMarketData at all and this can be removed from the script all together.

Instead you would just use the tick data in OnBarUpdate which will (should) be the same between real-time and historical.


So any data received in real-time for the added tick series should be the same historically in the added tick series.

[bluelou]

Chelsea,
Yes, I'm with you on your last posting, I agree.

Please see my last post again. Now, I'm trying to figure out differences in fill assumptions b/t Market Replay and Strategy Analyzer. I still can't figure out why some marketable limit orders aren't being filled in Strategy Analyzer but are being filled with Market Replay.

I'm not trying to "fix" a problem, I'm just trying to understand exactly what the differences in fills are given the circumstances I described in my last post.

Thx,
Lou

EDIT: Sry, I meant my post at 11:31

[NinjaTrader_ChelseaB]

Hi Lou,

Is the data the same?

Have you added prints to see the ask or bid price in historical and compare this with market replay?

Right where your order is placed, is the order submitted at the same price? (Can we see the trade orders information for this?)

Then right where it fills, what is the bid or ask tick prices at the time of the fill?

Are the times on these matching up exactly?

(I haven't mentioned that if your PC clock is not exactly in sync with internet time then your historical data isn't going to match real-time data and will be shifted when you load the data as historical by however far your PC clock is off)

If you compare the ask and bid tick data on a chart are all the ticks matching up at exactly the same time in historical vs market replay?


The limit order is going to fill using the same simulation engine. The question is did that limit order get the same bid or ask ticks so that the fill is at the same price.

So are you finding the data matches for every tick?
Are you finding there may be a time shift of a few seconds between historical and market replay?
Are you finding that the limit order did not fill at the bid or ask tick data shown on the chart?

[bluelou]

Chelsea,
When Strategy Analyzer has a fill it's mostly on the exact same second and price as market replay. However, Strategy Analyzer has about 1/3 less fills by count than Market Replay, despite the order being marketable.

How do I check to see if the ticks are the same in market replay vs historical? Are you saying run a market replay and then stop/pause it and then open a historical tick chart? Would that be part of the test?

Your last question is the key:
Yes, I'm finding that the limit did not fill on the bid/ask on the chart. This is mostly happening w/strategy analyzer when comparing against market replay.

To be clear, about 1/3 of the time the limit is marketable but only gets filled w/market replay and not with strategy analyzer. I've looked at the bid/ask/last series at 1-tick resolution and I don't see a reason to not fill. The other 2/3 of the trades match; that is, market replay and strategy analyzer agree on time and price of the trade.

~Lou

Hi Lou,

Is the data the same?

Have you added prints to see the ask or bid price in historical and compare this with market replay?

Right where your order is placed, is the order submitted at the same price? (Can we see the trade orders information for this?)

Then right where it fills, what is the bid or ask tick prices at the time of the fill?

Are the times on these matching up exactly?

(I haven't mentioned that if your PC clock is not exactly in sync with internet time then your historical data isn't going to match real-time data and will be shifted when you load the data as historical by however far your PC clock is off)

If you compare the ask and bid tick data on a chart are all the ticks matching up at exactly the same time in historical vs market replay?


The limit order is going to fill using the same simulation engine. The question is did that limit order get the same bid or ask ticks so that the fill is at the same price.

So are you finding the data matches for every tick?
Are you finding there may be a time shift of a few seconds between historical and market replay?
Are you finding that the limit order did not fill at the bid or ask tick data shown on the chart?

[NinjaTrader_ChelseaB]

Hi Lou,

You could either print the tick data to output window, write the data to a file, or yes just open a chart and look visually.

This would mean open 2 charts, one for the bid one for the ask. Open these before running the market replay. The stop the playback right after one of the trades that is different.

Screenshot this with the timestamps and prices showing and then compare the chart with historical data.

Are they the same tick for tick?


I would start with a specific trade to investigate. Take a look a look at the price the order is submitted at. Then look at the bid or tick data. Is the price hit for that order to fill?

What is the fill type you are using?

[bluelou]

Chelsea,
I'll look at some more trades. I haven't tried looking at a market replay chart vs a historical chart so I can try that.

For Strategy Analyzer fill types I use a custom fill type that requires top of book tick data (bid/ask/last). It's specifically written to fill on a trade-through. If I post a long limit order at the bid it is not considered filled until the market goes asked at my order price; i.e, the definition of a marketable limit order.

I'll get back to you tomorrow and let you know what I find. Thank you for your help!

~Lou

Hi Lou,

You could either print the tick data to output window, write the data to a file, or yes just open a chart and look visually.

This would mean open 2 charts, one for the bid one for the ask. Open these before running the market replay. The stop the playback right after one of the trades that is different.

Screenshot this with the timestamps and prices showing and then compare the chart with historical data.

Are they the same tick for tick?


I would start with a specific trade to investigate. Take a look a look at the price the order is submitted at. Then look at the bid or tick data. Is the price hit for that order to fill?

What is the fill type you are using?

[NinjaTrader_ChelseaB]

Hi Lou,

That may be it. In realtime, trades will fill when touched (as long as there is enough volume to fill it).

[bluelou]

Yes, I agree that's possible and I looked into that. However, in each case where I've checked the order was marketable, too, so it should have filled with Strategy Analyzer.

I can understand why market replay IS filling trades. I can't understand why strategy analyzer IS NOT filling the same trades when the orders are marketable.

Perhaps it's a timing issue as you've mentioned. Hopefully, we'll figure it out tomorrow. I'm okay with the difference if it's some NT peculiarity, I'm not okay with it if it's caused by my code .

[NinjaTrader_ChelseaB]

Hi bluelou,

I'm a little confused, in real time orders can be touched and filled.
In backtest they must be passed through.

The fill algorithm isn't the same, why would the trades be the same?

[bluelou]

Chelsea,
Are you saying market replay can fill 'if touched'? If so, then that's a pretty aggressive assumption.

Here's an example why:
1) Given market is 1.3500 (30) bid x 1.3501 (20) ask // inside the parentheses is contract volume quoted.
2) My long limit order for 1 contract is resting at the bid, 1.3500.
3) Let's say the bid and ask are unchanged but someone crosses the market and sells 1 contract at 1.3500
4) How do I know that it was MY contract that was FILLED? The only way that I could assume a fill is if I assume my 1 contract is at the top of the queue. That is, my 1 contract is at the top of a 30 contract queue.

Do you see the problem with this assumption in a sim engine? That's why NT's documentation warns about using the 'Liberal' fill method.

Note: The solution to this issue is to assume that only marketable orders are filled. I.e., it the market goes asked at 1.3500 that means the 30 contracts to buy resting at 1.3500 (or, 1 'level') were taken out.

Hi bluelou,

I'm a little confused, in real time orders can be touched and filled.
In backtest they must be passed through.

The fill algorithm isn't the same, why would the trades be the same?

[NinjaTrader_ChelseaB]

Hello bluelou,

The historical fill engine and real-time simulation engine are not the same.

The Historical Fill Processing only applies to historical orders.

Below is a link to the help guide on Discrepancies: Real-Time vs Backtest where this information is explicitly stated in the section 'The Fill Price of Orders'.

"During simulation using real-time market data or Market Replay, the fill price is based on incoming market data and volume, you may receive better or worse fill prices dependant on where the bid or ask price is and what volume is available at this market prices.
During real-time brokerage trading, orders are filled according to market dynamics."

http://ninjatrader.com/support/helpG...ime_vs_bac.htm

What is your proposed change to this documentation to make this clearer?
I will send your changes as a feature request to the product management team.


When you are running live you are not backtesting and the backtesting Historical Fill Processing found in the link below does not apply. (See the section 'Understanding historical processing options')
http://ninjatrader.com/support/helpG...a_strategy.htm

Market Replay is seen as real-time live data in NinjaTrader.

[bluelou]

Chelsea,
Okay, that solves a big part of the mystery - there are different sim/fill engines.

Regarding the documentation, I wouldn't have known to equate Real-Time with Market Replay given the title of the link you sent. So, that could be clearer for sure. But, it's great that the doc exists and thank you for pointing this out

2 items still unaddressed:
1) For the same strategy, given a backtest with 1-tick bars (ask/bid/last) and COBC = true AND given a market replay sim with GetCurrentBid/Ask, Last series - what would cause marketable limits in the backtest to not be filled. In other words, I'm getting some fills in market replay but not in strategy analyzer AND I've verified the data. There should be a fill.

Of course, what I can't do with Strategy Analyzer is use OnMarketData() to see the exact order of the ticks. Maybe it will remain a mystery?

Personally, I guess I'll live with the answer "different engines" - but that still doesn't get to the "why" part.

2) It's unclear from the doc if market replay will fill "if touched" (i.e., spread is crossed) and under what circumstances. While I agree with you that it's possible for this to happen it requires complex modeling to include in a simulation. Also, I'd want the option to shut this part off b/c it's probably wrong.

Hello bluelou,

The historical fill engine and real-time simulation engine are not the same.

The Historical Fill Processing only applies to historical orders.

Below is a link to the help guide on Discrepancies: Real-Time vs Backtest where this information is explicitly stated in the section 'The Fill Price of Orders'.

"During simulation using real-time market data or Market Replay, the fill price is based on incoming market data and volume, you may receive better or worse fill prices dependant on where the bid or ask price is and what volume is available at this market prices.
During real-time brokerage trading, orders are filled according to market dynamics."

http://ninjatrader.com/support/helpG...ime_vs_bac.htm

What is your proposed change to this documentation to make this clearer?
I will send your changes as a feature request to the product management team.


When you are running live you are not backtesting and the backtesting Historical Fill Processing found in the link below does not apply. (See the section 'Understanding historical processing options')
http://ninjatrader.com/support/helpG...a_strategy.htm

Market Replay is seen as real-time live data in NinjaTrader.

[bluelou]

Also, NT's Default fill option is wrong, especially when using top of book tick data. That's why I took NTs fill code and corrected it to come up with my own fill logic.

[NinjaTrader_ChelseaB]

Hi bluelou,

You can see the order of the order of the ticks in the 1 tick ask and bid series you have told me that you have added to your script.

This is what you have been comparing correct?


If the price is there for it to fill and you are requiring that it pass through in historical data while it only having to be touched in real-time I would expect that there would be less historical trades because really they need 1 more tick than real-time does. Since the requirements are different the fills are different. Why are you expecting them to be the same?

When you say the Default fill type is wrong, why is this wrong and how do you propose to change it? (Let me know so that I can provide that information as a feature request to our development department)


I'm not clear what you mean by shut off the fill if touched in Market Replay. Orders fill based on volume and price. If you remove this orders won't fill at all. What are you proposing we change?

[bluelou]

Chelsea,
Well, it wasn't totally clear to me until your last post that market replay definitely uses 'If touched" (i.e., 'liberal' fills) so you are right. Replay and backtest shouldn't be the same in this case and I probably shouldn't be using market replay if this is the case. FWIW, just like NT warns about using the 'Liberal' fill algo.

Regarding the 'Default' fill:
I could be wrong, of course, but my experience with the Default fill was that it was equivalent to a form of adverse selection and not realistic at all. Here's why...

1) Default appears to require 'trade-through' rather than 'quote-through'
For instance, if I have a resting limit at the bid and the market is 1.3500 bid x 1.3501 ask then NT's Default type will only fill if there is a TRADE at 1.3499.

2) What should happen?
If the market goes ask 1.3500 that means the entire level resting at the bid was taken out (or cancelled) and I was filled at 1.3500.

3) So, what's the big deal?
If you're trading intraday how the fills are modeled are a major part of the viability of the strategy. If you're trading on longer time frames you may not notice much of a difference.

4) What did I do to NT's fill code?
I just modified it to accept marketable limits as fills just like it would happen in a real market.

Does this make sense?

~Lou

Hi bluelou,

You can see the order of the order of the ticks in the 1 tick ask and bid series you have told me that you have added to your script.

This is what you have been comparing correct?


If the price is there for it to fill and you are requiring that it pass through in historical data while it only having to be touched in real-time I would expect that there would be less historical trades because really they need 1 more tick than real-time does. Since the requirements are different the fills are different. Why are you expecting them to be the same?

When you say the Default fill type is wrong, why is this wrong and how do you propose to change it? (Let me know so that I can provide that information as a feature request to our development department)


I'm not clear what you mean by shut off the fill if touched in Market Replay. Orders fill based on volume and price. If you remove this orders won't fill at all. What are you proposing we change?