Incorrect Fill Price even when using tickreplay and having bid/ask.

[rspine]

Hi

I have uploaded correctly Last, Bid, Ask ticks, with 10th of microsecond precision as per instructions for one instrument (e.g. FGBL 06-12 ).

Then from the Strategy analyzer I run the "Sample MA crossover" strategy with tick replay on a 1 tick. See figure below.
As you know this strategy uses the Enterlong() Entershort() methods that are market orders. However, I see that the fill price are not using the correct bid and ask they are (incorrectly trading at whatever is the last tick price).
In the example it sells market order at 140.90 around timestamp is 17:01:34 seconds
Indeed this is the ask price, Since the bid is never higher than 140.89 betweeen 17:01 and 17:02

The ability to do tickreplay and have max granularity is good (although one would like to be able to choose the latency) but that is only a minor improvement. The key point is to be able to trust that bid ask are used correctly and in controllable/transparent way in historical back tests. I understood this was now supported in NT8. I read in the help on the OnMarketData() event that NT8 "stores the best bid ask at each tick" ..but from my tests it appears this is a bit obscure or wrong.


ALSO RELATED
From other test during the OMarketData() event the marketDataUpdate.MarketDataType it is NOT updated it seems always set at "marketDataUpdate.MarketDataType==MarketDataType.L ast".


In general : Indeed the important point is that I would like to have total control on the Fillprice assumption (as it was somehow possible with NT7). Simply Allowing the user to customize it, e.g. exposing some data (e.g. the next high low of a timeseries... or anything like that). Or even just prices (last, bid ask) in the past ..(one can put a lag in the input signals anyway). I believe that the inability to control exactly the assumption on fill price makes any backtest, even for lower freq strategy, worthless.


Kind Regards
R.

[NinjaTrader_Brett]

rSpine,

Thanks for posting appreciate you taking the time to post your experience.

On the first point the fill engine is not setup to use bid or ask prices at all. We did not design the fill engine with tick replay in mind unfortunately on this aspect. Its good feedback and is something we should look at in the future if we wanted to support tick replay back testing. SFT-982

On the second point labeled "ALSO RELATED" I'm sorry but I do not understand what you mean. If you get an OnMarketData call with MarketDataType == MarketDataType.Last I expect that to be a trade occurring and to have its .Bid/ and .Ask properties set correctly to reflect the bid/ask at the time the trade occurred. What issue are you running into?

On the third point on no longer being able to adjust the fill type anymore, I will record this feedback. We originally removed the fill type due to lack of interest and we ultimately developed a fill type we felt would meet the majority of use cases needed. SFT-983

[rspine]

The FillPrice needs to be transparent and customizable

Hi Brett

Thank you for taking the time to read and answer to my post, however I must kindly but firmly say that at the moment, the answer describes a situation that is highly unsatisfactory.
This was a massive issue of Ninjatrader7, there is the chance now to reduce/solve the issue in NT8, I hope this will not get actually worse in quality or transparency.

All these points are related to the same key issue/bug that is how to control and trust the (hypothetical) fill price.
Just to make it clear that this is a deal breaker and top priority for any backtester consider the following.
If you do 10 trades a day (buy+sells+20) on something like oil (CL) the bid/ask is equivalent to 0.01, that makes 0.01 x 20 x 1000 USD = 200 USD per day, that is 50'000 USD per year per contract.
It is 100% annualized return on notional value of the current Oil future. Or 500% just using a 5to1 leverage. This is not a detail. On other less volatile assets it would be even worse.
I have worked as market maker, tested many different platforms (Deltix,TT, CQG, Stellar, Multichart and few other custom built backtesters).
If you can't control the fill assumptions (or worse if they are wrong) any back test result is irrelevant or at best you can interpret it as just way to "optimize how to cheat the fill assumption".
This is in your interest as a software developer as well as a broker, your client would go into trading suicide if they assume a back test with more than 10 trades a day has any correspondence with what happened in the past data (let alone the future).


Indeed in your list of NT8 enhancements

http://ninjatrader.com/Downloads/MajorEnhancements_NinjaTrader8.pdf

There are clearly two points where I believed this somehow was improved:
"Enhanced Backtesting Engine"
"New TickReplay Engine"
"Expanded NinjaScript® Access"

Moreover in the NT8 help:

http://ninjatrader.com/support/helpGuides/nt8/en-us/

"Developing for Tick Replay"
"OnMarketData()"
You are describing the OnMarketData() event as a way to access and back test with bid/ask and tick replay.

This was a move in the right direction however all would be wasted if you do not use the bid ask in the fills when available or as a minimum you have to tell how the FillPrice is calculated and open it to customization.


1) Can you point me or explain exactly how the fill price is calculated. With formulas and snippets etc.
2) Please escalate this to the highest level.
3) Check the next post where I explain better what I believe it is a bug/unexpected behavior on the marketDataUpdate.MarketDataType not being updated correctly.

Kind Regards
R.

[rspine]

In the OnMarkedData() event the MarketDataType field is always set to "Last"

In the OnMarkedData() event the MarketDataType field is always set to "Last"
when it should be "Ask" or "Bid"

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
if (marketDataUpdate.MarketDataType != MarketDataType.Last)
{
// is never reached
throw Exception("Bid or ask updated");
}
}

subscriptions code:

AddDataSeries("AAPL", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Last);
AddDataSeries("AAPL", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Ask);
AddDataSeries("AAPL", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Bid);

[NinjaTrader_Brett]

1) http://ninjatrader.com/support/helpG...ical_fill_.htm

This page here describes the fill processing. If you have further questions let me know and I will answer them and potentially update the page to have more detail as needed. I would not be able to provide code snippets or formulas however.

2) You are speaking to the Product Manager. I will note your feedback and discuss it internally. Early on in the design phase of NT8 we decided to not expose fill algorithms as NinjaScript since in NT7 we found low to no usage of the exposed fill algorithms and we felt we could innovate there with a proprietary fill engine which satisfied the use cases and just 'worked'. Even with NT7 we did not use bid/ask data for the fill of orders. This is an enhancement request which comes with the penalty of requiring having the bid/ask data (Most people don't have and even if you did have there is due diligence to do to know is the bid/ask data filtered) and comes at the cost of triple the cost in time/memory/etc since there is more data to process. With the cost and benefit analysis and the scope of clients who actually have access to the data we have decided not to implement this at this time.

3) Thanks for explaining, this is actually expected. Since Tick Replay is only designed to work with last. Instead you always monitor 'Last' events and utilize the "MarketDataUpdate.Ask" and "MarketDataUpdate.Bid" properties to get the Ask/Bid price when the last trade occurred. We store the ask/bid with the actual last data as it occurs so we only have to process a single series.

If your saying that we should also throw OnMarketData events and support tick replay for bid/ask dataseries then I would record this feedback so we can look into in in a future major release.

[ggggg]

Hi Brett,

Unfortunately, we cannot make OnMarkedData() work the way you have described. A minimal working example of strategy working with historical bid/ask would be extremely helpful.

We created the following artificial data

Bid:
20151211 060101;99;1
20151211 060104;100;1
20151211 060106;101;1
Ask:
20151211 060101;101;1
20151211 060104;102;1
20151211 060107;103;1
Trades (Last):
20151211 060102;100;1
20151211 060103;100;1
20151211 060105;101;1

and created a strategy with the following OnMarketData() handler.

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
	Print(string.Format("{0} Bid={1} Ask={2} Price={3} Type={4}",
	  marketDataUpdate.Time,  marketDataUpdate.Bid, marketDataUpdate.Ask,
	  marketDataUpdate.Price, marketDataUpdate.MarketDataType)
	);
}

If we do not subscribe to any additional streams besides the primary one specified in the GUI the output is

11.12.2015 07:01:02 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:03 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:05 Bid=101 Ask=101.01 Price=101 Type=Last

If, in addition, we subscribe to the bid ask streams in OnStateChange()

protected override void OnStateChange()
{
	#...
	else if (State == State.Configure)
	{
		AddDataSeries("GOOG", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Ask);
		AddDataSeries("GOOG", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Bid);
	}
}

the output becomes

11.12.2015 07:01:01 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:01 Bid=99 Ask=99.01 Price=99 Type=Last
11.12.2015 07:01:02 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:03 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:04 Bid=102 Ask=102.01 Price=102 Type=Last
11.12.2015 07:01:04 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:05 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:06 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:07 Bid=103 Ask=103.01 Price=103 Type=Last

In either case, we cannot access bid/ask data in a meaningful manner. Could you please indicate if there are errors in what we are doing or possibly provide a minimal working example. Thank you

[NinjaTrader_Brett]

ggggg,

http://ninjatrader.com/support/helpGuides/nt8/en-us/developing_for__tick_replay.htm

Please see the resource above for some guidelines to follow. Let me know if you felt that didn't go into enough details and ask questions and I will update the document.

For you question though there is actually physically Last (With Bid/Ask), Ask, and Bid physical series on the disk and saved. Tick replay plays back only the Last series which has its bid/ask stamped at the time the trade occurred. So if you are manually creating the data you would need to only create the last and populate the bid/ask fields on the last data.

[us8lander]

NT7 Insight

I am dealing with this same issue, but am still on NT7. Can someone point me to the details on Fillprice algo/approach that SImulator uses.... I am getting fills that don't make sense based on bid, ask and last tick data received through OnMarketData event processing ...
Thanks
us8lander

[ggggg]

Brett,

Thank you very much for your answer.

Accessing historical bid/ask via the OnMarketData() event

What the document does not describe is where the bid/ask data in tick replay comes from. As shown by our experiment, importing historical bid, ask and ticks does not generate the "Last" data with bid/ask at the time of the trades mentioned by you. This is quite counterintuitive.

It is not clear how one can create the desired "Last" with correct bid and ask data. In the historical data viewer, there are no bid and ask columns associated with "Last" so I am not quite sure what you mean by

So if you are manually creating the data you would need to only create the last and populate the bid/ask fields on the last data.

I am wondering if there a way to import such data from a single CSV file having additional columns for bid and ask or from separate CSV files with trades, bids and asks.

Historical Fills Processing

Since the fill simulation algorithm does not use the bid/ask data the usefulness of importing it is quite limited in the first place. I think the fact that bid/ask data is not used for calculating fills is worth mentioning explicitly in the documentation both in the section on Tick Replay and the one on Historical Fills Processing.

Speaking of the historical fills, the very detailed understanding of how it works is absolutely necessary for any serious user since it is a key assumption of a backtest. In my view, the documentation should make it extremely clear how limit and market orders are simulated given the data at hand and the backtest settings. The required level of detail can be demonstrated by the following examples

• Limit orders: if "fill on touch" is false, the standing buy (sell) limit order is filled when the current bar's low is strictly lower (current's bar high is strictly higher) than the order price. If "fill on touch" is true, the rules are similar with the strict equalities ("strictly lower" or "strictly higher") replaced with their non-strict versions ("lower or equal" and "higher or equal")".
• Market orders: once a market order is submitted it is filled at the end of the next bar. The fill price equals ... plus the slippage parameter.

The current documentation on fill resolution setting is insufficient:

Selecting order fill resolution of "High" will allow you to set a secondary bar series to be used as the price data to fill your orders, this allows you to bring in more granular data then you are currently running the strategy on. For example you may have a strategy that you run on "Daily" bars but then want to bring in "Minute" bars for the historical fill algorithm to be based on.

It is not exactly clear what happens when one uses, say, daily data for signals and minute data for fill simulations given that daily data lacks intraday timestamps. If one submits an order in OnBarUpdate() what is the time it will be considered submitted?

Another unclear point how historical fills processing works when one uses tick data granularity (since tick data is different from OHLC bars). Instead of high, low, open and close we have only one price so personally I am not quite sure what happens in this case.

Thank you.

[NinjaTrader_Brett]

ggggg,

Thanks for the response.

1) No way to manually import in the bid/ask attached properties to the last. Nor can you view them in the Tools > Historical Data.

As to what providers provide the data I will have this page update to reflect it: http://ninjatrader.com/support/helpG...y_provider.htm

Primarily its Kinetick, IQ Feed, and any provider utilizing the NinjaTrader historical data server such as CQG, Continuum, Rithmic.

2) The use is bid/ask trade analysis for indicators and strategies. Was never intended for use during backtests.

3) I will look into improving the documentation in this area. As I agree we could add some more detail to the doc then what we had in NT7 since we no longer have it 'open source'. Which I will discuss that decision internally as well.

[NinjaTrader_Brett]

us8lander,

NT7 is completely different then NT8 with fill logic. Please post a thread in the NT7 forums so we don't confuse the two. We will help you understand NT7 logic there.

[Matheyas5]

Hi NinjaTrader_Brett,

I'd just like to say I completely agree with everything rspine has said in this thread.

[NinjaTrader_Brett]

Thanks for letting me know.

We have increased our documentation on how the fill engine works, will be pushed with our next release.

As to the item of wanting the fill engine to also use bid/ask prices to fill when in Tick Replay mode then I will add your vote to our suggestions and feedback tracking. SFT-982

[Matheyas5]

Great Brett.

Regarding your quote below, can you please verify that you are talking about Bid prices and Ask prices that existed both on and/or in between Last Ticks, rather then just the Bid/Ask prices that existed On Last Ticks? I highly suggest the former.

As to the item of wanting the fill engine to also use bid/ask prices to fill when in Tick Replay mode then I will add your vote to our suggestions and feedback tracking.

Also, I highly suggest adding feature request SFT-1324 (shown here http://ninjatrader.com/support/forum...280#post456280) which would probably solve all the problems brought up in this thread (expect for the ability to customize/control the fill engine).

I truly feel Ninjatrader 8's and 7's main weakness is how complicated it is to run an accurate backtest/optimization in the Strategy Analyzer. I think Ninjatrader is a far more superior platform then Multicharts, but Multicharts does make it much easier to run a accurate backtest/optimization.
If any other users reading this agree, please vote for SFT-1324 as well.

P.S.
As somewhat mentioned in the SFT-1324 link above, Ninjatrader's support forums seem to be filled with an enormous amount of threads (such as this thread) regarding trying to understand/fix/workaround accuracy issues within the Strategy Analyzer. I think adding SFT-1324 will highly cut down on the time Ninjatrader's support staff (and its users) have to spend on tech support questions.

[NinjaTrader_Brett]

Great feedback, it's appreciated.

I edited the SFT to be clear on possible solutions you point out. I've had requests for either solution you point out over time and it is not clear yet which one we would implement based on demand. Right now our focus is getting NT8 into the wild so it will be some time before this is reviewed.

-Brett