Are Tick Replay and Bid and Ask historical files really necessary?

[Ricam]

According to the Ninjatrader website, Historical Bid/Ask prices are now stored with each last trade tick data.

1. If this is the case, why can't the bid and ask prices associated with the trade price of a secondary one tick price series be accessed directly in OnBarUpdate, without having to use Tick Replay?
2. The prices from the bid, ask and last historical data files cannot be relied upon to be synchronized because of overlapping time stamps. Which raises the question of what purpose the bid and ask files serve, since I can't think of any way to use them for anything. Why not store the bid, ask and last in ONE file, with all records in the correct chronological order as received from the data feed?

[NinjaTrader_ChelseaB]

Hello Ricam,

You can access the tick ask or tick bid data without using Tick Replay by adding these as secondary series to your script using AddDataSeries.

Enabling tick replay does this for you for convenience, but uses the same data that would be used if you add the series to your script yourself and uses this for providing snapshot tick data to OnMarketUpdate which typically only runs in real-time.

This new addition is meant as an easier alternative to coding your script to use intra-bar granularity.

Can you provide historical data that demonstrates the downloaded historical data is out of sync?

[Ricam]

Hello Ricam,

You can access the tick ask or tick bid data without using Tick Replay by adding these as secondary series to your script using AddDataSeries.

Enabling tick replay does this for you for convenience, but uses the same data that would be used if you add the series to your script yourself and uses this for providing snapshot tick data to OnMarketUpdate which typically only runs in real-time.

This new addition is meant as an easier alternative to coding your script to use intra-bar granularity.

Can you provide historical data that demonstrates the downloaded historical data is out of sync?

There seems to be a lack of internal communication over there.

The method you suggest is in conflict with Ninjatrader's own documentation and sample code.Tick Replay is not some kind of alternative shortcut "for convenience" It is the ONLY way to get the best bid and ask associated with a trade. These prices are hidden away someplace in the Last file where they can't be accessed other than in OnMarketData in Tick Replay.

You can't get the correct bid and ask associated with a trade by sequentially reading the bid, ask and last of a one tick data sereis.. The Ninjatrader Help article, buy sell pressure and volume indicators, and sample code for getting bid ask do not say to use the method you mention. I can't find it now, but at one time there was a specific warning to NOT use this method..I did try using this method and can assure you that it gives worthless, mostly random results.

The reason this can't work is that there are often multiple records in each of the files bid, ask and last that have the exact same time stamp. (Presumably this is because of the way the exchange matching engine works, generating the same time stamp for multiple events that can happen each time there is a match..)

Therefore in the historical state there is no possible way for the various BIP events to fire in the correct historical sequence since the sequence information cannot be reconstructed from three separate files that have overlapping multiple records with the same time stamps. This is the whole point of why according to the former Ninjatrader documentation that i can't find anymore, Tick Replay is currently MANDATORY for historical bid ask data. Maybe the information is in the Release Notes somewhere, I don't want to look right now.

This makes the bid and ask files completely useless. Meanwhile my queries about putting the bid, ask and last events in one file in correct chronological order, and about modifying GetCurrentBid and GetCurrentAsk to work historically on a one tick data series by getting the information from the Last file, continue to be ignored. Please let me know if I may be of further assistance.

[NinjaTrader_ChelseaB]

Hello Ricam,

From the help guide:
"Tick Replay was NOT designed to provide accuracy in backtesting concerning order fills and execution and should NOT be used to expect the exact sequence of executions as running a strategy on live data. For greater order-fill resolution and accuracy in strategy backtesting, you can use the High Fill Resolution in the Strategy Analyzer".

http://ninjatrader.com/support/helpGuides/nt8/en-us/developing_for__tick_replay.htm

If you are wanting the tick replay for accurate fills, this will not do that.

I highly recommend you review this document in the help guide.

That said, adding intra-bar granularity of ask and bid as I have suggested in my previous email, will do just that.

This is how intra-bar granularity was added for accurate fills and for intra-bar actions (typically in OnMarketData but instead in OnBarUpdate) with NinjaTrader 7.

We have a reference sample that demonstrates how to do this.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

For further information see:

http://ninjatrader.com/support/forum/showthread.php?p=463117#post463117

You can get the correct bid and ask in OnBarUpdate by adding a secondary series of 1 tick ask and a secondary series of 1 tick bid.

http://ninjatrader.com/support/helpGuides/nt8/en-us/adddataseries.htm

Do you have any historical data / output to demonstrate the ask and bid prices are out of sync?

[Ricam]

What are you talking about?
I never said anything about back testing.

Did you even read what I said? The points I raised have not been answered, and new straw men have been brought in, adding nothing other than confusion.

I think you need to refer my question to Brett,

[NinjaTrader_ChelseaB]

Hello Ricam,

Tick Replay is only used when backtesting (historical).
This does not apply to real-time data.

Are you referring to real-time or backtesting (historical)?

[Ricam]

I am using Tick Replay for accessing bid ask historical prices to calculate the deltas.

Your statement about Tick Replay only being for back testing is just plain wrong.

Since you don't know what I am talking about, please refer this to someone who does.

[NinjaTrader_ChelseaB]

Hello Ricam,

This is stated in the help guide.

From the description at the top of the help guide:
"In general, this means adding additional logic to the OnMarketData() event handler, however, Tick Replay can also be used to call OnBarUpdate() "OnEachTick" or "OnPriceChange" during historical calculations."

http://ninjatrader.com/support/helpGuides/nt8/en-us/developing_for__tick_replay.htm

Also, from the same help guide document:
"Tick replay was only ONLY designed to work with MarketDataType.Last. A TickReplay indicator or strategy should NOT be mixed with a MarketDataType.Ask or MarketDataType.Bid series."

I highly recommend you review this document in the help guide.

I will escalate this thread to the NinjaScript lead.

[Ricam]

I already know that. You do not seem to be able to understand what I have been saying.You are in over your head, please refer this to somebody else.

[NinjaTrader_Brett]

To clear up some confusion:

* Tick replay replays 'last' events (trades) and along with the 'last' event throws the current state of bid/ask at the time of 'last' which is recorded and saved with the last event.
* Yes you are correct in that bid/ask data files now no longer need to be used for such traditional type of analysis.
* They still are available since for several reasons:

1. Forex only has bid/ask and does not have last therefore you still need traditional bid/ask there.
2. There are some analysis that can be done based on rate of changes of ask/bid. Tick replay you would not be able to do that since we only store the current state of bid/ask with last at the time of a last event. You would mis all bid/ask changes made from the provider in between. Bid/Ask would retain that directly.

This is the technical difference between them.

-Brett

[Ricam]

Restating previous questions

There were a lot of posts in this thread since the first one, but my original questions remained unanswered.

According to the Ninjatrader website, Historical Bid/Ask prices are now stored with each last trade tick data.

1. If this is the case, why can't the bid and ask prices associated with the trade price of a secondary one tick price series be accessed directly in OnBarUpdate, without having to use Tick Replay?
2. The prices from the bid, ask and last historical data files cannot be relied upon to be synchronized because of overlapping time stamps. Which raises the question of what purpose the bid and ask files serve, since I can't think of any way to use them for anything. Why not store the bid, ask and last in ONE file, with all records in the correct chronological order as received from the data feed?

[NinjaTrader_Brett]

Sorry about that, see below my answers inline to your question below.

If this is the case, why can't the bid and ask prices associated with the trade price of a secondary one tick price series be accessed directly in OnBarUpdate, without having to use Tick Replay?

[Brett]: Since that assumes you will be required to run the indicator Calculate=On Each Tick to access the Tick Replay data. Having it playback in OnMarketData which is how you would access the data in realtime without using TickReplay is consistent and doesn't come with the penalty of forcing indicator calculations on each tick.

The prices from the bid, ask and last historical data files cannot be relied upon to be synchronized because of overlapping time stamps. Which raises the question of what purpose the bid and ask files serve, since I can't think of any way to use them for anything. Why not store the bid, ask and last in ONE file, with all records in the correct chronological order as received from the data feed?

[Brett] : If you want to do analysis on all Bid updates, use MarketDataType.Bid. If you want to do analysis on trade(tick) data but want to get the current state of bid/ask at the time of the trade (tick) then use tick replay.

[Ricam]

still no answer to item 1

Tick Replay is a performance disaster during chart loading so I will ask once again:


Originally Posted by Ricam View Post
According to the Ninjatrader website, Historical Bid/Ask prices are now stored with each last trade tick data.
If this is the case, why can't the bid and ask prices associated with the trade price of a secondary one tick price series be accessed directly in OnBarUpdate, without having to use Tick Replay?
The prices from the bid, ask and last historical data files cannot be relied upon to be synchronized because of overlapping time stamps. Which raises the question of what purpose the bid and ask files serve, since I can't think of any way to use them for anything. Why not store the bid, ask and last in ONE file, with all records in the correct chronological order as received from the data feed?

[timmyb]

I also am curious why I pay 80+ per month for a quality data feed that Ninja is unable to use the data for correctly. Especially with the new rollout of your ladder version etc. I would personally li**** see and ability to line up the trades in correct chronological order received from the data feed.

[NinjaTrader_Brett]

NinjaTrader 8 does have support for bid/ask stamped last events, so that you could use 1 tick data to do a bid/ask comparison to see if that tick occurred at/above the ask or at/below the bid.

This is part of the data Tick Replay uses to throw OnMarketData events when the tick replay simulation is running to rebuild the bars (and subsequently any applied NinjaScript indicators/strategies). You can use Tick Replay to get access to the bid/ask stamped data inside of OnMarketData, however it is not Tick Replays primary use case. Where Tick Replays primary use case is to simulate building bars as they would have been built in real-time, and hence have a performance penalty.

If all you need is the bid/asked stamped data on a 1 tick to do some analysis if that tick was a buy/sell, then using Tick Replay could be considered overkill. Executing the below code on a 1 tick series will allow you to get the bid/ask stamped data that is attached to a 1 tick trade without requiring the use of Tick Replay.

double askPrice = BarsArray[1].GetAsk(CurrentBars[1]);
double bidPrice = BarsArray[1].GetBid(CurrentBars[1]);

Note: There are some implication of this approach, since now you need to deal with building a multi-time frame indicator which is not as simple as a regular single series indicator. However, it performs much better then requiring Tick Replay if bid/ask stamped last data is all you require. Its for sure a trade off between selecting which method to use, there is no 'best' approach.

Note: The above code sample will only work on an Indicator/Strategies with an added 1 tick series.