Historical Data, Market Replay and Tick Replay

[demarcog]

Hi Manfred,

I'm sorry for the long response I never intended to do that, there is no rush at all in getting back to me and a short response with a general direction is fine, I don't need every question explicitly answered the came to me as i was writing this. This data recording, processing, back testing etc is not a simple concept with many subtleties. But for those of us who need to make a living doing this you really need to understand it thoroughly.

Thank you for the informative response. I always learn something new f along with getting my issue resolved which is awesome.

I had a OneDrive calamity yesterday and had to reinstall NT8 with the help of Zachary and others, and manually copy over all the customizable components and there may be one of two settings I was running with that I have to reenable/disable.

I have to admit this I am a little confused about the relationship between the Chart Tick Replay you mentioned, the Tools - Option Box "Show tick replay" , Tools Options "Enable market recording for playback" and Tools Options "Record Live data as historical" . Playback101 mode for tick or historical.

Her are my settings (attached) now. I noticed after the reinstall Enable market recording for for playback' is not checked. It was previously checked because every time I wanted to connect and download a days tick data to test a strategy on Playback101 I had uncheck that box, and usually check it back when I download the data.

If I can download the tick data anytime do I need to "enable market recording for playback" all the time?

Does having that checked actually record all the ticks for that instruments and why would someone need to do that?

What does "Record data as Historical" do" NT servers can supply all the historical data you need by setting the Data Series Days parameter?

What would be the point of running Playback101 testing a strategy using the rebuild Historical data?

I'm trying to get as close to real market fills as possible so isn't it more accurate to replay the tick data?

I did try to find the answers myself it's not fair to ask questions that are easily found in the doc, but this is confusing to me after reading the doc. I saw some code on how to process tick data within a strategy but the strategy would need to be coded for that purpose. Is that a way of getting a more accurate back test without doing a Playback101 session. The doc said that processing the data as historical didn't seem to provide more accuracy then tick by tick, and market fill resolution is a better option then what the purpose of processing by historical.


My understanding is that to run Playback I have to download the tick data one day at a time from the server.

Before I can do that I believe I have to disable the "enable Market for playback" box?

What I want to accomplish is have some means of doing the most accurate back test possible without bogging down my computer in the process.as accurate.

I am fine with manually downloading tick data one day at a time for the symbols I want to trade. Does this suggest turning off "enable market recording for playback?

Yesterday when I reinstalled and copy over everything manually, this biggest files looked like tick data for many symbols. If I'm not coding my strategy for processing ticks and the strategy analyzer does not yet have that capability, and the data is available on NT servers then what should my settings be?

Manfred I'm sorry for asking too many questions. Since COVID this is how I support my family and not doing such as good job at it or else I would never burden all you guys with questions that I probably should have gotten answers from the doc. The doc is for the most part beyond excellent. I was in IT for 30 years and wrote tons of doc and NT is the best I've seen for any product. I usually "get it" but can't quite put the pieces of this data recording set in my head. So I appreciate your help and apologize for all the questions.

Thank you

glen

[NinjaTrader_Manfred]

Hello Glen,

Please never worry about asking us questions, after all, that's what we're here for!

As this explanation is a bit longer and not directly related to the original post anymore, I've split the post off. This ensures it can easily be found in the forum, should another customer be looking for this information later on.

To first get the difference between historical and market replay data out of the way;

Market Replay data is tick-based data that will allow NinjaTrader's Playback feature to replay the data as it happened. This is, by default, the most accurate way to replay the data and generally suggested for playback.

Historical data, on the other hand, is the same data the platform requests whenever it needs to fill a chart. This data may be day-, minute- or tick-based.

Historical data can either be downloaded by opening a chart that requests this data or manually, through the Tools > Historical Data window, as long as you are connected to a provider that supplies this data.

Market replay data, on the other hand, is downloaded from NinjaTrader's internal server, does not require a real-time subscription, and is supplied for the most commonly traded instruments, going back 90 days.

To download market replay data, please click on Tools > Historical Data, then switch to the "Load" tab and click on "Get market replay data".

As this can reach several MB in size (per day and instrument), it can only be downloaded one instrument and day at a time. However, there is no limit to how much data you may download this way.

Instead of downloading this data, the NinjaTrader platform can also record it live, which is what the "Record live data as historical" and "Enable market recording for playback" options are for. If these are enabled, NinjaTrader will record any data that arrives as live data to either be used as historical or market replay data.

In general, enabling these options is not necessary, as both, the historical data and the market replay data can be downloaded directly as well.

To explain how "Tick replay" works; If this option is disabled (default), any indicator added to a chart would use the historical data as it is charted to calculate its values.

That means if you have a 1-minute chart going back 30 days, the indicator would be based on 60 x 24 x 30 candles (60 candles per hour, 24 hours per day, 30 days of historical data).

If Tick replay is enabled, however, the indicator would now be built based on tick-data alone. That means as the chart loads, the indicator will receive 'tick after tick'.

As this could include thousands (if not millions) of data points, it would then take significantly longer to calculate these values (as you have mentioned, tick-data can reach a quite significant size).

It's important to understand that only a few indicators would benefit from this granularity.

A regular moving average indicator, for example, would not benefit from data in this granularity, as it is already only using one value in each candle, whereas a complicated order flow indicator using volume in its calculation might.

More information regarding Tick Replay may be found in our HelpGuide at the following address:

https://ninjatrader.com/support/helpGuides/nt8/?tick_replay.htm

Please don't hesitate to let me know, should you have any further questions!

[demarcog]

Manfred,

thank you very much for the very informative response. I realize tick replay is the most accurate when back testing a strategy especially when a non time interval is being used.

What I find very useful as a replacement for doing real time sim trading during market hours is to run Bar Replay. The speed can be quickened a bit and its very useful to be able to place orders and have P & L tracked as the trading "day" progresses. Last night I did it for the first time and it was extremely useful. I Downloaded the tick data one day at a time and then switched to the Playback connections and the bars were painting like the would during normal market hour and I was able to "trade".

I've heard many a successful trader and for me it's absolutely the best training method to do playback trading as long as it take to show a consistent profit. So for me it's one of the best features of the product.

This evening I tried doing the same thing and had downloaded the tick data last night. I switched to the Playback connections for a symbol and started the recorder and saw the time increasing in the recorder box. However when I tried executing a trade a message saying the playback data has not been downloaded. I again downloaded two weeks of tick data and attempted to start playback and the chart never moved. I downloaded every type of data available and the chart would still not advance. The data series was the same dates as the data and I just gave up after a few hours.

This has happened many time for me with tick replay and bar replay. It takes a long time when the slider is moved or never come back to an active state. I see the downloaded data in edit for the days and symbols I downloaded and still no luck.

I have no idea what I'm doing wrong as it's a relatively simple function but I cannot get it to work?

No sooner did I hit save on this post that now the chart is moving and I changes nothing while I wrote this post. I'm interested in ding tick replay occasionally for validating strategies and bar replay for simulating a trading day. What are the simple steps that need to be done for those two replay types. Any ideas whey I'm seeing this temperamental behavior where the bar doesn't advance then springs back to life after putting a post to report the isssue


I don't think I need to record tick data for every symbol as it may slow things down so for now I've leaving that option off. It only takes a few minutes to download a few weeks of data.

thank you,

glen

[Carlito178]

What settings do you use to get the playback as accurate as live data? I'm having trouble doing this.

[demarcog]

Carlito,

Do you do tick or historical playback and what chart types?

glen

[Carlito178]

I download the Market Replay data and use that. Would you say Historical moves as fast as the live data?

[demarcog]

Carlito,

I'm hesitant to answer and welcome any Support Staff jumping because unless I'm certain about something I prefer not to give someone an incorrect answer.

There is a speed setting on both the historical and tick replay that goes from 0 to Max. I'm assuming 0 is a close as a live market as possible but unless that have some complex logic in there and have time stamps on all the data records this may be a relative and not a wall time factor increase. You can certainly increase the speed to a faster pace then a live market . The problem for tick data or non time based charts it's difficult to come up with a speed where you aren't sitting there as if in a live market for long time periods with no price movement. However then there will be a flurry of activity where the market screams in one direction and it's too late to put in a order and can't really rewind to prior to the move. That kind of defeats the purpose because then you know which want the market went.

It's not clear to me in tick replay you can choose tick or historical data, is the historical data the Historical Data used in tick replay the historical data downloaded in the Historical Data downloaded where you speecify tick, minute, daty, bid, offer,last etc. Or is the Historical Data Radio button in the replay recorder build from the tick data downloaded?

Asked another way what do I need to do to only run the historical option in playback mode?

Plus Ninja has tons of data available that a chart specifying a large number of days will immediately pull from their servers.

NT8 is the best trading, charting, testing product ever created and with that power and advanced functionality comes complexity so these problems and questions we have a good ones and speaks to how great the product really is.

It's difficult initially getting some of this down but one it's understood you think, that's simple what was my problem.

Fortunately NT support are the best I've even seen in responding with answers that clear up the complexities.

Good Luck with you testing.

glen

[NinjaTrader_Manfred]

Good Morning, Glen and Carlito!

First of all; Thanks for the praise, we might not be able to reply to forum posts 24/7, but I can assure you we will do our best to assist everyone that needs assistance during our office hours!

Regarding the replay, I've listed some of the questions that came up below and will answer them individually.


The "Playback Speed" during Playback

The way the speed-slider works is more or less like a VCR. On the lowest setting, it will play 1 second of recorded data every 1 second. Much like watching a movie in real-time.

The higher you push the setting, the faster NinjaTrader will attempt to 'feed' the data to your charts. However, it will only be able to feed it as fast as the charts/indicators accept it.

That means, even if you push it to 1000x, or "Max", it will always only run as fast as the chart can accept the data.

If your chart moves slower than expected, (e.g. less than about half a minute every second, when set to "Max"), I would suggest taking a look at anything that can slow your performance down, e.g. multiple open workspaces, Tick Replay, complicated indicators,...

One option would also be to create a dedicated workspace only for playback, which could then be limited to just what you need during the playback. In short; The less NinjaTrader has to calculate/chart, the faster the playback will run.


Most common issues if the chart does not move while playback is running

Most commonly, this is caused by either the lack of available data (e.g. if there is a gap just before the playback is meant to start or tick-data is not available on historical playback, see below for more information) or incorrect settings (e.g. charting a volume or Renko chart without available tick-based data or an expired instrument that has since rolled over).

As these reasons can be widespread, it'd be easiest to take a look at your log & trace files and/or schedule a remote session. Simply reach out to us at platformsupport (at) ninjatrader (dot) com and we can take a closer look.


Tick Replay in Historical/Market Replay

Whether Tick Replay is enabled or not would affect the data it loads, just how it loads the data. To give you an example, if you are using a 1-second chart on the ES 03-21 and don't use tick replay, each indicator will get 60 'packages' of data, one for each second. With Tick Replay enabled, this could jump up to a thousand or more 'packages' per minute, as now, every single tick is replayed as it occurred, which could easily be 30+ ticks per second.

This will also cause delays or slowdowns if used on performance-heavy charts, due to the nature of the Tick Replay itself.


Charts don't move despite historical data being available

While we would have to investigate further, the most common cause here is missing tick data.

While you may selectively download tick, minute, and day-data when downloading the historical data through NinjaTrader, the historical playback would not chart if tick data is not available.

To explain this limitation: If you only download minute-based historical data, even minute-based charts would not chart as they would only be able to chart a single update every 60 seconds (as minute-candles only contain the open, high, low, close, and volume-data of the 60 seconds they cover).

Should this be the case, please also download tick data, you should then be able to use historical data to replay.


Please don't hesitate to let me know, should you have any further questions in this regard. I have also linked both "Performance Tips" HelpGuides below:

• NinjaTrader 7
• NinjaTrader 8

[Carlito178]

Thank you guys for you replies.

My thing is that I just want the playback to be real time like it is when it's live, as in I want the candles to move as fast as they do when it's live at the opening, so I can test my strategy like Glen said.

I made a post about it and the response was

"Hello Carlito178,

Thank you for your note.

Unfortunately, this would not be able to be changed as this was done purposefully to optimize the performance of the Playback connection."



Surely it could be done some how? In what way would it affect the performance?

[NinjaTrader_Manfred]

Hello Carlito178,

To follow-up on Spencer's post in this regard; the offset would be minimal, and the result (e.g. profit and loss) would be identical. However, it might indeed run slower by a very small amount, even if set to 1x.

To explain this in a bit more detail; Real-time data arrives from the exchange in real-time. That means the platform permanently receives a stream of data, which it then charts.

As this data arrives as a permanent stream, the platform will always play a 'catch-up' game and chart all the data it receives, before feeding the indicators. In a best-case scenario, this buffer is empty and all data is being charted as it arrives.

In this case, the exchange is the platform's 'heartbeat'.

If the platform ever receives more data than it can chart, e.g. if you use very complicated charts and indicators, it will 'buffer' the tick data and try to catch up ASAP. However, it can never run faster than the exchange, as it relies on this 'heartbeat'.

If you use market replay, however, this behavior changes very slightly. In this case, the "fake exchange" that our replay adapter creates will send data to the platform, then wait until this data is charted before sending the next package. That means unlike the live exchange, this adapter will slow down if your chart can't catch up.

This ensures that even if you push the speed to 'max' it would not 'skip a beat', but always chart everything in order.

However, this can cause a very minor delay, as the process would always be to send the data, wait until it's charted, then send the next package. While it would still try its best to run at 1 second per second, this behavior (which ensures that no data is dropped during the replay) can cause a very minor offset. This offset should be minimal and can usually be neglected.

The best way to improve this behavior would be to slim down the workspaces and ensure as little processing power is needed as possible, e.g. by closing unused workspaces and limiting the charts on the active workspace.

I hope this explains this behavior a bit better. Please don't hesitate to let me know if you have any further questions!

[demarcog]

Mnafred,

thank you for that very informative post there was a ton of good information in there. You were dead on I have two two charts same symbol same strategy running Max.

One is updating very slowly the other looks like it doesn't like the most it pauses is a second or two and sometimes multiple paints of the screen per second.

The one that is running slower was 20 days specified in Data Series, once I changed that to 1 it kept up with the other.

I then took out all 20 indicators from one chart and left them in the other, there is not detectable difference.

I assumed for a strategy if you specified Max the results of the playback P & L test would be less accurate but it seems to handle it well, it's pretty a pretty cool thing to watch.

I did notice that even on a slow rate the bar where I'm getting filled, which is the correct entry bar based on the systems rules is not the bar filled in Market Relay. Which is really a bummer.

Occasionally I use non time based charts. I know they inflate the profitability for example HA or Unirenko the open of the next bar where filled is always a better price then you would get in real trading. But Having my criteria met and the trade not firing for 4 bars on entry an 1 or 2 on exit . The result is a system that when I manually test is profitable looses a lot of money.

As I'm typing both charts stopped updating about 1/2 the way through the slider and data, The CPU is about 70 percent higher usual so I suspect it's doing something.

Perhaps it's missing some data, I moved the slider up a bit and it took a while but is running again.

Thanks again for your help.

glen

[NinjaTrader_Manfred]

Hello Glen,

If the data is not consecutive and missing even a single minute in between, this would definitely explain if the chart stops progressing (it requires consecutive data to chart).

If in doubt, I would suggest redownloading the historical/market replay data to fill up the missing time.

Should you still encounter any issues, please let me know!

[demarcog]

Manfred,

Will do thanks.

glen

[rezik]

If you use market replay, however, this behavior changes very slightly. In this case, the "fake exchange" that our replay adapter creates will send data to the platform, then wait until this data is charted before sending the next package. That means unlike the live exchange, this adapter will slow down if your chart can't catch up.

NinjaTrader_Manfred


Based on other posts on the topic of replay, I understand that the market replay engine is sending the data in 1 second intervals, and based on your comment above could slow down if the platform can't catch up, which, for the integrity of having accurate data on the chart for replay makes sense.

However, it is possible to reduce the 1 second interval to be that of the actual data, down to the nanosecond that's recorded and have the replay engine send the data just like the live exchange would be? Why limit it? The live exchange is not limiting the data feed, and NT will do it's job based on the data it's fed, and based on your comments above, cache/buffer if it gets behind, so is there a way to pull the 1 second timer off replay and just let it feed the data properly and truly simulate the feed of data from the exchange? This would really open up replay to allow for a more realistic experience and not limit the indicators/UI/etc from processing in 1 second batch intervals.

Looking forward to your response.

[NinjaTrader_Manfred]

Hello rezik,

We have a currently pending feature request that would enable smaller granularity for the replay connection, we're recording interest in this feature under the internal ID SFT-5367, I have added a vote to this on your behalf.

Once a feature is implemented, you may find further information in our Release Notes:

https://ninjatrader.com/support/helpGuides/nt8/release_notes.htm

If there is anything else I may assist you with, please let me know!