How to Obtain Accurate Historical Tick Data

[simple_goodoboy]

Hello,

I am new to Ninja Trader and back testing programmed strategies. I am learning day to day.

I hope my question is not too simple.

I read this in the help files "During a backtest you can select conservative or liberal fill algorithms which will produce different results. Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest."

All my trading strategies I want to back test are on the 3 minute candle stick chart. Basic breakout strategy, buy/sell on a calculated entry/stop/profit target price. So using the OHLC 4 data points will not help much.

Questions:
1. How can I obtain high resolution historical data to use with the Strategy Analyzer?

2. Do I need to purchase a data feed from here https://ninjatrader.com/support/help..._strategy.htm?

3. When I am running my strategy in back test do I select 3 min like on the attachment?

4. What is the purpose of 1 second bar chart like in the attachment? And can this somehow help me with obtaining the tick data feed I need rather then buying data.

Thanks,

[bltdavid]

I read this in the help files "During a backtest you can select conservative or liberal fill algorithms which will produce different results. Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest."

All my trading strategies I want to back test are on the 3 minute candle stick chart. Basic breakout strategy, buy/sell on a calculated entry/stop/profit target price. So using the OHLC 4 data points will not help much.

Why do you say that?
I mean, isn't that kind of non-sensical?

The 3-minute bars on a chart are built internally by the NinjaTrader code from 1-minute bars, which is the only kind of minute data stored(*) as historical.

Every one of your 3-minute bars has 4 data points (actually 6, which I'll get to).

Each bar has an Open, High, Low, and Close price (known as OHLC).

5th data point: each bar has a Volume.

6th data point: each bar has a Timestamp.

What additional data points do you think you are missing?

(*) Minute historical data is stored with 6 data points (OHLCVT) for each 1 minute bar. All n-minute historical bars for all n-minute charts are derived as needed from this 1-minute historical data. This includes backtesting for the Strategy Analyzer.

Questions:
1. How can I obtain high resolution historical data to use with the Strategy Analyzer?

Chances are you already have the data.

Use the checkbox "Save chart data as historical" in the Data tab of Tools -> Options.

Or, use Tools -> Historical Data Manager to explicitly download the tick/minute/day data you desire for the time periods needed.

The Strategy Analyzer does a pretty good job of implicitly downloading the needed data (tick and/or minute and/or day) before backtesting.

2. Do I need to purchase a data feed from here https://ninjatrader.com/support/help..._strategy.htm?

Depends on the type of historical backtesting you want to do.

There are 3 types of historical data: tick, minute, and day.
Which do you need?
What date ranges do you need?

Why are these questions important?
Well, for example, NT only keeps about 1 years worth of tick data, but many many years of minute data.

3. When I am running my strategy in back test do I select 3 min like on the attachment?

Yes.

4. What is the purpose of 1 second bar chart like in the attachment? And can this somehow help me with obtaining the tick data feed I need rather then buying data.

It defaults to 1-second because some programmer at NinjaTrader, many years ago, when that code was being written, decided that that was going to be default. Think about it, those fields have to default to something, right? But, it doesn't matter, just change it to whatever you want, in your case 3 Minutes.

If your using 3-Minute bars for entry, are you sure you need tick data?

I mean, did you add a secondary data series to your strategy for better granularity, and thus you are sophisticated enough in your strategy design that you know you need tick data to support this tick granularity?

Summary:
Unless you are sophisticated enough to have a need for tick data during back testing, the minute historical data (which is plentiful and free) is probably all you need.

[bltdavid]

2. Do I need to purchase a data feed from ...

I seriously doubt it.

Why?

#1 - you're only using 3-minute bars, which are created just fine from NinjaTrader's own 1-minute historical data, which is already plentiful and free from NinjaTrader (assuming you have a brokerage account to connect to NinjaTrader, of course).

#2 - use Historical Data Manager to download Minute data for the date ranges you need.

#3 - if you were sophisticated enough with your strategy testing needs, you'd probably already know the answer, and wouldn't be asking here.

#4 - but, if you're more a newbie, the answer gravitates more towards "the free historical minute data from NT's historical data servers is almost assuredly enough" which means the answer you're looking for is: "no, you do not need to buy extra data".

[simple_goodoboy]

Why do you say that?
I mean, isn't that kind of non-sensical?

The 3-minute bars on a chart are built internally by the NinjaTrader code from 1-minute bars, which is the only kind of minute data stored(*) as historical.

Every one of your 3-minute bars has 4 data points (actually 6, which I'll get to).

Each bar has an Open, High, Low, and Close price (known as OHLC).

5th data point: each bar has a Volume.

6th data point: each bar has a Timestamp.

What additional data points do you think you are missing?

(*) Minute historical data is stored with 6 data points (OHLCVT) for each 1 minute bar. All n-minute historical bars for all n-minute charts are derived as needed from this 1-minute historical data. This includes backtesting for the Strategy Analyzer.



Chances are you already have the data.

Use the checkbox "Save chart data as historical" in the Data tab of Tools -> Options.

Or, use Tools -> Historical Data Manager to explicitly download the tick/minute/day data you desire for the time periods needed.

The Strategy Analyzer does a pretty good job of implicitly downloading the needed data (tick and/or minute and/or day) before backtesting.



Depends on the type of historical backtesting you want to do.

There are 3 types of historical data: tick, minute, and day.
Which do you need?
What date ranges do you need?

Why are these questions important?
Well, for example, NT only keeps about 1 years worth of tick data, but many many years of minute data.



Yes.



It defaults to 1-second because some programmer at NinjaTrader, many years ago, when that code was being written, decided that that was going to be default. Think about it, those fields have to default to something, right? But, it doesn't matter, just change it to whatever you want, in your case 3 Minutes.

If your using 3-Minute bars for entry, are you sure you need tick data?

I mean, did you add a secondary data series to your strategy for better granularity, and thus you are sophisticated enough in your strategy design that you know you need tick data to support this tick granularity?

Summary:
Unless you are sophisticated enough to have a need for tick data during back testing, the minute historical data (which is plentiful and free) is probably all you need.

Thank you bltdavid for the detailed response. You certainly help me. But still a bit confused.

I am certainly new and doing my reading for understanding before implementing anything for a back test.

Thanks for the knowledge, yes I can use 1 minute bar for better back test resolution because three 1 minute bar equals 3 min bar.

My goals:
Develop , program, and back test some intra day trading strategis on the 3 min chart. Back test can range 1 to 3 years of historical data.

My concern or overthinking:
I am concern that my pre calculated entry, stop loss and profit target in the strategy will not be filled because the calculated price may not be one of the prices of the OHCL of the next 3 minute bar.
I trade calculated resistance and support breakouts.
For example,
Resistance = 55.58

If bar close above 55.58,

Entry imit order at 55.60
Stop loss at 55.45
Profit target at 55.88

1. If i use 1 min bars, is it true that the OHCL of the next series of bars may not be 55.60?

2. I'm concern even if entry is filled, the stop or profit target ma nor be filled cause backtest only use OHCL data points.

3. I can change my strategy so my entry is lowest of next three 1 minute bars, but what about target in stop loss?

Sorry, just bit confused on how to use OHCL for my strategy even if using one minute bar cause there is still possible that calculated stop or profit target will not be filled

For example, profit target 55.88 could occur on a bar of low 55.80 an high 55.92. But my calculated target 55.88 never fills cause back test only has OHCL data.

Please tell me I am completely wrong and loss and this is not the case. I would be happy.

Thanks for your help as this make me think all day
Thanks

[NinjaTrader_AlanP]

Hello simple_goodoboy,

Thank you for your note.

Regarding you first question, If the previous bar closed at 55.58, its possible the high of current bar could be 55.59 and thus not trade 55.60, so yes.
Regarding your third question, it is not possible to have your entry set to the lowest of the next three bars, as the low of a bar isn’t known until that bar closes at which point that price may no longer be available.

I would suggest the following links for becoming better familiar with backtesting. I would also encourage you to use Market Replay and test your strategy and certain assumptions as a means of better understanding strategy creation and deployment.

Please see Discrepancies: Real-Time vs Backtest section of our helpguide.

http://ninjatrader.com/support/helpGuides/nt7/?discrepancies_real_time_vs_bac.htm

Adding a secondary data series of a smaller time duration would allow your strategy to enter/exit intra bar and give you results closer to market replay.

Please see Backtesting NinjaScript Strategies with an intrabar granularity for a reference example on how you’d set this up.

http://ninjatrader.com/support/forum/showthread.php?t=6652

Market Replay;

http://ninjatrader.com/support/helpGuides/nt7/?market_replay.htm

Please let us know if you need further assistance.

[simple_goodoboy]

Hello simple_goodoboy,

Thank you for your note.

Regarding you first question, If the previous bar closed at 55.58, its possible the high of current bar could be 55.59 and thus not trade 55.60, so yes.
Regarding your third question, it is not possible to have your entry set to the lowest of the next three bars, as the low of a bar isn’t known until that bar closes at which point that price may no longer be available.

I would suggest the following links for becoming better familiar with backtesting. I would also encourage you to use Market Replay and test your strategy and certain assumptions as a means of better understanding strategy creation and deployment.

Please see Discrepancies: Real-Time vs Backtest section of our helpguide.

http://ninjatrader.com/support/helpGuides/nt7/?discrepancies_real_time_vs_bac.htm

Adding a secondary data series of a smaller time duration would allow your strategy to enter/exit intra bar and give you results closer to market replay.

Please see Backtesting NinjaScript Strategies with an intrabar granularity for a reference example on how you’d set this up.

http://ninjatrader.com/support/forum/showthread.php?t=6652

Market Replay;

http://ninjatrader.com/support/helpGuides/nt7/?market_replay.htm

Please let us know if you need further assistance.

AlanP,

Thanks for your effort in response.

It will really help me if I give an example and questions.

Examples are for 3 min bars and I am running an Strategy Analyzer back test with the default OHLC data only. Lets assume a perfect world for the orders actually get filled for the examples below as well.

Example 1:
Current_Bar execute signal for long on next bar (Bar1).
Limit Buy Order = 55.52
Bar1 Open = 55.60
Bar1 High = 55.63
Bar1 Close = 55.57
Bar1 Low = 55.48

For example 1
1. Will the buy order be filled at 55.52?
2. Will the buy order be filled at 55.48?
3. If the answer to question 1 and 2 is No, then what price will the buy order be filled at?

Example 2:
Assuming buy order in example 1 is filled.
Profit target set at 55.72 with limit sell order
Time pass on and Bar20 is created with the following data points:
Bar20 Open = 55.68
Bar20 High = 55.75
Bar20 Close = 55.66
Bar20 Low = 55.63

For example
1. Will the sell order be filled at 55.72?
2. Will the sell order be filled at 55.75?
3. If the answer to question 1 and 2 is No, then what price will the sell order be filled at?

May I please have your answers for my understanding please?

Thank you. You may call me as well.

[NinjaTrader_AlanP]

Hello simple_goodoboy,

Both limit orders under your scenarios would be filled at their limit price.

Example 1:
55.52

Example 2:
55.72

Please let us know if you need assistance.

[simple_goodoboy]

Hello simple_goodoboy,

Both limit orders under your scenarios would be filled at their limit price.

Example 1:
55.52

Example 2:
55.72

Please let us know if you need assistance.

Thank you AlanP,

I been waiting for this answer. Thanks, but I am confused now!

I was thinking the answer would be Example 1 = 55.48 and Example 2 = 55.75.

Is it true that during back testing using the Strategy Analyzer that only the OHLC are available? If so 55.52 and 55.72 are prices not within the bars OHLC examples I mentioned.

I am confused. How was 55.52 and 55.72 filled?

[bltdavid]

Thank you AlanP,

I been waiting for this answer. Thanks, but I am confused now!

I was thinking the answer would be Example 1 = 55.48 and Example 2 = 55.75.

Is it true that during back testing using the Strategy Analyzer that only the OHLC are available? If so 55.52 and 55.72 are prices not within the bars OHLC examples I mentioned.

I am confused. How was 55.52 and 55.72 filled?

Yes, you are greatly confused.

The 4 prices for the bar's OHLC are not established in a vacuum by magic.

Just like in real time data, where prices fluctuate and move at the will of the buyers and sellers, the strategy analyzer mimics this same fluctuating behavior as much as possible.

Take your example #1. You have a buy limit order at 55.52 (which by itself, says stop the presses, the answer to what is the entry price is ... drum roll ... 55.52 -- Why? Because that's the definition of a buy limit order.)

So, how does this back-testing thing work, you ask?
[This is where your confusion lies]

Well, NinjaTrader strategy analyzer code knows each 3-minute bar consists of 3 separate 1-minute historical bars, which each have their own OHLC prices. NT figures out the 4 aggregate OHLC prices of your 3-minute bar from the 12 constituent OHLC prices of each set of three 1-minute bars.

Btw, NT can actually assign a timestamp for trade entry on a 1 minute granularity -- because it examines each 1-minute constituent bar for the first bar that dips in price to at least 55.52.

"Granularity" is important concept that will come up later when you become more expert at backtesting. Ok, let's continue.

If you're thinking that the only prices available for an entry price are the 4 OHLC prices of your 3-minute bar, this is completely wrong. Like I said, the back testing code is smart and tries to emulate the real world fluid movement of prices, such that it can predict which of the 3 internal 1-minute bars for each 3-minute bar that the entry order actually executed on.

Also, I betcha Alan punted on your question, like I first did, because when you're dealing with buy and sell limit orders, the execution price of the order is, by definition, the limit price or better. Think carefully, you're in depth question setup was rendered moot because you said limit orders -- make sure you understand why we rolled our eyes and Alan returned short, quick, simple answers.

Thus, I glossed over your question, because it hardly mattered what you were asking, Alan's short and simple answer is entirely correct -- because you were asking about limit orders.

I think you need to understand when bars are formed in the strategy analyzer, yes, the OHLC values come from historical data -- but they (aka, the artificially created bars re-assembled by the internal NinjaTrader.exe code) are still assumed to have been fluid, just like in real time.

That is, if a 1-minute historical bar has Open=55.50 and Close=55.40, thus price is falling, and you had a buy limit at 55.45 -- well, you can very safely assume that the NinjaTrader software is smart enough to know that every price point from the Open price down to the Close price is touched (duh! think about it!) and thus your buy limit order will naturally, of course, (how can it not be?), absolutely, without question, be executed at the limit price 55.45 ... inside that bar.

The back testing software emulates the fluid drop in price inside that bar, just like it would have happened in real time. Sure, price could be all over the place during that actual live one minute period for that 1-minute bar, but we know (aka, we can very safely assume) that the Open=55.50 and Close=55.40 means every single price point between these two probably had some trade activity. (I say probably because in a rapidly moving volatile market, movement of price can actually jump over one or more prices -- this happens if buyers or sellers at those prices suddenly disappear).

Does this help?

[bltdavid]

Studying these pages may help you a lot,

How Bars are Built

Discrepancies: Real-Time vs Backtest

[simple_goodoboy]

Thank you soo much for your time and effort in explaining. Everything makes sense now.

Yes, you are greatly confused.
If you're thinking that the only prices available for an entry price are the 4 OHLC prices of your 3-minute bar, this is completely wrong.

Yes, this is exactly what I thought and I instantly got concerned that my back testing would be inaccurate, and now I need to go buy tick data. Myself and friend was confused about this.

My confusion was from this statement "During a backtest you can select conservative or liberal fill algorithms which will produce different results. Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest."http://ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

My initial thoughts was so the only data available during a back test is OHLC prices literally. However, I was not thinking about how NT strategy analyzer code uses the OHLC data to strategically create the bar (my cause time base bars) and try to match the back test code entry with the bar.

Also, I betcha Alan punted on your question, like I first did, because when you're dealing with buy and sell limit orders, the execution price of the order is, by definition, the limit price or better. Think carefully, you're in depth question setup was rendered moot because you said limit orders -- make sure you understand why we rolled our eyes and Alan returned short, quick, simple answers.

Yes, I can understand the confusion in my question/example. The example was for the case of back test assumptions. Yes, in live data the limit order will fill at 55.52 based on market dynamics.

lol, you rolled your eyes because the question was deemed simple if talking about live data, but I was referring to using strategy analyzer and the statement "Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest."

The back testing software emulates the fluid drop in price inside that bar, just like it would have happened in real time. Sure, price could be all over the place during that actual live one minute period for that 1-minute bar, but we know (aka, we can very safely assume) that the Open=55.50 and Close=55.40 means every single price point between these two probably had some trade activity. (I say probably because in a rapidly moving volatile market, movement of price can actually jump over one or more prices -- this happens if buyers or sellers at those prices suddenly disappear).

The statement is bold could be the reason why live and back test results could be different. Because the Strategy Analyzer is code to fill the bar based on OHLC, so volume or volatile market causing gap in prices is not technical used during the strategy analyzer code, so this is what causes the discrepancies between live brokerage account, live market simulation, Market Replay and Back testing.

So why do traders buy tick data instead of just using Strategy Analyzer back testing algorithm, which is free?

Thank you

[simple_goodoboy]

Studying these pages may help you a lot,

How Bars are Built

Discrepancies: Real-Time vs Backtest

Thank you. I am studying in detail.

There is statements around the internet that states Ninja Trader back testing function (Strategy Analyzer) does not include the intra bar tick data, which could cause inaccurate loss or wins.

But from what I am learning, this is not totally true because the strategy analyzer back testing function (fill code algorithmic, etc) uses the OHLC data to mimic the intra bar tick data and can fill order to the trader strategy needs. And this depends on the bar type as well.

Am I correct in my understanding? I believe I am.

I will proceed to continue learning the strategy builder and test some strategies. And experiment myself.

[simple_goodoboy]

I seriously doubt it.

Why?

#1 - you're only using 3-minute bars, which are created just fine from NinjaTrader's own 1-minute historical data, which is already plentiful and free from NinjaTrader (assuming you have a brokerage account to connect to NinjaTrader, of course).

#2 - use Historical Data Manager to download Minute data for the date ranges you need.

#3 - if you were sophisticated enough with your strategy testing needs, you'd probably already know the answer, and wouldn't be asking here.

#4 - but, if you're more a newbie, the answer gravitates more towards "the free historical minute data from NT's historical data servers is almost assuredly enough" which means the answer you're looking for is: "no, you do not need to buy extra data".

Thank you so much for the help
Yes, historical minute data is enough for what a newbie like me plan to do which is nothing complex at all.

This information helpage alot. Thanks

[bltdavid]

So why do traders buy tick data instead of just using Strategy Analyzer back testing algorithm, which is free?

Well, if you're using Tick bars, then of course you need tick data for your back testing, but you may also need tick data if you're using a 1-tick secondary data series for extra granularity for some advanced order management techniques.

Anyways, with NinjaTrader, most historical data downloaded into your chart (or for back testing) does not come from your data feed provider but instead comes from NinjaTrader's own historical data servers.

My point is: your data feed provider such as CQG, Continuum, or Rithmic, is for live data only -- these feeds don't supply historical data (they could, I suppose, but they don't).

There's a reason for that: historical takes up a lot of space. Well, there's only 3 kinds, tick, minute, and day, -- and minute and day are not that much -- but, OMG, the tick data takes up a huge amount of space compared to the other two!

[EDIT: There's a bit of sarcasm there, because HDDs and SDDs are so big and yet so cheap, tick data of size 1GB or 10GB or 100GB hardly breaks a sweat on most newer PCs -- but bandwidth for downloading, both speed and costs -- well, that is an entirely different story.]

So, why do people buy it?

Because, like I said in a prior post, NinjaTrader's own historical data servers only have the most recent 1 year of tick data for all instruments. That means, presumably, they are continually dropping old tick data across the board as it ages past 12 months.

So, if you want more than 12 months of back testing using historical tick data, you'll need to get it from a source other than NinjaTrader.

And those sources charge for it.

[bltdavid]

So why do traders buy tick data instead of just using Strategy Analyzer back testing algorithm, which is free?

No, they are used together.

Having 20 years of historical tick data is useless without some kind of back testing program, like NinjaTrader's builtin Strategy Analyzer.

You can buy historical tick data from third parties, convert it to NinjaTrader's format, and have Strategy Analyzer use it.

If you're new to NinjaTrader and need more than 1 year of tick data for back testing, you're a bit screwed, yes -- so that's why traders might decide to buy more historical tick data.

If you've been using NT for many years, presumably you were smart all those years and enabled the check box "Save chart data as historical" and/or used Historical Data Manager to download the tick data before the NT admins retired it.