The Definitive Guide to Historical Data

[bltdavid]

Looking at the Data By Provider web page, it shows that most every
data provider has all of 'Historical Data' columns checked as 'Yes'.

But some don't.

The exceptions are IB and TD Ameritrade. For ex, these do not provide
historical tick data. What does that mean, exactly? Does this mean
these folks cannot load up any tick-based charts with any historical
tick-based data? Are these customers hosed when using NT8?
(Of course not, NT has its own historical data servers that will
provide that historical tick data, right?)

-=o=-

So, my first question is, explain how the provider's historical data
availability and NT's own historical data servers are used to provide
the historical data we ultimately see and download to the db folder.

I mean, where is the historical data really coming from?
You or the provider?
What are the rules?

And how much historical data is available?
How does this vary between Tick/Minute/Day and Playback data?
Does every data provider providing historical data provide the
same amount of historical data?

[I know Historical data and Playback data are viewed differently
by the NinjaTrader product, but it seems easiest to think of Playback
data as just another category of Historical data, because so many
questions can be applied to both.]

-=o=-

The basic cause of my lack of understanding is this:
The Data by Provider web page fails to discuss how much historical data
is made available by each provider. Not even a hint. Not even a sense of
guaranteed minimums. Little mention of any common constraints. This makes
it very difficult to do a detailed comparison for the many different providers
in that chart.

For ex, more information on Kinetick historical data is here.

Btw, that thread contradicts what (I thought) I was previously told.
From my past, I could have sworn these held true,

• historical tick data - 1 year
• historical minute/day data - usually 10+ years
• playback data - 90 days

perhaps this was referring to NT7, or to NT's own historical data and
historical playback servers?

In addition to the historical data from a provider, how much historical data
(Tick/Minute/Day data) is available via NT's own historical data servers?

How does this compare to the historical Data by Provider?

If the answers are rather involved (because it depends upon the provider)
I am most interested in a thorough understanding of the current historical
data capacities/amounts/restraints for,

• Rithmic
• Continuum
• Kinetick

I'd really like a definitive understanding of where the historical data is
coming from, how NT's own historical data servers are used with the
Data by Provider historical data, and exactly how much historical data
is available, and does the amount of available historical data vary by
provider, and by how much.

I imagine this thread may become a long conversation. I have many
followup questions just waiting in the wings...

I'd like to make this thread 'The Definitive Guide to Historical Data'.
(I know, famous last words, right?)

[NinjaTrader_Spencer]

Hello bltdavid,

Thank you for your post.

First to clarify, only certain connections use the NinjaTrader Historical Data Servers. You may see this from the Log tab in the Control Center and will show as 'Using HDS'.

I have included an example below:

• https://www.screencast.com/t/8BPoCCwjVL

With that said TD Ameritrade and Interactive Brokers do not use NinjaTrader Historical Data Servers and their Connection API technology to 3rd party platforms (NinjaTrader in this instance) does not provide historical Tick data.

Connections that use the NinjaTrader Historical Data Servers are Rithmic, CQG based connections, and certain Forex connections when selected within the connection settings.

The amount of historical data available for connections that do no use the NinjaTrader Historical Data Servers would be dependent on the provider and you may reach out to them for more information.

The amount of data stored on the NinjaTrader Historical Data Servers is what you have outlined already:

• historical tick data - 1 year
• historical minute/day data - usually 10+ years
• playback data - 90 days - Futures and Forex instruments only

Note this amount may vary slightly depending on the specific instrument.

[bltdavid]

Great!

But that begs an obvious question:

For the Kinetick user,
How do you get access to that largish amount of historical data
from NT's servers vs the smallish amount of historical data from
Kinetick?

[NinjaTrader_Spencer]

Hello bltdavid,

Thank you for your post.

This could only be accomplished by connected to a data feed that uses the NinjaTrader Historical Data Servers.

If you are using multiple connections then you may use Preferred connections and set the connection that uses NinjaTrader Historical Data Servers as the preferred connection for Historical data:

• Preferred Connections

If you do not have an account with a connection that uses the NinjaTrader Historical Data Servers then another option would be to obtain the historical data from a 3rd party vendor and import the data into your platform manually:

• Importing Historical Data

[tradgrad]

NinjaTrader_Spencer so to clarify, Kinetick does not use the NinjaTrader Historical Data Servers...meaning the limits from the linked post apply?:

• 180 calendar days of historical tick data (for building tick based charts, such as range and volume)
• 2 years of historical minute data
• 10 years of historical daily data

Those are the limits for Kinetick and IQFeed data?

[NinjaTrader_Spencer]

Hello tradgrad,

Thank you for your post.

Yes, you are correct, Kinetick and IQFeed have their own historical data servers and have the mentioned availability.

[bltdavid]

Connections that use the NinjaTrader Historical Data Servers are Rithmic, CQG based connections, and certain Forex connections when selected within the connection settings.

How come this is not made clear in the Data by Provider help page?

The way that page is written, it sounds like NinjaTrader provides no
historical data at all -- I mean, that page implies historical data is a
function of the data provider only.

Why doesn't NT make this distinction more clear?

I'm suspicious, for ex, that in reality Rithmic & CQG don't even have
historical data servers, and that (for those 2 connections) NT's own
servers are responsible for all historical data. Is that true?

[NinjaTrader_Spencer]

Hello bltdavid,

Thank you for your post.

Correct, Rithmic and CQG do not provide historical data,

That is recorded and stored on the NinjaTrader Historical Data Servers.

For Forex connections, you may chose to use the Data Feed provider's HDS's or NinjaTrader Historical Data Servers.

For Stocks this is dependent on the data provider.

[bltdavid]

With regard to index data like ^TICK and ^TICKQ, good reading here.