Very Slow Reload of Historical Data

[BarzTrading]

I have loaded the full history of ES ##-## 15 minute bars from Kinetick. That takes several minutes. This seems unnecessarily long to me, but that's not my primary question.

Then I change the interval to Day. It quickly reloads the daily data.

Then I change back to 15 minute bars and it takes several minutes again to load the data. When I look at my system, the internet bandwidth is running at several megabits for several minutes. It's clearly going back to the server.

Why? I was JUST viewing 15 minute bars?

It's a Sunday. The end date on my DataSeries is yesterday, a habit so as to try to avoid any overhead from real time updates.

Might this have anything to do with the "Use Backadjusted Data" on the Connections settings? I don't actually understand what that setting is supposed to control.

[NinjaTrader_ChrisL]

Hello BarzTrading,

Thank you for the post.

I was able to import a continuous contact into NinjaTrader 8 and the data was loading normally.

Please take the below steps to clear the NinjaTrader cache:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

The continuous contract is no longer supported in NinjaTrader 8 by default, you can only import a historical continuous contract data from a .txt file. When you pull up an ES 12-18 chart do you get the same results? Can you create a new continuous chart with no indicators applied to it?

The merge policy will tell NinjaTrader how to organize data between rollovers. Each setting is described on this page.

I look forward to your reply.

[BarzTrading]

Thanks for your reply, Chris.

OK, deleting the cache folder didn't change anything. But I've done several more experiments.

I have only one workspace and it has only one chart window. Using NinjaTrader 8.

---

STUDY #1 - Using ##-## Syntax (e.g. ES ##-##)


Data

6J ##-## (1 Minute bars)
Oldest Available Date: 11/22/2005
If the start date of the data series is 1/2/2007 or older, then we get this pattern. We'll call it Pattern A:

• about 15 second delay
• followed by 15 seconds to download ~60MB of data
• about 45 second delay before chart displayed

If the start date is 1/3/2007 or newer, then we get this pattern. We'll call it Pattern B:

• 5-15 second delay and chart is displayed
  • There is no big data download, obviously using local data or cache

ES ##-## (1 MInute bars)
Oldest Available Date: 9/6/2005
If start date of data series is 1/2/2007 or older, then we get Pattern A.
If start date of data series is 1/3/2007 or newer, then we get Pattern B.

CL ##-## (1 MInute bars)
Oldest Available Date: 9/19/2006
If start date of data series is 1/1/2007 or older, then we get Pattern A.
If start date of data series is 1/2/2007 or newer, then we get Pattern B.

ZL ##-## (1 Minute bars)
Oldest Available Date: 4/20/2007
If start date of data series is 4/20/2007 or older, then we get Pattern A.
If start date of data series is 4/21/2007 or newer, then we get Pattern B.

Conclusions

It appears that 1/1/2007 is some kind of hard cutoff date for NinjaTrader for symbols having
data that goes further back than 1/1/2007.

Secondly, it appears that the oldest available bar also represents a cutoff even if that bar
is newer than 2007.

So, there are two types of cutoffs. If the start date is older than those cutoffs, then a full
reload of the data is initiated and takes almost 10 times as long, even when a full reload
is not actually needed.

Questions:

1. Can you confirm this on your side?
2. Why impose these cutoffs?
3. The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.

Discussion: Why is this a problem?

The problem arises when dealing with multiple symbols. One doesn't want to have to
hard code the specific limits for every symbol to make sure that we don't trigger long
delays with Pattern A.

It would be nice to just apply a start date of say, 1/1/1950 and then NinjaTrader would
be smart enough to say, "OK. We have all of the available data and the cache is fresh
so we don't need to trigger a full download."

Now, you might think that you could just impose a limit of say 4/1/2007, but that won't work.
As you see above for ZL, since it doesn't go back that far, and many symbols don't, it will
trigger a full reload of the data. In fact, some symbols only have a few years of minute data.

Sure, I could figure out what all the limits are and be careful to only set appropriate start
dates to ensure full reloads don't happen for my given set of symbols. But why should I have to?

Shouldn't NinjaTrader be able to do the smart thing here? If the start date is older than the
oldest available bar for a given symbol and all of that data is local and fresh, do not refetch
it from the server.

This would lead to a dramatic improvement for minute studies such as I'm doing across a wide
spectrum of symbols going back as far as possible.

---

STUDY #2 - Merge Policy Settings and Specific Month Syntax (e.g. ES 12-18)


Data

1) MergePolicy: Do Not Merge

ES 12-18 (1 Minute bars)
Oldest Available Date: 9/19/2017
Always loads within 5-10 seconds regardless of start date. There appears to be some difference if the
start date precedes the oldest bar, but it's hard to tell because the data downloads so quickly on
my system. There is some small bandwidth usage for 1-2 seconds with every request.

2) MergePolicy: Merge Non-backadjusted

ES 12-18 (1 MInute bars)
Oldest Available Date: 12/12/2007
In this case, all minute data will redisplay in about 10 seconds or less regardless of the start date
of the data series. The date can be 1/1/1950 or 2/23/2010. Appears to be pulling from the local
cache and not hitting the server.

3) MergePolicy: Merge back adjusted

ES 12-18 (1 MInute bars)
Oldest Available Date: 12/12/2007
Reloads the chart within 15 seconds without pulling data from the server regardless of the start date.


Conclusions


It appears NinjaTrader and continuous contracts are designed to work better when using a specific
expiration month (ES 12-18) rather than this syntax (ES ##-##). Caching behavior seems to be
essentially as one would expect for the different merge policy settings.


Questions:

1. Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?
2. Is the specific month syntax preferred or recommended? If so, why?

Discussion: Why this (might) still be a problem?

For better performance it appears I could just be sure to use a merge policy of back-adjusted
along with specific expiration months. However, this is still a problem when doing studies with
multiple symbols. We have to list all of the specific expiration months for every symbol.

Now, it appears one could created an Instrument List with all of the specific expiration months for
each symbol. And then use Tools->Database Management -> Rollover to regularly advance the
expiration months in the instrument list.

This might work fine for my purposes, but it would be nice to have something like the ##-##
syntax that would just always choose the front month. Well, this does appear to work except
for the slow and unnecessary re-downloading of data.



Thanks,
Steve

[NinjaTrader_ChrisL]

Hello

Thank you for the reply.

Study 1.

Can you confirm this on your side?

I did confirm that it takes a bit longer to load data when you cross over a specific date in a given instrument with a continuous contract.

Why impose these cutoffs?

I will ask our product management team if this is expected.

The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.

What script are you inquiring on? Could you upload a script or post some snippets if this is your own script?

Study 2.

Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?

I will need to ask our product management team this question as well.

Is the specific month syntax preferred or recommended? If so, why?

The merge policy will organize the data for you automatically in NinjaTrader 8. It is a personal preference whether to use continuous contracts or not. It does seem that specific contracts load faster than continuous contracts. Note that the amount of historical data on your chart will not affect the real-time performance once loaded.

I look forward to being of further. assistance.

[BarzTrading]

Hi Chris,

Let's ignore the issue about "the code seems to be off by one bar" for right now. It's minor and only confuses the issue. I think once your team understands how data caching behaves differently when using "ES ##-##" in the chart verses "ES 12-18" back adjusted that this issue may become irrelevant.

Regarding the last question on whether the specific month syntax is recommended, my question may not have been clear or one of us has a different definition for what a continuous contract is.

Let's get clear on our terms.

A continuous contract results when data from more than one individual contract is used to create a series. Anytime a merge is done, the result is a continuous contract.
An individual contract is a single futures contract defined by a symbol and a specific expiration month and year. There is no merging.

If I enter "ES ##-##" in a chart I will always see a back adjusted continuous contract no matter what the merge policy is.

If I enter "ES 12-18" what I see depends on the Merge Policy. If the merge policy is "Do not merge" then I will only see the individual contract data for ES 12-18. If the merge policy is "merge back adjusted" I will see a back adjusted continuous contract. If merge policy is "merge non back adjusted" then I will see a continuous contract where the prices have not been altered from the underlying individual contract that they were pulled from.

So my question is not about whether to prefer continuous contracts over individual contracts. The question is whether typing the letters "ES ##-##" versus "ES 12-18" with back adjusted merge policy is preferred when you want to view a back adjusted continuous contract. In theory, these two things are exactly the same, only the symbol syntax differs. Well, and the caching behavior which I've outlined above.

I look forward to hearing back from the product team on why caching behavior differs for the two different symbol syntaxes:

1. ES ##-## minute bars (merge policy is ignored here). Does not use the local cache when the Start Date for the series is older than the available data OR older than 1/1/2007.
2. ES 12-18 minute bars (merge back-adjusted). In theory exactly the same series as #1. NinjaTrader DOES use the local cache as one would expect regardless of what the Start Date for the series is set to.

[NinjaTrader_PatrickH]

Hello BarzTrading,

Thank you for your response.

To put things simply; The current contract with the Merge Policy set to Back Adjusted or Non-Back Adjusted is altered on the local installation and the actual continuous front month (using ##-##) will result in pulling that data directly from the provider as already merged and Back Adjusted.

So if I am using the current front month (12-18 for example) and i use a Merge Policy that merges prior contracts together then I only need to download each individual contract needed. However, if I am requesting the data as merged and adjusted from the data provider (such as using ##-## with Kinetick) then I am requesting the entire data set again.

With that said, I see the exact same behavior as you do when requesting a front month versus the continuous contract month from Kinetick. In addition, while I do not know the exact circumstances for the server download request for Kinetick I suspect that not many people are re-requesting the same data multiple times a day where the data is nearly over a decade just to compare download speeds for such a large data set.

So it all breaks down to what you want in the end. Do you want the merge and adjusted (or not) data by the local installation? Then request the front month with the appropriate Merge Policy. Or do you want the merged and adjusted data from the provider directly? Then request the ##-## front month.

Please let me know if you have any questions.

[BarzTrading]

Thanks for the confirmation, Patrick. I can work with that.

[Lanticalt]

Hello, I was wondering how to map IB DX index and IB DX futures. The instrument mapping get realy confusing when one has to pick the exchange. I tried Nymex, default, ice, globex, ib combinations of them but still cannot get ^DX or DX data from IB. I suscribed to DX at IB. There got to be a way to get my historical data. NT8 keep returning an error: errors requesting bars series. IB was unable to locate instrument... Of course I can't figure out how exactly this ticker need to be map.

Regards,

[NinjaTrader_BrandonH]

Hello Lanticalt,

Thanks for your notes.

The DX futures instrument in NinjaTrader is part of the ICE Futures US market data subscription seen in the NinjaTrader Client Dashboard > Settings > Plans > Market Data section.

DX futures instrument is mapped in the Tools > Instruments window by default. See the attached screenshots showing this.

Does the issue persist if you use the settings symbol map settings seen in the screenshots and change the 'Merge Policy' setting to 'Merge Back Adjusted'?