Why does NT6 download new data when I already have history in the DB?

[zoltran]

Nt6 has a long pause/hang when connecting to either IB or Opentick.
I already have valid historical data, but it seems like NT wants to download and overlay this .

Both IB and OpenTick are slow backfillers, so this action for NT is a real pain.
And it can and does lockup NT from time to time.. especially if I get impatient and try opening another chart. This starts another backfill .. and that often locks NT up and I have to kill it via XP TaskMgr

I've attached a MarkAnalyzer template that will help show the problem.
Nothing special, just 3 columns with a 5//15/60 minute CCI
And I only have about 8 instruments in it.

Try this ..

Open 3 charts .. say YM, ER2, and 6E (euro)
Add this Mkt Analyzer

Connect to Opentick ...

IF NT survives this (50/50 for me) then disconnect and reconnect OpenTick
You get the same wait again.
Infact, you get wait just changing periods .. from 5 to 10 to 1 minutes for example.
Which is bad, as the minute data is already in the DB.

Change the symbol on a chart while it's got the 'data loading'
for the Mkt Analyzer .. usually good for a hangup.


I don't get any hangs if NT is not connected to a feed. It uses the historical data in the DB as expected.

[NinjaTrader_Ray]

imported post

I will check to see if there is anying unusual from the set up you experienced. A few things:

- Please make sure you are on the latest beta 9 as there were changes to internal bar data management
- Any time you open a chart, NT will take your bars back value and check if there is historical data in the database on that day. If it is empty, it will request the entire period up to today of data. If this day falls on a Saturday where the dbase is sure to be empty, then it will always re-request data. Assuming there is data there, it will only request the current days data. This must be done since you need to have the most current data. OpenTick and IB can be extremely slow, especially OpenTick. I have seen it backfill on 15 days for over an hour, I have seen it hang. This is their current level of service. They are working hard on improving it but thats what you get for afree service.

Ray

[zoltran]

imported post

I have attached a screenshot with steps to recreate the NT hang on backfill.

The trick to making it hang seems to be to adjust the period of a chart .. After it has backfilled, but before NT has finished backfilling all the symbols in the mkt analyzer

Yes ,, this is Beta 9

[NinjaTrader_Ray]

imported post

Zoltran,

Thanks for the detailed step by step walkthrough. I can reproduce and will forward to development.

Also, I will respond to your CCI post later on once I get clarification on a few items.

Ray

[zoltran]

imported post

Thank you.

Could you also inform dev't that NT is generally unstable anytime the 'Loading Data' dialogue is displayed. I've had several hangs other than the one I detailed below, just from trying to do 'things' while a backfill was proceeding elsewhere.

By "things" .. I mean modify a chart, change period, load a template elsewhere ...
At one time I thought it was when another dialog had to be opened .. but I can't be more specific.

fwiw,
Machine is XP-MediaCtr, SP2 , AMDx2, 3G Ram, tons of disk.

[NinjaTrader_Dierk]

imported post

The lock up problem when toggling chart intervals while NT is loading data will be fixed with next beta/release. Thanks for pointing that out.

[zoltran]

imported post

Another test case for you.

Disconnect from esignal
Create 3 charts
Create 3 Market Analysrs using the 3cci template posted previously
mktAnlz 1 - 1 - default list
mktAnlz 2 dow30
mktAnlz 3 nasd 100

Connect to esignal

Wait ... Wait Wait ... it still takes a loooonnnnngggg time, even with esig

And eventually NT will crash.

OR at least it has the last 3 times I've tried it.

[NinjaTrader_Ray]

imported post

I could not reproduce. Here is what I did step for step.

- Start NT
- Open 3 charts ES, ER2 and NQ, all 1 min with 15 days back
- Open 3 MA's with your template
- Default instrument list, DOW and NAZ lists
-Connected to eSignal
- Yes, it did take a long time to load and I will look into possibilities to improve this but after it was all loaded, it did not crash

See anything that I may have done differently than you? Also, next time it happens, is there error message with a details button? If yes, please press and copy the text and email it over.

Thanks

Ray

[zoltran]

imported post

I will try again later this weekend

[Antraman]

imported post

>I will try again later this weekend

Hi Zoltran. How did it go. I have a similar problem and if you resolved it I would like to know how. thx.

Antraman

[NinjaTrader_Ray]

imported post

We did improve historical data download for Market Analyzer with Beta 10. The problem prior to Beta 10 was that there were redudant requests for download taking place.

Ray

[zoltran]

imported post

Hi Ray

Sorry to say, NT6 Beta 10 still has some problems in this regard.

For example, I currently have a 5 min chart, with a Format Data Series back to early 2004. This was orginally opened using Opentick .. I wanted to see how far back I could get data.

This morning, I connected to Esignal .. and NT tried to backfill the whole 3 years again .. taking about 20 to 30 mins to do so.

I'm not sure of the logic being used.
Most programs I am familiar with only backfill the gaps and do not even refresh the current day. This is usually a very-good-thing, especially with provides like IB that have differing methods for r/T vs historical (.2 sec snaps vs 2 sec snaps)

To my thinking NT should only fill the most recent hole from the last existing quote in the DB to now. This would cover 'most' situations. And we should have a simple way to 'request' a longer backfill from the chart for those situatations the call for it.

[NinjaTrader_Ray]

imported post

Yes, eSignal can be slow on minute bar requests.

The current logic backfills based on the first day of the historical data set. If there is data in the database, it will only load the current day, if there is no data, it will load the entire set again.

The way to resolve this particular case is to change the lookback period to fall ona day where data exists in the database. I also suggest for long lookback periods not to have the start data fall on a Saturday where there will never be historical data.

Ray

[zoltran]

imported post

I"m not sure it's working as you say Ray.

It always seems to backfill, regardless of state or starting date.
For example, I just tested with charts that started last Monday.

Load NT, no connect - charts display fine.. the data is there
Start esig .. blank-out while it backfills
Stop esig (not NT) .. charts are fine
Restart esig connection ... same blank-out/backfill

Does it on opentic and IB as far as I can tell.
Of course, with IB, it's pretty certain you'll hit their throttle with NT working like this.

[NinjaTrader_Ray]

imported post

In your example, is the last data (Format Data Series) set to today? If yes, then this is correct behaviour. NT will always load the current days data into the chart on start of a new connection if the last day is set to today.

Ray