eSignal volume zeroed out in NT7, but price updating OK

[steeltoe]

Two issues:

1) Because Ninjatrader 7 constantly forgets the historical data it has already requested from the servers, I am absolutely required to turn off "Get data from server (if available)" in Options. Otherwise, I need to wait 3-4 minutes every other minute, or just when changing colours in charts, to wait for Ninjatrader 7 to hammer servers with downloading data from 2010 I've already downloaded, again and again (stupid).

So I absolutely have to turn off this option, but I wouldn't mind a Ninjatrader using a proper database instead of hammering official servers unnecessary, increasing bandwidth, lag and costs for everyone involved.

2) Turning off "Get data from servers", however has unintended consequences: Esignal 10.6 keeps price ticking in OK, but volume is NOT OK. All new real-time volume is all zeroes, which is NOT OK.

This must be a glitch in the driver of eSignal, because price is being updated correctly. However, combined with the other issue, it becomes impossible to use Ninjatrader 7 for anything serious (you need at least 10 years of daily data and 2 years of real-time data to process meaningful analysis). Ninjatrader 7 is by design flawed in its off-handling of crucial data to a destructive cache, which is NOT a database by ANY definition mind you.

Lesson: A database is a storage that actually PRESERVES your data, not destroying or distorting it.

Please have a look at the issues. A resolution here would render eSignal + NT7 combo a superior choice to most other competitors / combinations. However, issues like these in NT7 must not be swept under the rug. People complain for good reasons.

Sincere hopes for resolution.

[NinjaTrader_Brett]

Hello,

Thanks for the note and welcome to our support forum.

I'm sorry to hear of the frustration.

Before moving on to the second part of your question would like to address this first:

1) Because Ninjatrader 7 constantly forgets the historical data it has already requested from the servers, I am absolutely required to turn off "Get data from server (if available)" in Options. Otherwise, I need to wait 3-4 minutes every other minute, or just when changing colours in charts, to wait for Ninjatrader 7 to hammer servers with downloading data from 2010 I've already downloaded, again and again (stupid).

Would like to offer some clarification here. There are specific data reload rules that when triggered will cause NinjaTrader to re-download data from the historical data server. Please see these data reload rules for how NinjaTrader determines that it does not have all the historical data that it needs to requests re-download. With these rules you can make some changes so that data reload does not occur. Other then turning off historical data all together.

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

and the following section:

When does NinjaTrader load historical data?


Let me know if you have further questions after reviewing this.

[steeltoe]

Hello,
Before moving on to the second part of your question would like to address this first:

Would like to offer some clarification here. There are specific data reload rules that when triggered will cause NinjaTrader to re-download data from the historical data server. Please see these data reload rules for how NinjaTrader determines that it does not have all the historical data that it needs to requests re-download. With these rules you can make some changes so that data reload does not occur. Other then turning off historical data all together.

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

and the following section:
When does NinjaTrader load historical data?
Let me know if you have further questions after reviewing this.

I have reviewed this option "End date" (are there more options we have to manually modify?). I do not understand this option, nor is it properly explained in help. I am a programmer by profession, but these rules are cryptic and the entire purpose of the field is obscured from the user. My lack of understanding is not due to lack of expertise, I would assess, but lack of proper howto's and explanations.

Ok, let's go into the details, and thank you for reviewing this with me. My hope is that improvements in NT would render these steps necessary, or a light bulb in my head about what's going on:

Rule #1 states: The End date parameter of the Data Series window contains the current day
The end date of the chart says: 10/3/2011 (current date). Is this wrong and why? It is the current day, so is that why the chart reloads 2010, 2007 etc., whenever I change colours or do anything with the chart? I tried changing it to 10/2/2011, but then the chart won't update current day, and next time I enter Data Series edit the field is back to 10/3/2011. I tried changing it to 10/4/2011, but it always reverts back to 10/3/2011 (current date) automatically after reloading all my old data yet again (chart does display correctly though, with same gap into the future as the future date difference to current date).

What is meant by the statement "End date contains the current day"? Will 10/5/2011 "contain" 10/3/2011?

Rule #2 states: The End date parameter of the Data Series window contains the last day of data available in your data repository

Why should it not? What is "End date" anyways? I tried changing it to an old date, like a 2009-date, but then the chart won't show anything at all. So I presume the field refers to the last date to show. Why should I have to reload old data just to show latest days?

Rule #3 states: The data repository contains no data within 3 days of the first day loaded in the chart
How do I check this practically? What steps to ensure correctness on each of these steps?


Why all this constant reloading of already loaded data? Just because I change the End date parameters, NT7 reloads all my data from 2010 and 2011 etc. from server every time.



How is whatever we do in the "End date" property supposed to mean anything when it always reverts back to current date automatically? Next time we change something and press OK, the field would of course then point to current date because of this, without user intervention or knowledge.


I hope my questions and confusion is understandable, and for remedy for such.

[NinjaTrader_Brett]

Hello,

1)

End date is the last day of data to load.

Most likely your doing this on futures therefor you have to account for the contract being different back then. Most likely the 12-10 contract.

To test try switching end date to yesterdays date, it should load the chart with data from yesterdays date to the begin date for the chart date. This chart however would not receive live data, as for a chart to be active and recieve live data the end date needs to be todays date.

2)

You can check data with control center->tools->historical data manager in the edit tab for what data is available.

3)

Data reload occurs with the data reload rules, also you may be talking about the auto merge function on futures. This must occur to get the offsets to display a correct chart with the auto merge. You can turn auto merge off to avoid this. Control Center->Tools->Options->Data Tab->Merge Policy->Do Not Merge.

4)

Bottom line however is that it is normal in NinjaTrader to get data reload of the entire chart when NinjaTrader determines that you may possibly be missing some data. There is no logic to only download the data you are missing and will reload all data. This is expected and happens in seconds on most users PC for most our users.



Let me know if you have further questions.

[steeltoe]

Hello,
1)
End date is the last day of data to load.

Ok, I fail to see the connection with constant reloading of already loaded data.

Most likely your doing this on futures therefor you have to account for the contract being different back then. Most likely the 12-10 contract.

This is on $OSEBX-OSL on eSignal remapped to _OSEBX_OSL, but the issue is applicable to any chart I've loaded in NT7, as this is a responsive issue from the start. The chart loads OK.

To test try switching end date to yesterdays date, it should load the chart with data from yesterdays date to the begin date for the chart date. This chart however would not receive live data, as for a chart to be active and recieve live data the end date needs to be todays date.

I did this in my previous reply. Yes, the chart loads with fewer data. However, you fail to answer why the "End date" property automatically reverts back to current date after pressing OK or APPLY in Data Series edit.

I'm also not sure what changing this parameter should accomplish, as I'm of course interested in realtime data, but I can confirm this works partially, with the issue mentioned in paragraph above.


2)

You can check data with control center->tools->historical data manager in the edit tab for what data is available.

Ok, good.

3)
Data reload occurs with the data reload rules, also you may be talking about the auto merge function on futures. This must occur to get the offsets to display a correct chart with the auto merge. You can turn auto merge off to avoid this. Control Center->Tools->Options->Data Tab->Merge Policy->Do Not Merge.

Not using futures, so merge should not be an issue here.

4)
Bottom line however is that it is normal in NinjaTrader to get data reload of the entire chart when NinjaTrader determines that you may possibly be missing some data. There is no logic to only download the data you are missing and will reload all data. This is expected and happens in seconds on most users PC for most our users.
Let me know if you have further questions.

It is normal to have to wait several minutes to load data from servers that is already loaded?

There IS logic to only load data you are missing, both in terms of cpu load, network load and response times.

After all, loaded data should NEVER change. Once loaded, they are set in stone, unless server / provider is changing data. Are they? Or you have missed some events. Maybe you'd want to reload last 3 days of data or similar, but not the ENTIRE data repository!

This should be user configurable.

I do not see logic in loading already loaded data automatically, thereby introducing horrible lag and inefficiencies, as mentioned above, which is what this issue is about really. Waiting several minutes for every change to chart, or whenever NT7 thinks something has changed, is unacceptable behaviour. Especially since the user is not to blame, and powerless to remedy the situation without turning off the broken automatic data loading in the first place.

I'm hoping for solutions that can remedy the problem, not explain away "this is how NT works", because this does not work. People will lose money waiting for NT7 to actually show anything and then witch over to more responsive charting software.

I'm providing very constructive feedback here, and hope it will be well received. Excuses don't remedy my and others' issues.


Lastly, problem of missing new real-time 5-min volumebars remains, even with "Get data from server (if available)" turned on. New price bars are coming in OK, but not volume bars. I need to reload (all data!) for it to show. This is for $OSEBX-OSL on eSignal, mapped to _OSEBX_OSL. I'd rather we focus on this issue.

Thank you for your understanding.

[NinjaTrader_Brett]

Hello,

Lets reset here and then I can let you know if expected or not expected and then forwarad feedback to development based on that.

Please pull up a 5 minute chart with 30 days back of data with no indicators. Just a plain chart then right click on it and select reload all historical data to force a reload on that chart.

What is the load time?

To further up on your question here:

After all, loaded data should NEVER change. Once loaded, they are set in stone, unless server / provider is changing data. Are they? Or you have missed some events. Maybe you'd want to reload last 3 days of data or similar, but not the ENTIRE data repository!

This is the reason why entire repository is reloaded. Data is changed on the server from time to time and also the time stamps on data are reloaded based on server time stamps which are more accurate then local PC time stamps with some providers. Also, you could miss ticks, ultimately it was worth it for our clients and us to reload all data to make sure all is correct as having a correct chart is important to us.

I look forward to assisting you further.

[steeltoe]

Ok, I did just that and timed it to 66 seconds on a blank chart (reload all data). This is on 4GB, 64-bit core7 Windows7 computer, which should be adequate for any reasonably heavy task.

This happens when changing colours, when reloading ninjascript, when doing _anything_, and the problem is aggravated with more charts than one, which combined with the modal dialogs, freezes parts of the NT platform since you cannot even edit code while waiting for the dialog to close and the loading to complete. Of course, this becomes very unproductive and useless, despite all the features and bells and whistles Ninjatrader provides.

However, I'd rather we focus on the issue I reported originally, which were repeated in last post:

"Lastly, problem of missing new real-time 5-min volumebars remains, even with "Get data from server (if available)" turned on. New price bars are coming in OK, but not volume bars. I need to reload (all data!) for it to show. This is for $OSEBX-OSL on eSignal, mapped to _OSEBX_OSL. I'd rather we focus on this issue."

Incoming volume bars are missing. Why?

[NinjaTrader_Brett]

Hello,

Ok thanks for timing that.

I do the same test on my side. 30 days back with no historical data and a force reload.

It takes 1 second ! on my side for the chart to download all data and come up and this is on a basic system.

We have a serious bottleneck somewhere on your system, can you please do an internet speed test? What is your internet speed?

Swinging back around to the question you asked:

"5-min volumebars" What is this? There is no time aspect in volume bars, is this a custom bar type that is not included with NinjaTrader by default?

Can you also clarify here:

"New price bars are coming in OK, but not volume bars."

Both charts no matter if it is a minute chart or a volume chart would be price bars. Specifically what is not showing? The volume indicator reporting volume bars or a volume chart is not displaying data at all?

Most likely the above could be related to the bottleneck somewhere on the system with loading speed that I suggest we focus our attention on first.


I look forward to assisting you further.

[steeltoe]

Hello,
Ok thanks for timing that.
I do the same test on my side. 30 days back with no historical data and a force reload.
It takes 1 second ! on my side for the chart to download all data and come up and this is on a basic system.
We have a serious bottleneck somewhere on your system, can you please do an internet speed test? What is your internet speed?

My cpu and network speeds are fast. It may be related to my needs of loading alot of data. Right now, I have 520 days (2 years) of 5min bars loading. For the daily chart I'm loading 5200 days (10 years).

I require this, because I'm using a higher timeframe (120 min) in the 5-min chart, and need enough data to process meaningful data. This is required since Ninjatrader does not support accessing other chart data (other timeframes), and the 120 min timeframe is built on top of the 5-min chart, thus alot of 5-min bars is required due to NT restrictions.

Why should this be a problem? It wouldn't be if Ninjatrader had sane defaults when loading new data.

However, turning off automatic data loading solves that issue for now. I can update manually.

Swinging back around to the question you asked:

"5-min volumebars" What is this? There is no time aspect in volume bars, is this a custom bar type that is not included with NinjaTrader by default?
Can you also clarify here:
"New price bars are coming in OK, but not volume bars."
Both charts no matter if it is a minute chart or a volume chart would be price bars. Specifically what is not showing? The volume indicator reporting volume bars or a volume chart is not displaying data at all?
Most likely the above could be related to the bottleneck somewhere on the system with loading speed that I suggest we focus our attention on first.
I look forward to assisting you further.

It is the same chart, just the volume information (volume bars on a price chart). It goes to 0 for new incoming realtime data on eSignal for $OSEBX-OSL (an index), but works OK when reloading or using STL-OSL (a stock). I'm remapping the instruments in Instrument Manager.

Why the incoming real-time volume is 0 in realtime but is OK when reloading data I don't know, but may be an issue with the eSignal's Ninjatrader driver most probably. This requires a programmatical fix.

[NinjaTrader_Brett]

We still have a bottleneck somewhere.

Please see this video I created for you with a test on my side with your settings.

http://screencast.com/t/B0Ol9awQM

Thanks for clarifying on your volume question:

Index is not a traded instrument, it is a mathematical formulation as is with almost all index, you will not have volume on an index in this case and we simply display the data as we receive it from eSignal.

Let me know if I can be of further assistance.

[steeltoe]

We still have a bottleneck somewhere.

Please see this video I created for you with a test on my side with your settings.

http://screencast.com/t/B0Ol9awQM

Thanks for clarifying. Very confidence-inspiring.

However, you just opened the chart. Did you try "Reload All Data" in the 5-min chart, and it's this fast with loading minute-bars? Are you sure your provider supports GE with the necessary amount of data? (I think eSignal's minutebars goes back to April 2011..)

Also, you opened a weekly chart. However, weekly charts in NT are based on daily data, and not 5-min data. I know this, because for a chart missing daily data but containing 5-min data, weekly data does not show.

Minute data for 2 years should be noticable to download. Probably the provider doesn't even have that much data available, but still it is slow on my end even with just data from April 2011 (see points below).

I've now done the following:
1) Repaired DB, Reset DB, Reset Instruments
2) Closed NT, renamed workspace file, Opened NT
3) Open _STL_OSL, Reload All Data: timed at 44 seconds, history shows minute bars going back from Oct 2011 to Apr 2011
4) Open _OSEBX_OSL, Reload All Data: timed at 39 seconds, history manager shows minute bars going back from Oct 2011 to Apr 2011
4) Open GE, Reload All Data: timed at 67 seconds, history manager shows minute bars going back from Oct 2011 to Apr 2011

Why it is loading data from 2010 while I reload, but not showing in historical data manager beats me.. But 2010 does flash down in Control Center status panel, when reloading the 5-min chart. Something is odd there.. Maybe just polling the server for non-existant data (which does take more time).

However, the 2011-data does takes the longest, almost on a month by month basis it is flashing in the Control Center status panel.

My specs:
Ninjatrader: 7.0.1000.7 (Current) (assuming this is the one and only version of this presented on ninjatrader.com)
Download speed: 2.99 Mbps
Upload speed: 0.56 Mbps
Ping: 21ms
Provider: eSignal 10.6.2425

Update on system specs:
Core i7 64-bit (using 32-bit NT) 8GB, nothing fancy installed in OS other than Avast Antivirus

Am using only 5-min charts, no tick, no volume or range charts.

I'm using eSignal as provider. May that be a difference between our setups?

I tried setting 155 in DataSeries "Days to Load". 2010 does not flash in Control Center status panel (quicker), however 5 months of minutebars in 2011 is still taking 22 seconds.

Grateful for suggestions and clarifications. Aim is anyways to be able to resolve the issues in any way applicable.

Thanks for clarifying on your volume question:
Index is not a traded instrument, it is a mathematical formulation as is with almost all index, you will not have volume on an index in this case and we simply display the data as we receive it from eSignal.

Let me know if I can be of further assistance.

Ok thanks. I've also put forward a ticket with eSignal. I just think it is strange volume is there when loading data, but not for incoming real-time data. Maybe I can find some workaround, or hopefully eSignal may resolve this issue.

One suggestion to resolve speed issues with downloading data is to use asynchronous data transfers (multiple connections).

However the best way to avoid delay, is to avoid too much downloading of data when data have already been downloaded for that day/week/userconfigurable timeperiod anyways.

[NinjaTrader_Brett]

Hello,

http://screencast.com/t/45x6m3ra

Please see this second video I made comparing eSignal download speed to kinetick download speed.

There was a noticeable difference. Therefor with your internet connection (Which is on the low side of high speed internet may be expected with eSignal in this case.)

However you would notice a huge increase in speed with Kinetick as shown in the video.

Have you considered checking out www.kinetick.com as this seems it would help with your data loading times.

As far as your comments on data loading, I will send this into development. Thank You.

[steeltoe]

Hello,

http://screencast.com/t/45x6m3ra

Please see this second video I made comparing eSignal download speed to kinetick download speed.

There was a noticeable difference. Therefor with your internet connection (Which is on the low side of high speed internet may be expected with eSignal in this case.)

However you would notice a huge increase in speed with Kinetick as shown in the video.

I'm wondering what kind of history you're getting. With eSignal, it seems capped at going back to April 2011 for minute bars.

How far does Kinetick minutebars go?


Of course, downloading more data than the provider's limit doesn't really provide any more information, while slowing down the queries. So I might as well cut down on the Days to load in DataSeries.

Your test seem to be about 2-3x quicker than mine, which is probably due to internet connection. However, although my connection is not in US and not the quickest one, 2.99 Mbps should be quite fast for most applications. Maybe for tick data, one should raise the bar more. Anyways, faster internet connection is unfortunately not an option for me.

However, this is taking too long to load on your side as well, just for you know, changing colours and refreshing Ninjascripts (as I've probably mentioned too many times now Having to wait 30-60 seconds for every change is of course a hopeless situation to be developing in. I thank you for putting this forward to the developing team.

Have you considered checking out www.kinetick.com as this seems it would help with your data loading times.

As far as your comments on data loading, I will send this into development. Thank You.

I have researched into ALL providers for Ninjatrader. The 2 top contenders for international use were IB and eSignal. Since my requirements are not US futures, but global stocks and indices ++, IB and eSignal were unmatched by most other providers and brokers. Kinetick did not meet requirements, as it was too US-centric, along with most of the other providers.

Since IB required huge deposits and deeper customer-broker relations, and eSignal provided also for Norwegian stocks (small index) as well as many many more options of global datastreams. It just had to be eSignal. I didn't need a broker, just a global datastream with broad market coverage. I don't think there are any viable alternative to this for us folks outside US, at least for Ninjatrader. Luckily, eSignal do provide the excellent features and options which are required for international customers.

With eSignal, a whole world of global stock, index and currency datastreams open up, compared with just US stocks and futures. Sadly, the superior option of eSignal datastreams is largely unsupported by Ninjatrader (have to manually remap every ticker symbol and other caveats). Kinetick may be simple, fast and great for US customers, however, impossible to work with with differing requirements for international customers.

Anyways, I see this loading-issue as a design-issue. Something which may be transparent for developers working side-by-side of the servers with small amounts of test-data, but which is very noticable for us outside US, with different providers than "recommended", and with a sizeable amount of data. I'm happy to provide any information I can. There are probably many others out there with similar data loading issues with NT7 (too many data-reloads during the day).

[NinjaTrader_Brett]

Hello,

Kinetick offers:

Extensive Historical Data

Access up to 120 days of tick data, two years of intraday minute data and ten plus years of daily data.






Thanks for the feedback on the load speeds. In this case this is expected, there is no resolution to the load times other then what you have found and has been discussed.



I will forward this feedback to development as well.


Let me know if I can be of further assistance.