Beta 8 and Interactive Brokers

[marcel-nl]

I have several questions

• Beta 8 has been modified to use the latest version of the TWS-software. However NT-7 is still on a lower level of the TWS-software. Is there a way to remain for NT-7 on the lower level of TWS and use the latest version of TWS for NT-8
• I have been able to use NT-8 with the lower version of TWS as used for NT-7. Is this supported
• There are performance issues with the data load which have been confirmed by testing in beta 6. Are these performance issues supposed to be solved with beta 8. And does it need the latest version of TWS

Note: Beta 8 looks much better, and is using space much more efficiently

[NinjaTrader_Matthew]

I have several questions

• Beta 8 has been modified to use the latest version of the TWS-software. However NT-7 is still on a lower level of the TWS-software. Is there a way to remain for NT-7 on the lower level of TWS and use the latest version of TWS for NT-8
• I have been able to use NT-8 with the lower version of TWS as used for NT-7. Is this supported
• There are performance issues with the data load which have been confirmed by testing in beta 6. Are these performance issues supposed to be solved with beta 8. And does it need the latest version of TWS

Note: Beta 8 looks much better, and is using space much more efficiently

1. Yes, you can use a lower version if you wish for TWS. If you do so, you will not be able to use the 'Auto logon' feature within NinjaTrader 8. Alternatively, you could just install the IB Gateway 954 and use that for NinjaTrader 8, and remain on TWS 944 for NinjaTrader 7. You will find each software version on the respective NinjaTrader connection guides for 7 and 8:

http://ninjatrader.com/ConnectionGuides/Interactive-Brokers-Connection-Guide

2. The only issue we're aware of with using a lower version of TWS with NinjaTrader 8 is that you will not be able to use the 'Auto logon' feature-> You must first start and login to TWS. If TWS is started, but will not connect, you may need to adjust the TWS Title bar text as outlined in this post: http://ninjatrader.com/support/forum...46&postcount=6

3. There are no changes to the data load rules in Beta 8. If you are referring to the amount of time it takes to return large data request, we have no immediate plans during beta to change these data load rules due to the pacing policies used by the provider. We may revisit this in the future as it's highly demanded, but will take some conceptual changes that we just cannot accommodate right now.

Thanks for your questions, I hope my answers are helpful.

[marcel-nl]

1.

3. There are no changes to the data load rules in Beta 8. If you are referring to the amount of time it takes to return large data request, we have no immediate plans during beta to change these data load rules due to the pacing policies used by the provider. We may revisit this in the future as it's highly demanded, but will take some conceptual changes that we just cannot accommodate right now.

Thanks for your questions, I hope my answers are helpful.

Thanks for your extensive reply. I am a bit concerned about point 3. I would like to stress that currently it is not workable and I would not move to NT 8, and I am even requesting little history with my data requests. I hope I should read that having no immediate plans during Beta, means that it will be solved before NT 8 will be released as general available.

[NinjaTrader_Matthew]

Thanks for your extensive reply. I am a bit concerned about point 3. I would like to stress that currently it is not workable and I would not move to NT 8, and I am even requesting little history with my data requests. I hope I should read that having no immediate plans during Beta, means that it will be solved before NT 8 will be released as general available.

Can you give an example of the type of requests you're making?

e.g., AAPL, 1 Day Period, 365 days to load, takes N minutes/seconds to load.

I will be happy to compare what you're seeing to our QA benchmarks just to ensure we're talking about the same situation.

[marcel-nl]

Can you give an example of the type of requests you're making?

e.g., AAPL, 1 Day Period, 365 days to load, takes N minutes/seconds to load.

I will be happy to compare what you're seeing to our QA benchmarks just to ensure we're talking about the same situation.

I do not have these figures at hand. However, Chris Jameson has tested on my computer before and he has collected all the data. We are talking about the same issue. And I have sent data about my system with case number1412096

[NinjaTrader_Matthew]

I do not have these figures at hand. However, Chris Jameson has tested on my computer before and he has collected all the data. We are talking about the same issue. And I have sent data about my system with case number1412096

Thank you for posting the case number, that is the same issue I had in mind.

There will be no changes to the way data is loaded in this scenario, as a result, we will expect the same results and behavior in the early production releases of NinjaTrader 8.

[fpa75]

3. There are no changes to the data load rules in Beta 8. If you are referring to the amount of time it takes to return large data request, we have no immediate plans during beta to change these data load rules due to the pacing policies used by the provider. We may revisit this in the future as it's highly demanded, but will take some conceptual changes that we just cannot accommodate right now.

I also think that NT should properly handle the "pacing violation" of IB, as some of its competitors do very nicely (such as MC). I don't remember the thread where it was discussed, but some say this being useless because of inaccuracy of IB data (due to the merging of several ticks). I agree with this, but when you are a beginning trader, you really appreciate the ability to use IB as a data feed. In the beginning, you don't know exactly how you will trade, what market, etc etc etc. and this level of accuracy becomes important later (I use IQFeed). Plus, as it was stated in this thread, IB as data feed can be useful because most of the futures data feed vendors do not include european stocks, unless you go to eSignal that may probably be too expensive for many, especially at the beginning. I believe this feature would really be useful, especially to new traders, even if they switch to another data feed later when they know better their requirements for their style of trading.

Regards,
F.

PS: and while we talk about IB as data feed, it would also be an important feature to implement their ability to correct in (almost) real time the OHLC of minute bars. This feature is called "5 seconds bars" and provided in the IB API by the reqRealTimeBars() function. It has been available since at least 2012. This feature is implemented by some of NT's competitors (such as SC), and the result is that minutes charts they display in real time are exactly the same that minutes charts made by TWS.

[zweistein]

ib adapter needs improvement

I totally agree with fpa75,

the IB feed should be improved. It is better in B8 than in B7, but please

- add proper handling of the pacing violation
- reduce reload of market data to a minimum and store older data always locally in the NT database.
- after a disconnect/reconnect cycle (a 1 minute disconnection for exmple) only the few missing data should be reloaded and the NT code should be optimised to make for a quick restauration of trade operation possibility.
(A classic example is a short internet connection at fast market conditions. NT will take ages to recover , so the only cure is to follow up positions directly from within IB TWS which always shows extreme quick recovery and handles those errors very nicely.


Andreas

P.S. I don' t mind merged tick data as mentioned. For me as a trader it is much more important to have the latest prices and no data lag. This is where IB excels, Maybe for bandwith problems you would not get all tick data, but you will always get the latest. This is important when in fast market!

[marcel-nl]

I also think that NT should properly handle the "pacing violation" of IB, as some of its competitors do very nicely (such as MC). I don't remember the thread where it was discussed, but some say this being useless because of inaccuracy of IB data (due to the merging of several ticks). I agree with this, but when you are a beginning trader, you really appreciate the ability to use IB as a data feed. In the beginning, you don't know exactly how you will trade, what market, etc etc etc. and this level of accuracy becomes important later (I use IQFeed). Plus, as it was stated in this thread, IB as data feed can be useful because most of the futures data feed vendors do not include european stocks, unless you go to eSignal that may probably be too expensive for many, especially at the beginning. I believe this feature would really be useful, especially to new traders, even if they switch to another data feed later when they know better their requirements for their style of trading.

Regards,
F.

PS: and while we talk about IB as data feed, it would also be an important feature to implement their ability to correct in (almost) real time the OHLC of minute bars. This feature is called "5 seconds bars" and provided in the IB API by the reqRealTimeBars() function. It has been available since at least 2012. This feature is implemented by some of NT's competitors (such as SC), and the result is that minutes charts they display in real time are exactly the same that minutes charts made by TWS.

I have noticed in the past (NT7) that sometimes the 1-minute bars are a bit different from the ones displayed during the session when when I do a reload of the charts. I would be interested in an explanation of the 'limitations' of the IB-data feed and what would be solved in NT8.

[zweistein]

loading.. very slow

Hello,

the problem with Data Loading with IB as broker still persists in beta 8.

To be clear, when I load charts within the TWS interface then they show up almost immediately. So the market data connection is good.


As I deduce from communications with the NinjaTrader support team the problem is due to a too simple data loading logic. It seems that NT8 simply does a full reload of all market data for all open charts. This is nonsense as we do have older data on disk in the NinjaTrader 8\db directory.

As a consequence of this stupid data loading logic there will always be problems with imported data and data loaded from different market data providers. Basically you will not be able to use such data properly when connected to your main broker.

As an example: I have a weekly chart open for the GC 04-16 (Nymex Gold future) . A weekly chart will easily go back to 2005. You can load such data using the free Kinetick end of day connection.

So now we have all that data on our hard disk.
Then connect using IB as a broker ( or any other broker , problem will be the same).
On connection the above mentioned stupid logic will try to reload all data . As IB has only market data back a few years and not until 2005 there will be a lot of IB data load errors for the GC Nymex future contracts that are outside the IB market data range. Each error will delay data loading and data loading will become extremly slow.
I have measured 15-30 minutes data loading times.
This is completely unaceptable as is is unnecesary to load all the older data.

I ask you kindly to fix the mentioned problem and :

1. Implement an intelligent data loading logic minimizing data load.

2. Add an option "Automatic market data history backload" and set this by default to 1 week or 1 month. So in automatic mode NT8 will only reload data less than 1 week back and all older data is taken only from disk.

3. Add an API function to NinjaTrader that allows to intercept the marketdata loading. The Idea is that a user can implement its own logic to loading data.

For example
MarketDataRequest OnMarketDataRequest(symbol, period){

if (period longer than 1 week) return MarketData.LoadFromDb;

return MarketData.LoadFromBroker;

}




Attached you will find a sample workspace.

To all users, please could you report loading times for this workspace

Best regards

Andreas

[Eric M.]

Hello Andreas,

I downloaded your sample workspace and using I.B. data exclusively (no loading using the free Kinetick end of day connection), it took approximately 60 minutes to load all charts.

I'm also experiencing very long loading times when using I.B. data - My workspace consists of one futures contract of 3m, 5m and 15m timeframes and I find loading of chart data painfully slow.
I'm finding even with 3 charts, it takes several minutes to load !!

With my NT7 workspace, I have up to 20 charts on a multi-monitor setup and it loads considerably faster.

I'm hoping that the slow loading of I.B. data in NT8 will be addressed in a future update.

Regards,
Eric

[zweistein]

Hello Eric,

thank you for reporting. Just for the records, NinjaTrader support in an email just said "we didn't find any problem..."


The workspace is not even full of charts.

There are a total of 10 charts. 5 charts are minute bars (600) going back about 3 days.
The other 5 charts are weekly charts (800 bars) .

So in total we are talking about a really small amount of data


Regards

Andreas