• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Interactive Brokers slow data load

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Interactive Brokers slow data load

    All NT8 beta versions have this problem:

    Data loading is extremly slow:
    Example 2 charts, FDAX 09-16 1 min and FDAX 09-16 weekly, each chart 1000 bars.

    When connecting to IB data loading takes several minutes (something between 5 and 10 minutes) .
    In comparison NT7 takes less than 3 minutes.
    Data loading from within IB TWS is immediate (less than 5 seconds)


    Conclusion: NT8 beta 12 is not useable for trading yet.

    Looks like it is time to look for software alternatives.

    @NT : Are you silently dropping support for IB?

    Best regards

    Andreas

    #2
    Hello,

    Thank you for your post.

    I am not aware of Interactive Brokers being dropped as a supported provider/brokerage.

    So I may look into this further please go to Help->Email Support and be sure to include Attn: Chris Jameson in the Subject line, a link to this thread in the body and a check in the box for "Log and trace files".
    Christopher J.NinjaTrader Customer Service

    Comment


      #3
      Originally posted by NinjaTrader_ChristopherJ View Post
      Hello,

      Thank you for your post.

      I am not aware of Interactive Brokers being dropped as a supported provider/brokerage.

      So I may look into this further please go to Help->Email Support and be sure to include Attn: Chris Jameson in the Subject line, a link to this thread in the body and a check in the box for "Log and trace files".
      The earlier response from NT was that this issue would not be solved before release of NT8. It would be fixed in an early version after go Live. I would also like to remember that there is also an outstanding and confirmed issue with data spikes with the IB data feed which did not occur in NT7. So I am also concerned with the level of support regarding IB.

      Comment


        #4
        Do you happen to have a link to the thread that covered this or a ticket number with support you can provide?
        Christopher J.NinjaTrader Customer Service

        Comment


          #5
          http://ninjatrader.com/support/forum...ight=marcel-nl

          http://ninjatrader.com/support/forum...ight=marcel-nl

          Comment


            #6
            As a test can you please start NinjaTrader in Safe Mode.
            Safe Mode will prevent NinjaTrader from:
            • Loading workspaces.
            • Connecting on start-up.
            • Loading custom assemblies.
            • Getting instrument updates from the server.

            To enable safe mode, please use the following steps:
            • Exit NinjaTrader.
            • Hold the CONTROL key and double click the NinjaTrader icon.
            • Keep the CONTROL key held down until you see the Control Center.
            • You can verify you are in safe mode by going to Help -> About.


            Next connect to IB and let me know if you see a faster load time.
            Christopher J.NinjaTrader Customer Service

            Comment


              #7
              Originally posted by NinjaTrader_ChristopherJ View Post
              As a test can you please start NinjaTrader in Safe Mode.
              Safe Mode will prevent NinjaTrader from:
              • Loading workspaces.
              • Connecting on start-up.
              • Loading custom assemblies.
              • Getting instrument updates from the server.

              To enable safe mode, please use the following steps:
              • Exit NinjaTrader.
              • Hold the CONTROL key and double click the NinjaTrader icon.
              • Keep the CONTROL key held down until you see the Control Center.
              • You can verify you are in safe mode by going to Help -> About.


              Next connect to IB and let me know if you see a faster load time.
              In the past the issue has been investigated. NT connected to my platform and have done own testing. The reaction was that there need to be a redesign of the IB connector and there was currently no capacity for it. I have stopped testing NT8 waiting for the solution (and solving the issue with data spikes , which is also a show stopper for me).

              Comment


                #8
                Hello Andreas,

                I think that in NT8 historical data request always seems to get both Bid and Ask from IB.
                Whereas in NT7 it only got one or the other.

                I think this means twice as many IB API historicalData() callbacks.
                If these IB callbacks are slow this might double the time to load the same period.
                Also there's probably twice as many calls to write to the NT database.

                Comment


                  #9
                  very slow IB data load

                  OK, so I have 2 charts open. Each chart has 700 Data bars.

                  Instrument is FDAX 09-16.

                  One is a 1 Minute chart, the other is a weekly chart.

                  The 1 minute chart loads in less than 5 minutes, still much slower than IB TWS charting which is almost immediate.

                  The weekly chart is still loading, and more than 50 minutes have passed now... Unuseable.

                  It seems obvious that whenever there is a bar loading problem ( bar not in data base, etc) then NT8 unwinds all the exceptions generated. But this unwinding the exceptions is much too slow, it cannot take more than 45 minutes for 700 bars!

                  Also, once the data is loaded there is no need to reload everything on a reconnect.


                  P.S. Have send you workspaces etc. in the past already on the same problem in early NT8 beta versions.


                  Andreas

                  Comment


                    #10
                    Originally posted by zweistein View Post
                    OK, so I have 2 charts open. Each chart has 700 Data bars.

                    Instrument is FDAX 09-16.

                    One is a 1 Minute chart, the other is a weekly chart.

                    The 1 minute chart loads in less than 5 minutes, still much slower than IB TWS charting which is almost immediate.

                    The weekly chart is still loading, and more than 50 minutes have passed now... Unuseable.

                    It seems obvious that whenever there is a bar loading problem ( bar not in data base, etc) then NT8 unwinds all the exceptions generated. But this unwinding the exceptions is much too slow, it cannot take more than 45 minutes for 700 bars!

                    Also, once the data is loaded there is no need to reload everything on a reconnect.


                    P.S. Have send you workspaces etc. in the past already on the same problem in early NT8 beta versions.


                    Andreas
                    700 bars of weekly data would be more than 10 years of historical data which may run into problems with Interactive Broker's pacing violation.

                    I would like to look into your diagnostic files to see if this is what is occurring,

                    Please send me your log and trace files so that I may look into what occurred.

                    You can do this by going to the Control Center-> Help-> Mail to Platform Support.

                    Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

                    Please reference the following ticket number in the body of the email: 1543111 ATTN Patrick G
                    Patrick G.NinjaTrader Customer Service

                    Comment


                      #11
                      Please prepare me a simple workspace with FDAX 09-16 with 2 charts, 1 min and weekly with each 700 bars.
                      Then test and if you cannot find slow data load then please post the workspace here so that me and others can cross check your findings.

                      As soon as I have a workspace from you I will report my findings.
                      I want a workspace from you so that there can be no misunderstanding .



                      Best regards

                      Andreas

                      Comment


                        #12
                        Originally posted by zweistein View Post
                        Please prepare me a simple workspace with FDAX 09-16 with 2 charts, 1 min and weekly with each 700 bars.
                        Then test and if you cannot find slow data load then please post the workspace here so that me and others can cross check your findings.

                        As soon as I have a workspace from you I will report my findings.
                        I want a workspace from you so that there can be no misunderstanding .

                        Best regards

                        Andreas
                        I believe the long load time might be expected due to the current behavior of the IB API and NinjaTrader. IB only allows 60 historical data chunks to be requested every 10 minutes. A chunk of historical data roughly equates to 2 days of minute data. The IB adapter attempts to queue chunk requests at a rate of 5 seconds per request. This is in an attempt to prevent hitting the pacing violation. With this information, it is easy to see why large historical requests can take time when using IB.

                        I experience long load times as well. The minute bar loads relatively quickly. The 700 bar weekly chart is over 10 years of data and did not load in the time I was able to test. Providing a workspace would not get us anywhere at this point.

                        I would like to see, in detail, what Interactive Brokers is reporting in your NinjaTrader trace file so I can match it up with the information from my testing.

                        Please send me your log and trace files so that I may look into what occurred.

                        You can do this by going to the Control Center-> Help-> Mail to Platform Support.

                        Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

                        Please reference the following ticket number in the body of the email: 1543111 ATTN Patrick G
                        Patrick G.NinjaTrader Customer Service

                        Comment


                          #13
                          well,

                          send me the workspace and I will then test it.

                          when I disconnect and reconnect to IB , then there is at maximum 3 minutes of data lost.
                          Loading time is then also extremly slow.

                          So, please post a workspace here that other users can then give you feedback also

                          Andreas

                          Comment


                            #14
                            You need to switch to Kinetick. I use IB as my broker but the data feed into Ninjatrader is just so slow and unusable. I think its their way of trying to get people to use TWS instead of Ninjatrader. Just go with Kinetick for charting data and then use TWS to place trades...

                            Comment


                              #15
                              Originally posted by zweistein View Post
                              well,

                              send me the workspace and I will then test it.

                              when I disconnect and reconnect to IB , then there is at maximum 3 minutes of data lost.
                              Loading time is then also extremly slow.

                              So, please post a workspace here that other users can then give you feedback also

                              Andreas
                              My workspace is not performing any better than your workspace when I recreate it as you described in your earlier post. Me providing a workspace would not be a useful tool for testing.

                              As Pete Collins mentioned, the IB API has certain restrictions that other providers do not.

                              Please consider the information I've shared regarding the IB API and send me the requested information.
                              Patrick G.NinjaTrader Customer Service

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by stasha, Today, 12:49 PM
                              0 responses
                              11 views
                              0 likes
                              Last Post stasha
                              by stasha
                               
                              Started by denalek2419, Today, 10:25 AM
                              3 responses
                              26 views
                              0 likes
                              Last Post denalek2419  
                              Started by jrp2099, Today, 11:20 AM
                              0 responses
                              2 views
                              0 likes
                              Last Post jrp2099
                              by jrp2099
                               
                              Started by jonbennett, Yesterday, 04:01 PM
                              1 response
                              16 views
                              0 likes
                              Last Post NinjaTrader_ShawnB  
                              Started by jonbennett, Today, 09:43 AM
                              0 responses
                              2 views
                              0 likes
                              Last Post jonbennett  
                              Working...
                              X