Automation / Connection problem for NT8 after (auto restart) of TWS

[PeterGolz]

Hi all,

i'm working on automation for a stable service environment for my strategies.

Following Applications / Versions are in place:
NT8 (8.0.20.1)
TWS from IB / Captader (978.1h)


1.) First i tried/checked auto restart for TWS by it self. It is still working good. After a quick shotdown of around 20-30sec. TWS is reconnected to IB and up and running fine, without any issues!!!

2.) Second, i connected from NT8 to running TWS (started manually or by NT8 makes no differnce), works also without any issue!

BUT, if time for (auto restart) for TWS is comming and is processing this task, connect from NT8 to TWS is complete gone and cannot automatically reestablished!!!!
NT8 provides an error message, that shows that connection to TWS was lost and that this problem is a big issue. (PANIC) (see screenshot)


After 20-30sec. when TWS is up and running again you can click by hand / manual to establish a new connection from NT8 to TWS.
Those an approach seems very unprofessional for me... :-(


Also it brings up a big, big issue, for my strategies...
With strategy-setting (Wait until flat) in combination with a lost connection to my account comes in place, strategy will not know right current position for the strategy and wait until all positions would be closed by hand.
This seems not practical at all... :-(


How does any body handle this system behavoir and also have some practical informations to make this combination of applications/services working professional / stable over mind. one week?


Hope some one can help me out... :-)


With best regards,
Peter Golz

[NinjaTrader_ChrisL]

Hi Peter, thanks for your post.

Firstly, you should install the supported version of TWS, please uninstall the version you have now and install the version listed on this page:

https://ninjatrader.com/ConnectionGuides/Interactive-Brokers-Connection-Guide

TWS 973.2 - Download

We have tested this version and confirmed it will re-connect after a disconnect. Please try this out and let me know if your inquiry is not resolved.

Best regards,

-ChrisL

[PeterGolz]

Hi Chris,

thanks for your feedback!
Unfortanly in this very old version of TWS (973.2) function (auto restart) of TWS was NOT implemend. (Tried this first out - It was a mess...)
Also in this old version there was/is no way of proffesional reconnection without any third party tools (like IBC).

That is the reason why IB includes this function directly into TWS to garantee the highest way of processing (auto restart) and stability!!!

Also IB advice clearly to use TWS (>= 976) in way of security! That is a real contradiction, that NT should clear...



How do other customers handle this system combination?
You really have to restart (TWS) / reconnect (NT8) to TWS every monrning? (I can't believe that...)



With best regards,
Peter Golz

[NinjaTrader_ChrisL]

Hi Peter, thanks for your reply.

I must ask why this auto restart is needed. Is your NT connection dropping every day from TWS? It would be worth it to try the supported version and just run your strategy normally, I have seen users doing this no problem. The specific TWS version is needed because that is the version of the API NinjaTrader uses to connect. If there is some functionality in a non-NinjaTrader-supported TWS version this will likely not work with NinjaTrader.

[PeterGolz]

Hi Chris,

thanks for your Feedbaxk!
I'm not really sure, do you know TWS for real?
It needs a connection reset to IB every night!!!!
This is also needed by your recommended TWS-Version (973)!

Just with the small drawack, that TWS hangs up and need a (restart) process by third party tools like IBC etc. (https://github.com/IbcAlpha/IBC)

Citation:
Here are some of the things it will do for you:

• It can keep TWS running indefinitely if required, by dealing with the autologoff mechanism (however this mechanism no longer works for TWS 974 and later version because Interactive Brokers have changed TWS functionality to prevent it)
• It allows TWS to be shut down at a specified time on a specified day of the week (again, this functionality no longer applies for TWS 974 and later versions) !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
• It allows the Gateway to be shut down at a specified time every day

I would like to repeat my question:
How do other customers handle this system combination?
You really have to restart (TWS) / reconnect (NT8) to TWS every monrning? (I can't believe that...)


Looking for up comming suitable hints... 8-)


With best regards,
Peter Golz

[NT-Roland]

Hi,
IB interrupts the connection every day for a short moment, but TWS 976 or greater allow "Auto restart".
File -> Global Configuration -> Lock and Exit, check the box Auto Logoff Timer. Save your settings in TWS !!!
Auto restart works for up to one week, IF AND TO THE EXTENT the disconnect is triggered by IB/TWS, not if you close the TWS by yourself.
In the former days, the TWS itself didn't have the "auto restart" feature, which explains why many use(d) the IB Gateway although it had less other features.

@PeterGolz: You might want to pay utmost attention to the "syncing account positions" section of the help guide .

https://ninjatrader.com/support/helpGuides/nt8/syncing_account_positions.htm?zoom_highlightsub=Wait+until+flat

I would assume, "Wait until flat" is not the best choice in the scenario you described.
Kind regards
NT-Roland

[PeterGolz]

Hi NT-Roland,

thanks for your reply!
With your statement i think i came a littlebit closer to a solutions... :-)

But some things should be clarified for me:
- Do you advise / think to use IB-Gateway instead of TWS?
- Is there not the same issue / behavior with connection interruption from IB Gateway to IB (day by day) that cause the same error message in NT that i have already provided?
(https://ninjatrader.com/support/foru...hotoid=1084128)
- Do you use Tools like IBC etc. in any kind of automation?
- Is it possible for you, that NT8 automaticly re-establish his connection to IB-Gateway or TWS after daily connection interruption from IB?
- Do you have to do any manually task - day by day - to reconnect NT8 to IB-Gateway or TWS?
- May you speak german, to fight with that issues in native langue? ;-)


With my current investigations i think there are just 2 things be optimized to get this work in way of proffesional automation! (Without downgrading none modern version of Gateway/TWS)
1.) Error message from NT8 must be fixed or suppressed
2.) If connection to IB (Gateway/TWS) is lost, what error messages show up, try of reconnection shoud be started

Doing this by hand, fix the problem at all!!!
1.) Close error message by clicking button (OK)
2.) Open Connection-Center on NT8
3.) Click on connection-string for IB (Captrader) to trigger a reconnection
4.) Connection is directly re-establish


I just can't believe that sombody do this day by day in way of profesional system-trading!!!



With best regards,
Peter Golz

[NinjaTrader_ChrisL]

Hi Peter, thanks for your reply.

I will need to test out our connection while running a strategy to see what kind of behavior I get. I will also ask my supervisor what he thinks about the situation.

Thanks in advance for your patience.

[NinjaTrader_Jim]

Hello Peter,

It is a known situation where TWS will force a disconnection. Many users report that IB Gateway does not exhibit this behavior, so we may suggest using IB Gateway instead to keep the connection running.

NinjaTrader currently cannot automatically reconnect in the event of a broker side disconnection and also would not be able to automatically re-enable strategies. We are tracking interest in providing that sort of functionality in the feature request ticket ID SFT-2395. I have added a vote on your behalf. Note that these are internal tracking numbers and we cannot offer an ETA for implementation. You may check the Release Notes page of the Help Guide to see if a feature has been added in a new release of NinjaTrader which would satisfy the functionality requested here.

Release Notes - https://ninjatrader.com/support/help...ease_notes.htm

Using the ImmediatelySubmit Start Behavior will have the strategy attempt to match historical orders calculated when the strategy is enabled and processes historical data with the realtime orders it has submitted a previous time it was enabled. If the strategy matches the historically calculated orders with the live orders, the strategy will resume managing these orders. I recommend considering to use this start behavior when you want to have the strategy resume a previous position.

https://ninjatrader.com/support/helpGuides/nt8/index.html?syncing_account_positions.htm#ImmediatelySubmit

Please let us know if you have any additional questions.

[NT-Roland]

Hi Jim,
Quote: "NinjaTrader currently cannot automatically reconnect in the event of a broker side disconnection and also would not be able to automatically re-enable strategies."
Please add my vote for ticket ID SFT-2395. This is a "MUST" have before NinjaTrader can be used for automated trading for those not wanting to sit to in front of their computers all day long. And auto-re-enabled strategies should take back up exactly as they stood before, i.e. each of them with the FULL trade history in the performance log, which right now, isn't the case, at least not for me.

No issue in none of these respects with my other piece of software. Thus, it can be done.

@PeterGolz: If and to the extent, I use NinjaTrader for real time trading, I combine NT8 and IB Gateway 9.72.
You cannot use the IB Gateway and the TWS for the same account in parallel, unless you apply a little trick. Send me a message if you want to learn more, and feel free to do so in German / gerne auch in Deutsch :-).

NT-Roland

[messi]

Hi NT-Team,
I have the same problem as Peter has. I totaly agree with Peter that the current situation is not acceptable and does not allow professional trading with NT and IB. Of course I passed the issue to NT support (Ryan S and others), and not only one time, but the answer was that it is a TWS problem. IB does not answer to the issue at all. So what to do as paying and now pissed off customer, who payed NT for license and IB for the datafeed? Change the broker, kick off NT, make the problem public in social media warning other traders before they get trapped? Maybe, but what is the reality? NT and IB are partners, taking benefits from each other. So it should be a basic interest of NT to get the problem solved and also IB will get more fees if the customers basic needs are adressed. If IB does not wan't to support the issue, what ever may be the reason for that or even more problematic - is not able to solve the problem. So NT please - quit working with them! Do not advertise on your homepage for IB as approved broker, simply because your NT customers will be disappointed after they start trading with NT/IB. Dissatisfied customers, NT does not want to have - right? I am looking forward to get a acceptable solution on this item very soon, since i dont want to waste any more time with such basic problems. There are other working solutions for trading out there, so it is possible to automate trading without NT. There are alternatives, and they will be taken - for shure!

[reubenlasky]

I had struggled with this TWS issue for the longest time, until I switched to IB Gateway. Still every night at around 9:30PM Pacific Standard Time, it will disconnect and re-connects by itself. Otherwise it is good and does not affect my autotrades. With CQG connections, there is no interruptions until Friday NY close. I am just thankful I have two grandfathered NT licenses which I can trade via CQG.

[NinjaTrader_Jim]

Hello all,

I have added additional votes on your behalf.

I have also tested leaving a connection to IB Gateway running overnight with a strategy enabled. I had set Connection Loss Handling in the Control Center's Tools > Options > Strategies menu to Keep Running. I did experience some connection losses, but NinjaTrader has not experienced the same "server side disconnects" where it will not re-establish the connection. My test strategy stayed running.

I would recommend anyone using IB Gateway with an automated strategy to switch Connection Loss Handling to Keep Running to keep their strategies running.

I look forward to assisting.

[Rob2011]

Dear All,

I am having the same challenge and I agree with Peter and Messi, this should be solved!!

From NT side the connection should be checked and if disconnected reconnect automatically (like the steps Peter and myself are doing manually (including sync. accounts/position))

@Jim, you describe the situation correct, but your solution is a poor workaround. The strategy " keeps running" but does not know it receives no data -> nothing happens, even not an error or alert.
I also saw that my strategy was not sync anymore and did not react in changes in position.

How can we Autotrade when this is not firmly secured?

By the way for me it is still "simulated trading" but without this solved I will not invest in NT8 and have to look for alternatives.
(which would be a pity as I really like the strategy C# abilities)

Looking forward to a solution (yes add my 10 votes to SFT-2395)

[NinjaTrader_Jim]

Hello Rob2011,

To clarify here:

NinjaTrader does have built in functionality to automatically reconnect. This case is in regards to forceful disconnects where NT does not attempt to reattempt connection. This would be seen with weekend server maintenance and with server side disconnections as seen with TWS.

When I had tested with IB Gateway the other day, I only received connection losses of 1 second. The strategy stayed running. As long is the strategy is the only thing operating on that account/instrument, it will stay in sync with the account. Other strategies or order manual order submissions to the same account/instrument can cause a desync. If you need to take further action when the strategy sees a connection loss, you can override OnConnectionStatusUpdate.

OnConnectionStatusUpdate - https://ninjatrader.com/support/help...atusupdate.htm

You can also configure email alerts to alert you if there has been a connection loss to your broker/data provider. You would first set up an Email Share Service, and then set Tools > Options > General > "Email log alert messages" to the email address you would like to have emails sent to.

Nonetheless, I have added a vote on your behalf.

Please let me know if there are any specifics that may still need to be addressed.