TD Ameritrade Connection Issues

[nickslad]

Hello,

I wanted to create a thread collecting the various issues with the TD Ameritrade connection (as recommended by bertochi ). The connection was notoriously unreliable for dropping one’s order status in NT7 as to border on unusable (I built a strategy as a workaround), but unfortunately a number of issues persist in NT8, including the dropping of order status.

This is what I have noticed thus far since upgrading at the start of the year:

1) I have noticed repeated instances of the updating of my order status (broker TDA) getting dropped. This was a major issue in NT7 with the old TD API, to the point where I wrote my own custom strategy to handle the order status getting dropped. I have only noticed it occasionally so far, but when it does happen, NT will show the order status as being “submitted” (with no position being shown on screen), when in fact the order was completed and a position open. Therefore, I have to log-in to the broker to manage the position, cancel the order that is stuck on submitting and hope the status will refresh, or (potentially, haven’t tried yet?) close and relaunch the application to reset. This is my primary concern, and always an unwelcome surprise.

2) I’ve noticed that when changing instruments, a few seconds worth of price data will get dropped and not displayed. It does seem to actually be getting recorded, because if I switch back and forth between instruments yet again after a few seconds, that same missing stretch will fill back in.

3) I’ve noticed errors in the Trade Performance “Summary” calculations, after exiting and relaunching the application. My trades get double-counted, due to NT (presumably) recognizing something from the broker (TDA) and seeing “External” trades, which can counted on top of what NT already has recorded. This unfortunately makes the information in the “Summary” tab of little use.

4) Account P&L values (Gross P&L, Realized P&L, Unrealized, Total) correspond to some undetermined interval that doesn’t correspond to the current instance the application was opened. It’s impossible to know what interval the account P&L value actually represents.

5) Application on reboot will regularly fail to connect with TD (I have it set up to automatically connect to the TD Ameritrade connection), and require a restart. Not a big deal, but worth noting.

I have submitted a support request with log and trace files, but I do want to share this on the forums, as others have experienced the same issues and do need to be made aware. Effectively, the TDA connection wasn’t properly supported in the past on NT7, but users weren’t made aware of the fundamental issues with managing orders. There is still the same problem - albeit less severe - plus new bugs that weren’t present. My understanding from a previous support request was that the connection would have been ironed out for NT8.

[bertochi]

Hi @nickslad, thanks for taking the time to write all this down.
I see the same issues in your comment #1. Orders submitted in the first hour or two every single day, stay with the submitted status, sometimes changing to working, and fail to transition to "filled". This bug/issue goes away, as said around 11:30hs Eastern Time every day.
I have now also written to ([email protected]) to report this issue as a TDA customer, and I would suggest that all TDA customers do the same thing.
I have explained how I deal with this bug inside my strategies, basically triggering an alert if any order was not executed after x minutes. Once the error message pops up due to the alert triggered, I have another external program baby sitting NT, and waiting for an error window, and which will take care of it by restarting NT, upon restart I have "YET" another program to automatically enable all my strategies. In 2017, I requested the option to have strategies enabled by default but still this request did not make it to a new NT version.
I refuse to believe we are left in this situation, until a new TDA API, which we are being told is being developed, and which will hopefully resolve these issues. Being 100% honest, I do not see how a new tda api will resolve issues when the issue is not being addressed in the first place. Patrick from customer support has kindly told me he tried to replicate these issues at his end and did not find them, therefore he concluded that it might have been an isolated connection issue at my end. While I appreciated the response, I replied I doubted that the issue was isolated as I was not the only user having to deal with it, but also an issue that has a clear pattern of happening within the same time span every single day.
Hopefully all gets working as it should soon.

[nickslad]

I see the same issues in your comment #1. Orders submitted in the first hour or two every single day, stay with the submitted status, sometimes changing to working, and fail to transition to "filled". This bug/issue goes away, as said around 11:30hs Eastern Time every day.

This occurs for you every single day? That was the case for me while trading in NT7 with TDA. Right now, it is very much the exception rather than the rule for me, being a fairly rare, albeit regular occurrence. Maybe 1 in 10 or 15. This is pretty fundamental and really shouldn't happen at all though.

I have noticed that my order status errors I've logged are concentrated towards the open (almost all within first 5 minutes, save one) for what it's worth. In the past with NT7, what I experienced was far more frequent as what you have described.

Once the error message pops up due to the alert triggered, I have another external program baby sitting NT, and waiting for an error window, and which will take care of it by restarting NT, upon restart I have "YET" another program to automatically enable all my strategies. In 2017, I requested the option to have strategies enabled by default but still this request did not make it to a new NT version.

Not sure if this may be of any use, but have you tried using a "CancelOrder" command? With NT7, I used an unmanaged approach and had the strategy send a CancelOrder command if the status stalled - which would not execute because it was already filled, but would serve the purpose of forcing it to refresh. And when it refreshes, it comes back with the correct information. Because NT7 took forever to launch, I had to come up with a workaround that didn't involve restarting.

[bertochi]

Not sure if this may be of any use, but have you tried using a "CancelOrder" command? With NT7, I used an unmanaged approach and had the strategy send a CancelOrder command if the status stalled - which would not execute because it was already filled, but would serve the purpose of forcing it to refresh. And when it refreshes, it comes back with the correct information. Because NT7 took forever to launch, I had to come up with a workaround that didn't involve restarting.

Thanks for the tip. I will try experiment with the idea. Regards.

[dpolyakov]

I have been using TDA with latest NT for a little over a month. Here is what I've experienced so far:
1. Traded once right at open - the order get filled almost a minute later. After that I did not risk it again.
2. Market Watch window - I have 5 tabs. Each tab in average has around 8 stocks (some more, some less). So, it is frequently there is a random stock or two that is not updating in real time and not getting bid/ask feed (it just says n/a). Restarting NT helps sometimes but not always. If I force to reload historical data - it reloads data, but then no updates. Last price on chart will be the last cached price when I just opened NT, so what I see sometimes is a large bar from current open to cached last even after reloading data.
3. (minor for me, as I almost never use it) P/L reports are mostly correct, but I noticed that it also accounts cash deposits into account. So, there is a discrepancy.

Other than that I did not see any other issues.

Just to say: I have AT&T 1 GBit fiber, so speed is not an issue here.

[nickslad]

Hi dpolyakov - thanks for adding your experiences to the thread. Important to hear that others are seconding this issue with the order status refresh.

[bertochi]

Not sure if this may be of any use, but have you tried using a "CancelOrder" command? With NT7, I used an unmanaged approach and had the strategy send a CancelOrder command if the status stalled - which would not execute because it was already filled, but would serve the purpose of forcing it to refresh. And when it refreshes, it comes back with the correct information. Because NT7 took forever to launch, I had to come up with a workaround that didn't involve restarting.

nickslad I tried your tip this morning, and the order statuses changes to "pending cancel" and does not change from there. A restart and they are all updated as filled.

Maybe we would just need some supported or unsupported "syntax" to run inside our strategies to replicate what happens every time we restart NT. So until the APi runs as it should, whenever we see this issue, we could issue such syntax. It is just a suggestion.
Thanks

[nickslad]

bertochi I'm sorry that it didn't do the trick! I haven't yet ported my code over from NT7 into NT8, but I went back to review it (it's been awhile now) and didn't see anything else that I may have left out. I suppose I may have some extra work waiting for me when I tackle this. Thought I'd put it out there though in case it helped!

if (
( Position.MarketPosition == MarketPosition.Flat
&& entryOrder != null
&& entryOrder.OrderState == OrderState.Working
&& entryTime != 0
&& (ToTime(Time[0]) - entryTime) >= 000030
&& (ToTime(Time[0]) - entryTime) < 000130
)
||
( Position.MarketPosition != MarketPosition.Flat
&& entryOrder != null
&& entryOrder.OrderState == OrderState.PartFilled
&& entryTime != 0
&& (ToTime(Time[0]) - entryTime) >= 000030
&& (ToTime(Time[0]) - entryTime) < 000130
)
)
{

CancelOrder(entryOrder);

Print("Order status stalled at Working. Cancel order command sent. Duration in Bars: " + BarsSinceEntry(0, "", 0) + "Time in position: " + (ToTime(Time[0]) - entryTime) );
PlaySound( "Airbus_Stall.wav" );
if ( enableMail == true && Account.Name == //)
{ SendMail( strMailFrom, strMailTo1,
strStrategy, "Entry order status stalled at Working. Cancel order command sent. Duration in Bars: "
+ BarsSinceEntry(0, "", 0)
+ Position.MarketPosition + ". " );
if ( enableMailCopy1 )
{
SendMail( strMailFrom, strMailTo2,
strStrategy, "Entry order status stalled at Working. Cancel order command sent. Duration in Bars: "
+ BarsSinceEntry(0, "", 0)
+ Position.MarketPosition + ". ");
}

}
}

[bertochi]

@nickslad Please no need to be sorry. I very much appreciate your suggestion and follow up on this. In this particular case, it might be and it looks like that the TDA-API connection is somehow stalled (broken) at these particular times and not responding to NT, which is waiting for the updated order statuses.

A question for the experts would be if there is any "CBI.ACCOUNT code" as said on my previous post, supported or unsupported, to call and update both account orders and positions, without having to restart NT, when the API goes into stall mode.

[NinjaTrader_PatrickG]

Thanks to everyone in this thread for providing details on what they're experiencing and for your patience while we investigated further.

We were able to hit the symptom this morning of order modifications getting 'stuck' in a pending status. The Development Team is now researching further. The internal tracking number is 14419. We were also able to confirm that this symptom does not occur with the new TD Ameritrade API that is currently in development.

I'll follow up when there is any new information.

[nickslad]

This is awesome to hear! Thanks so much for investigating this.

[dpolyakov]

This is how Merket watch looks like when there is no updates for random instrument (MS here)

[dpolyakov]

I played a little bit more with Trade Performance window. It is totally off, not even remotely close. It is correct while current session is opened, but goes completely off if restart NT with closed trades for today. Just for example: I purchased USO yesterday and sold today, shown P/L is correct. Then restart NT, P/L is already wrong.

TD's website does not have tools to see your performance. The only way around to get your correct P/L for the day/week/month is to get CSV file and work in Excel. There I can get everything right. If anyone knows better way, please let me know.

[NinjaTrader_PatrickG]

dpolyakov , the Trade Performance report has no bearing on actual trades that occurred and should always be considered secondary to your brokerage statement. NinjaTrader has no concept of 'trades', but simply pairs together executions. There are many reasons why a Trade Performance report can become out of sync. A few examples of this:

• Partial fills
• Missing executions
• Duplicate executions
• Holding overnight positions
• Running a report from a start date on which you were not flat
• Using another application or device to place an order may not record the execution within NinjaTrader, including another instance of NinjaTrader on another computer.

[nickslad]

The only way around to get your correct P/L for the day/week/month is to get CSV file and work in Excel. There I can get everything right. If anyone knows better way, please let me know.

I do the same thing, and keep a separate log in Excel. It's probably the only way to go right now.

Regarding the Trade Performance report, what I have noticed in my own database is that after closing and restarting, NinjaTrader populates with duplicate executions, which is what throws off the calculations. While the application is open in the current session it's correct. But after restarting, it populates with duplicate executions, which throws off the calculations. What I have noticed is that everything is roughly doubled over a given time period (with some occasional errors). It's unfortunate as one can't really use any of the summary or analysis tools.