IB adapter bug - can not run live

[heech]

Hi,

I'm opening a thread here, although I've also been exchanging emails with NT support. It's hard having a consistent conversation over email, and I'm desperate.. so I'll give this a try.

I've had 2-3 good weeks with IB and NT, but now bugs are popping up to make the platform unusable.

The symbols are this:

- I opened stock and (wrote) option positions in the last few days; everything worked fine, strategy worked fine.

- now, upon connecting to IB, I get this warning in the log:
[code] 3/19/2009 3:11:48 PM Default Error validating request:-'xb' : cause - When the local symbol field is empty, please fill all option fields (right, strike, expiry) ServerValidateError[/quote]

- looking into the TWS log file, I find this:

MF 06:38:15:428 JTS-EServerSocket-40: SECDEF  Sending  reqId=FixSecDefReqBySymbol38  symbol=FSYS 0 SMART
MF 06:38:15:475 JTS-EServerSocket-40: [0:20:42:1:0:0:0:ERR] -'wb' : cause - When the local symbol field is empty, please fill all option fields (right, strike, expiry)
MF 06:38:15:475 JTS-EServerSocket-40: Anticipated error
MF jextend.d: When the local symbol field is empty, please fill all option fields (right, strike, expiry)
	at jextend.wb.d(wb.java:300)
	at jextend.y.h(y.java:233)
	at jextend.pd.L(pd.java:800)
	at jextend.pd.run(pd.java:660)
	at java.lang.Thread.run(Unknown Source)

- if I try to continue (no matter how many times I restart NinjaTrader), I get the "NinjaTrader has had an error and must close down" error message.

- the NinjaTrader trace shows this:

2009-03-19 15:11:43:421 (xIB2) Cbi.Connection.ProcessEventArgs.PositionUpdateEventArgs1: Instrument='FSYS' Account='' Avg price=10.17374705 Quantity=1,500 Market position=Long Operation=Insert Currency=UsDollar
2009-03-19 15:11:43:468 in OnUnhandledThreadException
2009-03-19 15:11:43:546 ********* exception trapped *********
2009-03-19 15:11:43:546 The changes you requested to the table were not successful because they would create duplicate values in the index, primary key, or relationship.  Change the data in the field or fields that contain duplicate data, remove the index, or redefine the index to permit duplicate entries and try again.
2009-03-19 15:11:43:546    at System.Data.OleDb.OleDbCommand.ExecuteCommandTextErrorHandling(OleDbHResult hr)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
   at System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
   at Wilson.ORMapper.Internals.Connection.TransactionCommand(Guid transactionId, Type entityType, CommandInfo commandInfo, IDbTransaction transaction, String sqlStatement, Parameter[] parameters)
   at Wilson.ORMapper.Internals.Instance.InsertObject(Transaction transaction)
   at Wilson.ORMapper.Internals.Instance.PersistChanges(Transaction transaction, PersistDepth persistDepth, Boolean parentDeleted)
   at Wilson.ORMapper.Transaction.PersistChanges(ICollection entityObjects, PersistDepth persistDepth)
   at Wilson.ORMapper.ObjectSpace.PersistChanges(ICollection entityObjects, PersistDepth persistDepth)
   at Wilson.ORMapper.ObjectSpace.PersistChanges(Object entityObject)
   at NinjaTrader.Cbi.Connection.CreateInstrument(String name, InstrumentType instrumentType, Exchange exchange, DateTime expiry, Double strikePrice, Right right)
   at NinjaTrader.IB.Adapter.Convert(Contract contract, Currency currency, Double tickSize, Double pointValue, Exchange[] exchanges)
   at NinjaTrader.IB.AccountUpdatesRequest.ProcessPortfolioUpdateNow(Adapter adapter, PortfolioUpdate portfolioUpdate)

Note that the trace file message and the TWS error log both refer to FSYS. (It happens with 15 other symbols on the TWS side as well; NT probably just errored out and quit after the first exception on its side.) I'm pretty sure these two errors/exceptions are directly related to each other.

- final result: I get no bars. If I launch a live strategy, nothing. If I open up strategy analyzer and run a strategy, no bars. If I restart NinjaTrader, strategy analyzer works fine until I connect to IB... at which point this repeats itself.

This problem originally started with NT .8 and the previous version of TWS (889..?). I've just updated to NT .9 and TWS 892, and the problem still exists.

HELP.

[heech]

By the way, I've found one temporary work-around:

- if I run my strategy in the analyzer once *before* connecting to IB... it works fine.

Why? I have no clue. And it means I can't use the automation I've worked for weeks to put together.

[NinjaTrader_Dierk]

a) make sure you don't have options on your TWS since NT does not support options
b) make sure the instrument you trade (stock?) has assigned the Default exchange and (!) the native exchange (see instrument manager)

[heech]

a) make sure you don't have options on your TWS since NT does not support options
b) make sure the instrument you trade (stock?) has assigned the Default exchange and (!) the native exchange (see instrument manager)

Dierk,

a) you can't really be serious about this. I don't need NT to "support" options, just not crash from the fact that I own some in my IB account.

I've also been trading an active options/stock strategy for 3 weeks on IB (through NT), w/ over 45 options in my position (but of course never traded by NT). Up until yesterday, no problems.

b) yes, default/native exchange all selected.

[NinjaTrader_Dierk]

a) We are serious about our recommendations: NT does not support options.

b) please double check that instrument FSYS has:
- Default and native exchange assigned in your instrument manager
- it has the IB mapping "FSYS"
- no other instrument has the mapping "FSYS"

Worst case give a try by a fresh DB by renaming your DB file <my documents>/NinjaTrader6.5/NinjaTrader.mdb to <my documents>/NinjaTrader6.5/NinjaTrader.old.mdb. You could restore it later on

[heech]

a) We are serious about our recommendations: NT does not support options.

b) please double check that instrument FSYS has:
- Default and native exchange assigned in your instrument manager
- it has the IB mapping "FSYS"
- no other instrument has the mapping "FSYS"

Worst case give a try by a fresh DB by renaming your DB file <my documents>/NinjaTrader6.5/NinjaTrader.mdb to <my documents>/NinjaTrader6.5/NinjaTrader.old.mdb. You could restore it later on

Just once in my life, Dierk, I'd love to hear you say "that's a bug in my software, and I'll do my best to fix it".

FSYS is mapped correctly, and note that the problem exists for any number (all?) of the instruments. I think it's incredibly odd that running strategy analyzer "fixes" the problem. There must be some caching behavior involved here.

I've tried deleting the .mdb several times. I also have to delete the logs, or the positions get refilled in the .mdb upon startup... and everything is still broken.

If I do all that... everything works, the first time I connect to IB. But the next time I close/reopen NT and connect to IB, it breaks again in exactly the same way.

I sent in my database + log files several times over the last 24 hrs. Please take a look.

[NinjaTrader_Dierk]

Please mail me to "dierk AT ninjatrader DOT com":
- your zipped DB
- your IB login credentials

Thanks

[heech]

Please mail me to "dierk AT ninjatrader DOT com":
- your zipped DB
- your IB login credentials

Thanks

Sending now. I'm West Coast US time, so have to sleep soon. I'll be logging in at 6:20, and will report anything new at that time.

[heech]

Thanks to Dierk.

Within 30 minutes, he was able to provide a custom build that eliminates the problem by filtering out all option-related messages, allowing me again to connect and run with IB.

[zstheorist]

Can this be true?

Oh my. PLEASE tell me that having options in your account was not the cause of this. Could it explain the bizarre behaviour I've had, where market orders occasionally get rejected from IB - this has cost me over 15% of my account this year alone. This really picked up on the last forced "upgrade".

The symptom is I get an error -1(###) and the strategy crashes (not the whole program). I have been running the previous NT .8 build and TWS from NT's site. I was just about to update to .9 and TWS892.

I have been flummoxed about this so far, because it is rare, and there is never an informative message in the logs/traces - it just crashes. The TWS trace never even has a record of any attempt at a market order - hence my suspicions that something is happening on the NT side.

On the other hand, based on the timing, I think it may happen when I try to submit a market order when there is a pending account update request. I do have options there...

Can I turn off account update requests entirely? I really do not either want or need them. This would greatly reduce the possibility of message interference.

Also, what version of the API do I need to run coincident with the upgrade to TWS892 to be consistent with NT .9? Perhaps there is a subtle issue there?

[heech]

Hi,

Well, I can only take Dierk's word for it... but he says options were the cause of my problem. (And the custom version filtering on options-related info resolved it.)

But I traded full-time for about 3 weeks with 45 instruments + options in my account, and didn't have any problems. So whatever the problem is... it isn't general.

My issue doesn't sound remotely similar to yours though, in terms of how it happened... no run-time issues at all. You'll probably have to dig through your log + trace files to get a better idea of what's happening... have you upgraded your .NET framework? Uncaught exceptions that just cause NT to crash should be extremely rare.

As far as versions go, .9 is compatible with the newest PC-based TWS (892 I believe). That's what I've been using for the last week, with few issues so far.

EDIT: Sorry, just re-read your code. Strategy crashes, but NT doesn't? I have try (Exception e) { } and catch { } blocks in every single one of my functions that NT might call into. (OnBarUpdate, OnMarketData, OnExecution, etc...). That way, at least I'll get debugging messages when something goes wrong.

[zstheorist]

Which versions should I run?

Thanks for the response. It was just that Dierk's words were frightening. I've been using NT for 2 years (1 year live) on futures trading, so when these issues arise the results can be ... unfortunate. Overall, I've been reasonably satisfied with my lifetime license.

I went through my problems in detail before, stabilized the platform, and did not change ANYTHING for almost a year until IB FORCED me to upgrade last month. Now the old intermittent issues return.

Since I trade futures exclusively via NT, I neither need, nor want to know about the rest of my IB holdings, e.g. options, so if they can cause a problem by existing in the portfolio, I would REALLY like to know. Would TWS 892 and NT .9 solve this?

All I want is stability...

If TWS 892/ NT.9 is stable, then I am thrilled to update. The only other question is: I have the old IB API 9.41 installed as well. Which version of the API is safe to have installed with the new TWS 892/NT.9? I could imagine that there is a subtle conflict buried in there.

[NinjaTrader_Dierk]

No need for confusion:
- stay with latest 6.5.1000.9
- IB API version is irrelevant, since NT does not use the IB API software components
- strategy crashes are not related to options in TWS, please proceed as per heech's suggestion to isolate your strategy bugs