B12 Race Condition - Connect on Startup to Data Provider

[RanchoDinero]

The race condition is that OnStateChange() with State.Terminated can be called when indicators that are loaded in a workspace already have their historical data loaded and are in a render pass (executing the OnRender() event).

So the chart's RenderTarget (among other things SharpDX, ChartControl) get stomped on underneath the indicator that's doing the rendering. The same NT dispatch thread really should do the rendering and the state changes, otherwise there's no way for an indicator to know it's been terminated while in the middle of a render pass. No mutex or critical sections that can be created by an indicator can stop or even mitigate this.

The issue goes away entirely when the data provider is not connected automatically on startup, because nothing is getting terminated while it's in a render pass, which happens when the data provider connects.

Also, this conditions seems to only happen on startup. If in real-time, the data provider gets disconnected then reconnected, no issue. Again... race condition due to the processing load on startup.

Here are the relevant items from log my log and trace files.

2016-08-20 19:31:30:280|3|16|Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Upgradeable lock may not be acquired with read lock held.'


2016-08-20 19:35:47:542 System.NullReferenceException: Object reference not set to an instance of an object.
at NinjaTrader.Data.Bars.GetNewBarsProxy(Bars barsProxy)
at System.Linq.Enumerable.WhereSelectArrayIterator`2. MoveNext()
at System.Collections.Generic.List`1.InsertRange(Int3 2 index, IEnumerable`1 collection)
at NinjaTrader.Gui.Chart.ChartControl.<>c__DisplayCla ss535_2.<RefreshIndicators>b__8(Object o)
at System.Threading.ExecutionContext.RunInternal(Exec utionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionCon text executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System. Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

EDIT -

Same scenario but one time today I got a different unhandled exception with 2 indicators on a chart:

2016-08-21 15:36:15:055 *************** unhandled exception trapped ***************
2016-08-21 15:36:15:055 Exception has been thrown by the target of an invocation.
2016-08-21 15:36:15:056 System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeExceptio n(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List`1.get_Item(Int32 index)
at NinjaTrader.Gui.Chart.ChartControl.ApplyNinjaScrip ts()
at System.Windows.Threading.Dispatcher.Invoke(Action callback, DispatcherPriority priority, CancellationToken cancellationToken, TimeSpan timeout)
at System.Windows.Threading.Dispatcher.Invoke(Action callback)
at NinjaTrader.Gui.Chart.ChartControl.OnBarsToLoadCol lectionChanged(Object sender, NotifyCollectionChangedEventArgs e)
at System.Collections.ObjectModel.ObservableCollectio n`1.OnCollectionChanged(NotifyCollectionChangedEve ntArgs e)
at System.Collections.ObjectModel.ObservableCollectio n`1.RemoveItem(Int32 index)
at System.Collections.ObjectModel.Collection`1.Remove (T item)
at NinjaTrader.Gui.Chart.ChartControl.<>c__DisplayCla ss478_1.<OnGetBarsCallback>b__0(Bars bars, ChartControl chartControl, ChartBars chartBars2)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeIn ternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at System.Windows.Threading.ExceptionWrapper.Internal RealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatch When(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

[NinjaTrader_Brett]

Rancho,

Thanks for bringing this up and running through.

We have reviewed similar case in the past and found that due the multi-threaded nature of NinjaTrader you needed to add 'null' checks to any item which could be cleaned up in OnTermination. This is to ensure that you don't try to render something that has been cleaned up or doesn't exist. The reason this is required is that NT is multi-threaded and an indicator can be terminated from another thread as you found out and reported. Its not as simple to just make the Render/Terminated call on the same thread as you suggest since that only solves the OnRender use case. There are a ton of other multi-threaded use cases such as OnExecution, OnOrderUpdate, OnBarUpdate, etc and making them all in the same thread removes the advantages of multi-threading. In the end we elected to just come to terms with the fact that the application is multi-threaded and code in events should be coded in a way that if dependent resources were not available then the code should not run by adding null checks. This is good programming practice in any case anyways.

Now the case you reported specific to Connect On Start up it was not clear to me if this is just NT with a lone chart having issues or only with your added indicator. Since if there is an NT bug causing exception then I would look into that. If its just your code causing the exception then please add the null checks as described above.

Let me know if any questions.

[ETFVoyageur]

This comment is a generalization, so take it fwiw.

NT being multi-threaded is good.
NT multi-threading internally to its heart's content is good
NT running different indicators and indictaor instances on different threads is good.

What I see no benefit for is running a single indicator instance on multiple threads. If that is what you are doing I doubt there are significant gains, and I expect it will complicate things that should be simple for indicator writers. The less an indicator writer has to consider multi-threading the better.

For example, you cannot just check for a null pointer due to multi-threading -- that would be a timing race. You have to lock the code that checks and uses that pointer. Would it really be worth the added complication to have constant locking? Would that really help performance?

--EV

[RanchoDinero]

Brett -

I hear you, and i think you've made the right choices. But I'm only referring to this specific scenario. all other scenarios work as expected. A couple of things which may help:

1) All the call stack is in platform code when these errors occur, as you can see from the traces.

2) We're working with references copied from the platform, such as RenderTarget and others. These references can be non-null, appearing to be OK to my code, yet still be in a state in which they can't be used. For example ones where directx handles the references pass into the native APIs have been already destroyed. There's no way for 3rd part code to guard against this. Or is there? Can you shed some light if there is?

3) The problem is far worse when using a data feed which, relatively speaking, takes a long time to connect up such as IQFeed. NT has to fire up another process and then connect to it via the loopback TCP interface. This can take a few seconds sometimes. When there's a full load of charts loading cached data and starting to render, the length of time it takes gives them a chance to get into the first render pass. Then... bam.

4) There's no way to repro this problem without a full load of charts loading cached data and rendering at the same time the connection is made. I suspect this is why it hasn't been seen so far, or maybe it has and it's not been reported. I dunno.

Maybe you have some ideas for how we can protect ourselves... I don't think checking references obtained from the platform code for null will \ be enough. Repeating myself here, but the underlying objects can still be in an invalid state if the platform begins tearing down before OnStateChange(..) returns with State = Terminated. Especially if another thread is in OnRender() in the same object instance.

Hopefully this is making sense. It's a tricky one for sure. Even if multiple threads are rendering and calling state changes, it might be enough to ensure all chart objects to return from OnStateChange(Terminated) before touching ANY platform references that are public to indicators, add-ons, drawing tool, etc.

Any additional guidance/insight you have here would be most welcome.

P.S. Just wanted to add that my render code and the code executed in OnStateChage(Terminated) is already inside the same mutex. So different threads are definitely calling render and statechange on the same instance when this happens.

[NinjaTrader_Brett]

Thanks your analysis, I am following and understand. Callstacks below suggest potentially an NT problem at first glance as you suspect. I'm asking development to take a quick look at your post and suggest next steps.

-Brett

[RanchoDinero]

Brett -

Thanks much. FWIW, it just happened again, only with a different exception this time, grabbing a secondary data series from OnStateChange() State == State.Configure:

2016-08-23 17:29:58:966 Cbi.Instrument.RequestBars (to Provider): instrument='NQ 09-16' from='08/23/2016 17:00:00' to='08/23/2016 23:59:59' period='1 Tick'
2016-08-23 17:30:02:638 Data.Bars.RequestBarSeries1 ERROR: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
2016-08-23 17:30:02:828 *************** unhandled exception trapped ***************
2016-08-23 17:30:02:828 Value cannot be null.
2016-08-23 17:30:02:834 System.ArgumentNullException: Value cannot be null.
at System.Threading.Monitor.Enter(Object obj)
at NinjaTrader.Data.Bars.<>c__DisplayClass91_1.<GetBa rs>b__1(Bars b, ErrorCode c, String m, Object s)
at NinjaTrader.Data.BarsSeries.<>c__DisplayClass243_0 .<RequestBarsSeries>b__6(Bars b, ErrorCode c, String m, Object s)
at NinjaTrader.Data.BarsSeries.<>c__DisplayClass243_2 .<RequestBarsSeries>b__4(Bars b, ErrorCode c, String m, Object s)
at NinjaTrader.Cbi.Instrument.<>c__DisplayClass201_0. <RequestBars>b__1(Object o)
at System.Threading.ExecutionContext.RunInternal(Exec utionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionCon text executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System. Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.QueueUserWorkItemCallback.System. Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

[NinjaTrader_Dierk]

@RachoDinero

There potentially might be different issues intertwined (guessing). Let's start with something simple in order to eliminate variables:

What exactly causes your indicator to go State=.Terminated? Do you have some custom code which calls SetState(.Terminated) or is this done by NT itself as a result of your scenario or else?

Thanks in advance

[RanchoDinero]

Thanks for the idea... but no such scenario where my code sets its own state. That would be a little nuts, even if its possible.

From my POV, the evidence points to these as the next courses of investigation:

1) What are all the possible internal ramifications to an indicator/drawing when 2 different threads call OnRender() and OnStateChange() when one thread is in OnRender() while the other thread is calling OnStateChage(Terminated)? Consider when there are say 20 indicators/tools loaded across many charts and several threads both rendering and trying to tear down running code because of a data provider was connected.

2) If one or more indicators throws an exception out of its scope and it's caught by the code which calls OnStateChange(Terminated) or OnRender(), what are all the possible state that the platform code could be in? Would there be a problem with calling OnStateChange(SetDefaults/Configure/etc) on another instance (or the same one?) right afterwards?

It looks to me, again based on the evidence, it's one or the other or some combination of the above.

You'll just have to trust that based on my explanation of the situation, if it were something really simple we wouldn't be having this conversation in the first place.

[NinjaTrader_Dierk]

@RanchoDinero

To clarify: none of your code is actually calling SetState(.Terminated) nor SetState(.Finalized) nor SetState(<anyState>), correct?

[RanchoDinero]

That's what I said in the first line of my reply.

[NinjaTrader_Dierk]

Thanks for clarification.

Next step: would you have a scenario + workspace where the exceptions/error logs/issues you reported could be reproduced with a 'certain' likelihood? If so, could you please:
- describe the steps you're performing to produce the exception/error log/issue
- send in your workspace(s) to platformsupport AT ninjatrader DOT com, reference this forum thread in your email and add 'for Dierk'?

Thanks in advance

[RanchoDinero]

Yep... I've already described this in my earlier posts.

Everything about this points to a race condition... so it's by definition not deterministic. That said, if you create a workspace where there are say 20 indicators that do custom rendering and load a lot of historical data across a bunch of charts then auto-connect, you should be able to repro.

You'll have to create such a workspace yourself. I don't have one I can give you because all the workspaces I have like this reference my code.

[NinjaTrader_Dierk]

We will try to reproduce and get back to you in case we can't.

[QuantKey_Bruce]

I just want to add to what has already been commented here. While it makes sense to check for null on required resources, unless those resources are locked, it cannot be guaranteed they will be disposed of during the execution of your code.

As an example, if during OnRender, you check (if Bars != null) { ... } there is no guarantee that Bars will not be disposed of halfway through your OnRender routine because the indicator is being broken down by a different thread. Even if you were to check before each line that accesses Bars or any similar resource, context could switch in between the execution of the if statement and what follows, or the disposal of the indicator could be proceeding in parallel.

This is an architectural design issue, and not one that can be left to individual indicator developers. While I can appreciate what you are saying that everyone should "check for null," that is not enough to assure proper operation, unless (at least) it can be assured that the indicator will not be destroyed during an OnRender invocation.

[strategesis]

I have some workspaces which show the problem with some reliability. However, they do reference my code. I'd have to also provide you with protected DLLs for my indicators. Or, if you really need to see the source code (debugging,) I'd require that you sign an NDA.