Chart Rendering - Upgradeable lock error

[QuantKey_Bruce]

Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Upgradeable lock may not be acquired with read lock held.'

Could support please elaborate on what sorts of issues are involved in this log message? I see many references to it on the forums. I am seeing it from time to time on code that does not involve any locks.

In one post, support indicated it could be a result of "threading problems" but that is not really enough to go on in this case. Under what circumstances could this error message arise?

[NinjaTrader_PatrickH]

Hello Bruce DeVault,

Thank you for your post.

This messages can be seen when we do not dispose of objects used in OnRender correctly. These objects can include Brushes, TextFormat objects, and TextLayout objects.

You can find further details at the following link: http://ninjatrader.com/support/helpG...arpDXResources

Please let me know if you have any questions.

[QuantKey_Bruce]

This messages can be seen when we do not dispose of objects used in OnRender correctly. These objects can include Brushes, TextFormat objects, and TextLayout objects.

Thank you - are there any other cases you are seeing besides these already listed in the documentation?

[NinjaTrader_PatrickH]

Hello Bruce DeVault,

Besides general threading related matters this could something very specific to the implementation or the code. Without viewing the code it would be difficult to give a correct assessment of the cause.

Please let me know if you have any questions.

[QuantKey_Bruce]

Patrick,

I have in at least one case a machine in which none of the indicators have any custom OnRender at all, yet it still gives this message some of the time at data feed connection time. Because the message is about a SharpDX problem, and OnRender is not being used by any indicator, I'm a little perplexed. It seems the message is not directly about the real issue, whatever it is.

If an indicator or indicators give this OnRender error, yet they do not have any OnRender functions, how would you go about proceeding?

[NinjaTrader_PatrickH]

Hello Bruce DeVault,

Thank you for your response.

If possible can you send over your workspace files, and your log and trace files?

Please follow the steps below to manually attach your log and trace files to an email to platformsupport[at]ninjatrader[dot]com so we may investigate this matter further.

• Open your NinjaTrader folder under Documents.
• Right click on the 'workspaces', 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
• Send the 3 compressed folders as attachments on your email.
• Once complete, you can delete these compressed folders.

Please list 'ATTN: Patrick H' in the subject line and reference this thread in the body of the email.

We look forward to your responses on this matter.

[QuantKey_Bruce]

Thank you. I will provide documentation. When 8.0.5.0 was released yesterday, I noticed the release notes referenced some SharpDX rendering issues being fixed, so I started re-testing; however, unfortunately, this is not fixed. I will write it up and send you documentation as soon as I finish testing the new build.

[QuantKey_Bruce]

I have isolated the issue reported in this thread to a multi-threading race condition still present in NT 8.0.5.0 in which, during the connection to the data feed, the platform fatally attempts to lock in-use SharpDX resources while OnRender is still running. The reason it did not manifest every time is because it is a race (and the majority of the time, OnRender wins), however, I was able to contrive a way to easily demonstrate the issue without using any Thread.Sleep and without the risk of any error in implementing the OnRender.

To reproduce this issue and to make 100% sure this was not an error in our code, I simply loaded up the NinjaTrader provided SampleCustomRender indicator and saved it under a new name, TestCustomRender03082017. I made a single change to the indicator, which was to enclose the entire, perfectly functioning OnRender example in a loop for (int i = 0; i < 1000; i++) { /* rest of OnRender here */ } so that the OnRender would take longer to run, causing the issue to no longer be intermittent but to happen every time for NinjaTrader support's ease of reproducing the problem. Note that the example disposes of all of its resources (in fact, it is the textbook example of doing so) and that it does not call the base OnRender, so performing its valid operations repeatedly causes no issue other than making it stay busy for longer, exactly as NT8 would be busy were there simply more complex indicators or more simple indicators on more charts.

With this minor contrivance to make it take longer, the race condition now manifests just about every time. To demonstrate this, simply create a 10-day chart of 1-minute bars of ES, and add the attached indicator to it (or make exactly the change I just described to the stock example, perhaps modifying the number of iterations to account for how fast your CPU is), saving this chart in a workspace by itself. Do not have your data feed set to connect at startup, and restart the platform with only that workspace open. Upon connecting the data feed, with only this one chart, and with only this one indicator running (a perfect NT OnRender example with no coding errors, only modified to make it run a thousand times as long so that OnRender will consistently lose the race instead of only losing once in a while completely randomly), it crashes NT 8, resulting in the 'Upgradeable lock may not be acquired with read lock held' message in the log, but requiring the platform to then be ended via Task Manager as Control Center is no longer responsive.

Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Upgradeable lock may not be acquired with read lock held.'

To fix this, it is necessary during this "refresh everything" process at data feed connection for the platform to hold any further renderings from starting, and wait for any running rendering to be completed before attempting a complete lock of the resources currently still in use.

I want to emphasize that we cannot fix this on our side - catching an exception will not leave the platform in a usable state and it must be restarted if this happens, and we cannot simply check for null here - the platform is taking an action that results in the platform crash, and we cannot prevent the platform lock from our end because we have no control over when the data feed may connect from a running OnRender.

It is also important to recognize that while I modified an OnRender example to make OnRender take longer, this issue is not restricted to only indicators that have a custom OnRender. It could happen even to a stock SMA (with less probability) because the base OnRender could be running - it is just that by modifying the overridden OnRender in this way, I am able to make it take longer, and therefore, demonstrate the race condition more clearly.

The reason, down the thread, I was questioning the answer that I had not disposed properly of unmanaged SharpDX resources is that I had already eliminated this (even going so far as to get rid of all OnRenders completely leaving only the base Render, but the issue remained), and that is why I was asking if there were any other reasons possible. Eventually, I was able to establish that this can happen even with stock indicators (just less likely, because they render more quickly) and created this modified example just in order to be able to reliably demonstrate it, that it isn't anything I am doing but rather is a problem with the platform in general that this race condition exists.

This problem seems to be more severe on multi-monitor systems with a large number of charts open, and that makes perfect sense - in total, OnRender is running for longer because there are more charts and more indicators running on the same number of cores, increasing the chance of the race failing. It is for this reason that I had to create an artificial example to force it to happen on just one chart so you can easily test it - if you simply throw SMA on a single chart, it is unlikely to happen because in the aggregate and considering the cores you have available, OnRender across all of them isn't running very long. Importantly, the indicators involved don't have to be complex - there just have to be enough of them (enough charts, or charts on enough monitors, or enough indicators on the charts) that OnRender takes long enough in total to lose the race.

The fact that this race condition exists (and is so severe that it crashes the platform, and cannot be caught with a != null or .IsDisposed check or with a try catch block) is something that really needs to be addressed on the platform side - it is a critical failure because it cannot be predicted nor controlled from our end.

Edit: I have subsequently emailed platform support with log and trace files corresponding to following these instructions, an excerpt from which is posted below. There is a second error shown after the important one - the second error is that an object reference is not set to an instance of an object, and this is during the attempt to save the workspace after the original fatal error. Remember, this is OnRender from the NT-provided example custom rendering indicator, not an OnRender I wrote - these errors are unavoidable as well unless you wait for OnRender to exit before disposing of the resources that are in use. At 10:36:03 the data feed connects, resulting in the Upgradeable lock message, and eight seconds later at 10:36:11 I saved the workspace as I was trying to exit the platform, resulting in a completely different, second set of errors as NT apparently disposed of the required resources while OnRender was still running (that too should likely be handled by holding for OnRender to exit first). Note as well that it would not be enough to add a bunch of foo != null and !foo.IsDisposed() checks to your own code sample for custom rendering - the race could easily happen such that the check completes but the object is disposed on the other thread before the resource finishes whatever the SharpDX operation is, and the first time this happens, the platform usually must be ended with Task Manager so we cannot recover from this by simply doing a check or a try..catch block; for maximum safety it must be handled on the platform side. I want to emphasize this is NT's own code example of how to do everything right, not something written by a user incorrectly.

2017-03-09 10:35:37:841|2|4|Session Break (Version 8.0.5.0)
2017-03-09 10:35:38:938|1|4|Verifying license at primary server...
2017-03-09 10:35:39:046|1|4|Type=Simulation State=Verified Start Date=12/31/2004 End Date=11/30/2099
2017-03-09 10:35:40:138|1|4|Global simulation mode enabled
2017-03-09 10:35:44:274|1|2|Using IS (is-us-002.ninjatrader.com/31658)
2017-03-09 10:35:44:288|1|4|Automated trading disabled
2017-03-09 10:35:44:474|1|32|Order entry hot keys disabled
2017-03-09 10:35:44:475|1|4|Auto close enabled=False
2017-03-09 10:35:45:095|1|4|Restoring workspace 'Test Custom Render 03082017'...
2017-03-09 10:35:54:009|1|2|AMP/CQG: Primary connection=Connecting, Price feed=Connecting
2017-03-09 10:36:03:005|1|2|AMP/CQG: Primary connection=Connected, Price feed=Connected
2017-03-09 10:36:03:050|1|2|Time to auto close position='12:00:00 AM', Enabled=False
2017-03-09 10:36:03:053|1|2|Using HDS (hds-us-002.ninjatrader.com/31655)
2017-03-09 10:36:03:239|3|4|Failed to resolve instrument for market data: F.US.EP.H16 could not be resolved.
2017-03-09 10:36:03:374|3|16|Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Upgradeable lock may not be acquired with read lock held.'
2017-03-09 10:36:11:286|1|4|Workspace Test Custom Render 03082017 saved
2017-03-09 10:36:11:334|3|4|Indicator 'TestCustomRender03082017': Error on calling 'OnRender' method on bar 10840: Object reference not set to an instance of an object.
2017-03-09 10:36:11:336|3|16|Chart rendering failed. There is likely a problem with a chart object's OnRender method. D2D error = 'Object reference not set to an instance of an object.'
2017-03-09 10:36:11:359|1|2|AMP/CQG: Primary connection=Disconnecting, Price feed=Disconnecting
2017-03-09 10:36:11:640|1|2|AMP/CQG: Primary connection=Disconnected, Price feed=Disconnected

[NinjaTrader_PatrickH]

Hello Bruce DeVault,

Thank you for putting together this test and providing the steps.

I am able to reproduce the message with your edits to the SampleCustomRender. I cannot, however, get the platform to be unable to close. This of course does not take away from the fact that you have provided a clear example.

We will follow up here when we have further details.

[QuantKey_Bruce]

PatrickH,

Whether the platform is unable to be closed is likely a function of how the specific video driver handles a severe failure e.g. segmentation fault because of an operation on an already disposed, unmanaged resource, and also depends on which of the many DirectX functions was executing when it failed - that might potentially be different every time.

[NinjaTrader_Brett]

Bruce,

Really appreciate you putting this together and sending in, we are taking a look into it now. By the way we did try to reproduce below error by using Thread.Sleeps() internally before with no success. Hopefully your approach down below will allow us to reproduce the below problem and track it down.

Will get back with results.

-Brett

[reach4thelasers]

I'm getting this issue as well. I don't think its related to the number of charts open, the OnRender overloading, or to data connections, as I see the issue in my "Development" workspace, which generally has a couple of charts and a debug window and no data connection, and I'm using an indicator that uses Simple Draw.Line and Draw.Text calls. It could be related to the Screen resolution as I see it more on a chart maximised at 3840x2160 - probably because the actual rendering takes longer the more pixels it needs to draw over.

I don't think its related to the blocking of OnRender, furthermore, Wrapping the contents of OnRender in a FOR loop doesn't actually make rendering take longer; it takes the same time..... But happens over and over again; which increases the chance of a race condition happening.

In my scenario; I see the issue when I bind draw events to MouseMove therefore causing them to happen repeatedly in fast succession - same as wrapping OnRender in a For loop.

if (State == State.DataLoaded)
{
  this.ChartControl.MouseMove += delegate(object sender, MouseEventArgs args)
 { 
	var bar = mouseEventArgs.GetBarIndex(this.ChartControl, this.ChartControl.PrimaryBars);

	//if (bar == _lastMouseBar) return; - uncomment this and the issue does not transpire
	_lastMouseBar = bar;
			

	var barsAgo = CurrentBar - bar;
	var highPrice = High.GetValueAt(bar);
	var lowPrice = Low.GetValueAt(bar);

	Draw.Line(this, "LowMouse", true, barsAgo - 15, lowPrice, barsAgo + 15, lowPrice, Brushes.DarkRed,
				DashStyleHelper.Solid, 5);

	Draw.Text(this, "LowMouseText", false, "Low", barsAgo, lowPrice, -15, Brushes.DarkRed,
				new SimpleFont("Arial", 15), TextAlignment.Center, Brushes.Transparent, Brushes.Transparent, 0);

	Draw.Line(this, "HighMouse", true, barsAgo - 15, highPrice, barsAgo + 15, highPrice, Brushes.Green,
				DashStyleHelper.Solid, 5);

	Draw.Text(this, "HighMouseText", false, "High", barsAgo, highPrice, 15, Brushes.Green,
				new SimpleFont("Arial", 15), TextAlignment.Center, Brushes.Transparent, Brushes.Transparent, 0);

	ForceRefresh();
  };
}

The problem goes away if I do a check in my mousemove handler to see if the bar is actually different to the last time it was called, therefore only re-rendering if I'm over a new bar.

Also, If I Use System.Linq.Reactive Extensions to wrap up the mousemoves into an Observable event stream, I can use the debounce method called "Sample" to ensure the handler is not triggered more than once every 100ms. This works too.

if (State == State.DataLoaded)
{
	
	var mouseMove = Observable
		.FromEventPattern<MouseEventArgs>(ChartControl, "MouseMove")
		.ObserveOn(this.Dispatcher)
		.Sample(TimeSpan.FromMilliseconds(100));

        mouseMove.Subscribe(args => ChartOnMouseMove(args.Sender, args.EventArgs));

}

Either way there is an issue somewhere. I was only using Draw.Text and Draw.Line; perhaps the issue lies within the OnRender implementation of these tools?

[QuantKey_Bruce]

Remember that just because you are not overriding OnRender does not mean it is not being called. The base class Indicator renders the plots and each drawing object has its own OnRender. The issue is just as I described it - the OnRender methods (of the base class, or of the drawing objects you are using) are being called potentially while another thread breaks down the SharpDX resources involved. The reason sampling "eliminates" the error is that you have made it less likely (perhaps a lot less likely) in exactly the same way as an SMA's OnRender of its plot "eliminates" the problem compared to the custom render that runs a thousand times, but the issue is still there - it just may be one in a million instead of one in two. The only solution that will be guaranteed is if NT does not do this (break down the resources while OnRender is running, whether it's the indicator's overridden method, the base method, the drawing tool's method, etc.).

Regarding wrapping OnRender "in a loop" what I actually did was put the loop inside of the OnRender method, so that its main task (the one that formerly was all OnRender did) is repeated a thousand times before OnRender exits. So, OnRender itself does take longer before exiting, even though one iteration of its loop takes about the same time it took before to run the whole method. The reason I did this is to increase the odds of the race condition failing, and it worked - it clearly demonstrates the problem because now OnRender does not exit for so long that the resources are tied up in total for a much longer time, resulting in a near certainty of the race failure being exposed to the support staff so they can try to fix it.

You hit the nail on the head when you say that screen resolution affects it only in so far as it takes longer to render more pixels. My artificial example that loops OnRender's main method body is just forcing the issue to the surface so NT can easily reproduce it and then fix it. Nobody would do that in real life - this is a test rig so they can see what's happening and know if they fixed it.

Just as you would have this issue even without overriding OnRender (with less probability) I can demonstrate the issue using indicators that do not override OnRender. I chose to override OnRender so I could force the probability of failure up in the direction of 100% so I could give them a test case they can hammer on until they fix it, but the reality is overriding OnRender is not necessary to have this failure, and I have this failure here at times on indicators that do not do their own rendering and only use plots. The native render methods of plots and drawing objects have the problem as well, and if you use enough of them (or enough resolution, or big enough charts, etc, etc) it will still fail some small percentage of the time. Reopening the workspace or restarting NT8 after making some change makes it tempting to say "aha, it's fixed!" but the reality is it's lurking out there, waiting to spring the failure on you ten restarts from now when you have maybe long forgotten what you last tried changing.

This uncertainty is why this issue is so important, and why I took the time to make a careful test case so I could help them put their finger on it and fix it. Having it every once in a while, whether it's 1 in 3 or 1 in 300, fail unpredictably is just not okay for trading technology.

[NinjaTrader_Brett]

Hey Guys,

We were able to debug further into it using the approach Bruce provided (Thanks Bruce!).. We have applied a fix and will be testing it internally. If anyone else is running into Upgradable lock error message issue I would love to send you a custom build to make sure it will resolve the issue for you so we make sure we have this one squashed. Please contact me at platformsupport at ninjatader dot com and reference this post and I'll supply you a custom build early this week. Thank You!

[jjhou]

This problem also happened on my NT8.0.5.0 serveral times