Performance Profiling latest NT8 release.

[JimB17]

On my system am seeing memory continue to climb when running NT8. I have a number of AddOns that are doing things but in the prior release the memory usage stabilized at about 600MB. Now I am seeing the memory continue to tick up. Have been running just a few minutes and am at 3.6 gigabytes of memory.

When doing a Profile seeing that the top uses of memory are the Logging thread and also get_MachineName() which are allocating 4.6GB and 2.2GB in the short run did just starting up the program than shutting it down.

Any chance your team could look at what they are doing and get this under control?

Have four screen snapshots of the Profiling will try to attach them. Will do this in followup assuming this posts correctly.

[JimB17]

screen snapshots.

below are the screenshots if they will let me upload them...

[JimB17]

Log Thread breakdown.

It looks like the top offenders are from the Log Thread because it is calling get_MachineName() as well as SplitInternal() many times. Obviously the machine name really only needs to be called once......

I am not doing any significant logging and in the NinjaScript Output window only have five log lines showing up. I do use Output 2 for my own logging but not very much.

See attached breakdown.

[NinjaTrader_RyanM]

Hello JimB17,
Please send me your log and trace files so that I may look into what occurred to platformsupport[at]ninjatrader[dot]com.

You can do this by going to the Control Center-> Help-> Mail to Platform Support.
Please reference the following in the email body: ATTN RyanM

http://ninjatrader.com/support/forum/showthread.php?t=103646

[JimB17]

Hello JimB17,
Please send me your log and trace files so that I may look into what occurred to platformsupport[at]ninjatrader[dot]com.

You can do this by going to the Control Center-> Help-> Mail to Platform Support.
Please reference the following in the email body: ATTN RyanM
http://ninjatrader.com/support/forum...d.php?t=103646

The item that needs to be reviewed is the VSP file which is 13.4GB generated automatically by Visual Studio 2015. This is the function call traces.

If you want me to get this to your development staff let me know how. They should be able to easily do this themselves however and actually reviewing this process is part of making efficient code.

I have used NT8 to submit log/trace files but they are not going to tell you anything compared to what the VSP file is telling you. Calling "get_MachineName()" over ten million times is not going to be in the log and trace files.

I have seen commercial products having one area of code inefficient and doing 90% of the processing of the entire program, with very simple changes to fix. It appears this is the scenario in this case simple fix could correct if programmers looked at Profile information and drilled down. However simple log files will not tell you this info.

[NinjaTrader_RyanM]

Hello JimB17,
I will first need to review the log and trace files to find if any direct items are reporting within NinjaTrader regarding your inquiry. I will be able to relay any information to the Product Development team at this point as well. I will be unable to do so from the forum as the files I am requesting will contain sensitive information.

[JimB17]

As stated I did as instructed sent log/trace files including in body link to this conversation.

Look forward to any resolution. I am modifying my code to make it more efficient but on startup still seeing NT8 memory head up to 3.6GB. Am only looking at ES as well. The primary underlying issue appears to be logging thread which is not a GUI item.

Prior to the last release the memory would periodically spike up, but with the most recent release it is more consistent. My machine is not limited and have 6 physical cores, 16GB memory.

[JimB17]

I've traced down the problem a bit more. There is an exception being generated:

"The current process has used all of its system allowance of handles for Window Manager objects"

I am bringing up NT8 without any of my addons, and removed JigSaw, just two charts, the NinjaScript Output and the Control Center.

The memory is going up to about 3.5GB and the number of handles is 2400 or so. According to the web this is probably a "Handle Leak". Not sure if number is accurate which shows in task manager the threshold is supposedly 10000.

The market is currently closed when I am doing this as well, so no updating ongoing.

The log file is going crazy with this message being written out, I have one which is larger than a gigabyte in size. But base issue appears to be a handle leak. Of course things like get_MachineName() should still not be called 15 million times, but the excess log file generation is from a handle leak. See attached.

[JimB17]

trying to get task manager image here - let's try again. the memory of NT8 goes to this level in just under a minute from clicking the icon with no other input just program startup. then it stabilizes at this level.

[sledge]

Have you done a complete uninstall/reinstall?

Rename your my documents/nt8, do windows programs nt unmistakable then rerun the installer, and start fresh.

Copy the items back in to the my documents/nt8 until it starts happening again(if it does) to try to narrow it down.

[JimB17]

Have you done a complete uninstall/reinstall?

No but I have been around since the beginning of time and this was always the solution Microsoft proposed with any problems.....

This does not track down or fix the issue at all. And I have found bugs never "go away" this will just come back. It seems finding the issue is more important than glossing over it..

I know NT8 is a complex program and they have been working hard to get it right, but it seems like the time is ripe for them to determine what is ongoing when the program is really going rogue at startup and sucking down over three gigs of memory.

My first programming was on 32K memory computers, could not even imagine a megabyte back then and now we have three gig leaks?

[sledge]

I know what you mean.

All my systems here (6?) Are affected with this replay bug : http://ninjatrader.com/support/forum...ht=LastBarTime

They could not reproduce it.

I've been upgrading my systems for just about each beta through release. They recommended a clean install, and I currently can not reproduce on 2 machines. I am keeping my eye out for the issue to return though.

Well, if you do a clean install and it comes back when you move the add on folder back, you can hopefully track it Down to something in there.

If it doesn't happen, you can always move your original bad folder back in and see if it is better or still bad.

[JimB17]

I know what you mean.

{replay bug called out - can't repeat}

Well you are brave to use the replay logic - I found it too frustrating as someone who writes software myself knowing how much better it could be to navigate.

Ok here is the kicker... I sit down to try to do a test run with the simple workspace capture Performance metrics.. Guess what - NT8 runs without a hiccup! Can use full workspace (includes four AddOns and then other Indicators and Strategy).

The memory used is as expected now sitting at about 700K which is standard. No data is flowing but yesterday no data and at least a half dozen times it spiked up right away to 3.5GB usage..

My computer has not been changed, including rebooting or sleeping or even turning off monitors.. But the program behaves as expected.

The log files from both yesterday and today show nothing out of sorts. But now the wild child rogue behavior is all gone.

I have seen the memory ramp up over time this past year, can't tell when it will happen, sometimes up to 5 or 7 GB. Sympathize with the developers because these can be hard to find. I've had bugs which are very elusive know how it feels.

They should really put someone on watching the Performance Profile though - way too much going on with logging and such. Programmers have gotten lazy with garbage collection and easy object management scenarios.

They could probably double overall performance if one person sat down for a week and just hammered down where the code was spending it's time - usually there are some surprises what is busy and can be simply fixed.