Memory Problem With RC2

[GaryAlbers]

no joy

I did a 'DB Repair', restarted. TickReplay chart will not load, memory still runs a-muck.
I did a 'DB Reset', restarted. TickReplay chart will not load, memory still runs a-muck.

I did not load a workspace, so I am not having a 'corrupt workspace' issue. (I have not experienced that expect when it was my fault due to issues in my code.)

Next I will try forcing Ninja to create new database.

Gary

[GaryAlbers]

Complete wipe out needed.

I unistalled RC2 and then renamed the Ninja directory under ..\user\documents.

I did fresh install of NT8 rc2 (totally new).
First two charts with TickReplay=true loaded.
I restarted Ninja and those two charts did not load again.
Memory started running a-muck.

I restarted.
I tried the CL, NQ, FDAX with TickReplay=true. They worked.

I then tried the ES multiple times and it always resulted in memory running a-muck
(CPU goes to about 40% of quad core, memory steadily climbs until it reaches physical limit and then Ninja and machine die). I know it has more ticks, but not that much more


Gary

[tradesmart]

I unistalled RC2 and then renamed the Ninja directory under ..\user\documents.

I did fresh install of NT8 rc2 (totally new).
First two charts with TickReplay=true loaded.
I restarted Ninja and those two charts did not load again.
Memory started running a-muck.

I restarted.
I tried the CL, NQ, FDAX with TickReplay=true. They worked.

I then tried the ES multiple times and it always resulted in memory running a-muck
(CPU goes to about 40% of quad core, memory steadily climbs until it reaches physical limit and then Ninja and machine die). I know it has more ticks, but not that much more


Gary

How many bars are you attempting to load for ES?

[GaryAlbers]

Only 3 days back.
Using RC1 this normally takes about 5 seconds and results in tens of megabytes increase in the memory footprint of Ninja.
However, using RC2, it never stops until allocated memory reaches my machine max (12 gig).

So I don't believe it is because I asked for too much

Gary

[NinjaTrader_JessicaP]

Hello GaryAlbers, and thank you for your report. I am reviewing the information in posts 28-34 and am putting together a scenario I can test on my end and will return to this thread with my results. Thank you for using NinjaTrader 8.

[NinjaTrader_JessicaP]

I wasn't quite able to see what you were seeing on my system with these steps,

• Create 2 ES 12-16 1-minute charts with Tick Replay and the Volume Profile indicator

Could you help me tweak my procedure so that I can see what you are seeing? For reference, these three indicators have built-in support for tick replay, in that they have MarketData events for MarketDataType.Last :

• BuySellPressure
• BuySellVolume
• VolumeProfile

Thanks for your report and for any additional information you can provide

[soulfx]

Hello in relation to my post #14 in this thread and on adding a new tab which uses a template and tick replay = true; i reported issues with this running in RC1 and the memory quota error log message.

I am pleased to report after the other testing today i just tried this and i took note to only load 3 days of data and it loaded fine no problems with expected small increase in overall memory in task manager. I will test this with more days data later on ie 10+. I just ran through the RC2 release log and did not see anything specifically mentioning tick replay fixes as it did seem to occur mostly with those data type settings ; can you share anything in particular or generally that was addressed in this area? It seems Gary is still experiencing issues for tick replay = true in his scenarios so i was curious what has fixed. Anything that can be shared would be appreciated.

The other point to add to the issues i was experiencing is that i was using mostly ES (as per gary's testing) which has many more ticks than other instruments and also the 6E.

Also i did not receive a response to my question on the NT8 multi threading policy in use to better understand the context and environment of NT8 architecture

thanks

[NinjaTrader_JessicaP]

Hello soulfx,

First in answer to your question, the documented information we can provide regarding multi-threading is here,

http://ninjatrader.com/support/helpGuides/nt8/en-us/multi-threading.htm?zoom_highlightsub=threading

I would be happy to clarify any of the information on that page.

I was able to review our back end ticketing system, the enhancements that affected Tick Replay directly were all documented in the release notes page,

http://ninjatrader.com/support/helpGuides/nt8/en-us/8_0_0_14.htm

Please bear in mind that NinjaTrader is a complicated application whose parts all share code, and that work on a part completely unrelated to Tick Replay can have an impact on tick replay. To summarize the reports known to affect Tick Replay :

• 10452 : Playback controller was sometimes disabled while connecting
• 10415 : Tick Replay was not correctly disabled for Renko or LineBreak
• 10189 : Tick replay should no longer run on bid/ask data series

I have used the ES and 6E instruments in my own test scenarios as well. Please let us know if there are any other ways we can help.

[soulfx]

Hello Jessica, thank you for the reply and i will read the links below.

Today i launched N8 with my 10 tabs 7 of ES 3 of 6E, 9 of these chart tabs use tick replay = true.
The memory seemed a bit high almost 1000MB as opposed to what seems avge 490-500MB

Ahead of the US open is started to climb as tick rate would increase and around 10 ET reached 1500MB. I elected to save workspace and close down normally. I then did a delete of DB/cache and restarted ... it has now been running for the last 1.5 hours at a steady 440MB - about what i would expect.

Given my 16GB can obviously handle this and even up to a sizeable chunk 1000 to 2000MB i believe i have sufficient memory and avoid loading too many browser pages.

Is there something with tick replay that is triggering this increase and in deleting the cache what is this doing such that it rebuilds interestingly even though i have ES and 6E there is only a directorty for CME Index futures and not one for 6E that i do see from time to time.

I will continue to observe and see if i get the same tomorrow when i relaunch;
If there were issues with the indicators i have written would i not be observing this ? It seems more related to tick replay functionality and its impact on the cache.

Any further information you require let me know
thanks

[NinjaTrader_JessicaP]

Thank you soulfx for this additional information. It will help us isolate what is happening if we can come up with the most basic scenario with the fewest variables we can. With that in mind, the built in indicators

• BuySellPressure
• BuySellVolume
• VolumeProfile

All use MarketDataType.Last and therefore are Tick Replay responsive. Therefore I would like to request that we restrict all our testing of the Tick Replay engine, itself, to these three indicators only.

Could you review my process in Post # 36, see if it produces a memory spike on your end, and let me know if I can do anything differently to produce a memory spike on my end using those three indicators?

[NinjaTrader_Brett]

We did identify a memory usage case with RC2 where if you would hover over the taskbar the preview image for charts would take tons of memory for each second the preview image was visible.. That will be resolved in the next beta.

We did internal testing with Tick Replay and memory usage and was unable to isolate any memory leaks or excessive memory usage cases. Note: Tick Replay is at its core memory intensive, so we're not looking for expected higher memory usage but unexpected usage in compared with RC1.

If anyone in this thread is still running into memory problems with RC2. Can you please contact me at support at ninjatrader dot com ATTN: Product management and reference this post. I'd like to work directly with you if possible since the above fix might solve some of the reports here but I'm concerned we might have another case I needed to look into that I have yet so far to reproduce and get a finger on.

Thank You!

[ashmanz80]

soulfx,

In addition to the information I requested previously and so that I may have a more complete picture of what is happening on your end, would it be possible to use the freely and publicly available screen capture software Jing, https://www.techsmith.com/jing.html , or a similar program which can record video of what is happening on your screen, so that you can show me what is occurring on your end? These quick instructions can get you started with Jing.

1. When you start Jing, a yellow half circle will appear near the top of your screen. Please hover over it with your mouse
2. A small + sign will extend out of this half circle. Please click on it
3. When your cursor becomes two intersecting lines, please click and drag your mouse over an area of your screen you would like to record
4. Please press the film strip button that appears to begin recording
5. When you are finished recording, please press the square stop button that appears
6. Please press the share button that appears. It consists of three vertical upward pointing arrows.

You will then have the option to view your video on screencast.com . This is the link which can help me diagnose what is happening on your end. It is possible you will need to set up a free account in order to share videos.




Thank you very much in advance for providing us with this information. If this procedure is not an option for you, or if there are any other questions we can answer, please let us know.

Hello,

Any update on this? I got 64GB RAM and when I do optimizations on tick data, it just clogs the memory slowly till it reaches 100% usage. I closed all the other windows and just kept Strategy analyzer and the command center, but the memory kept decreasing a rate of 1MB/sec. Please help, was about to purchase an additional 64GB RAM until I read others are having the same issues.

[NinjaTrader_JessicaP]

Hello ashmanz80,

There were several memory usage enhancements between release candidates, and there are more ongoing. These should address all of the concerns within this thread.

An actionable report here which we could use to improve NinjaTrader 8 and deliver better performance to you is one that

• is an instance where RC1 performance is superior to RC2, and
• is reproducable with the SampleMACrossover strategy, and
• is not addressed in the Help Guide documentation, for instance, Tick Replay's documentation, http://ninjatrader.com/support/helpG...ick_replay.htm

For comparing, you can download RC2 (8.0.0.14) and RC1 (8.0.0.13) here, http://ninjatrader.com/support/helpG...en-us/beta.htm


If it will help to demonstrate the difference, the instructions you quoted will make it easy to produce a video showing the differences. You can examine your memory usage in the Task Manager, which can be brought up with ctrl + shift + esc.


Should you encounter a situation that matches the above three criteria, please send

• an e-mail with your log and trace folders
• a picture of your strategy analyzer settings (instructions below)
• a picture of your system control panel
  • I like to bring this up by pressing Win + e, right-clicking My Computer (renamed This PC in Windows 10), and going to Properties
  • You may desire to color over any serial numbers and identifying information using your favorite image editor
    • I prefer Microsoft Paint, which you can bring up by pressing the Windows key to the left of your space bar and the letter r at the same time, typing mspaint into the window that comes up, and pressing enter

The above should go to platformsupport[at]ninjatrader[dot]com , and reference the unique ID number 1586003 . Please also include Attn:Product Management in the subject line of your e-mail.

[soulfx]

testing 123

[soulfx]

ok ninja SERIOUSLY i have typed 2 long messages and recieved the expired token and lost ALL text - please fix this and allow more time to type message - this is ridiculous