• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

NT8 Memory Optimization Issues

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    NT8 Memory Optimization Issues

    Hi!

    I have written a Strategy in NT8, but am experience stability issues due to excessive memory use. In order to get around that, I'd like some help understanding the architecture NT8 uses when it comes to data streams, as well as somebody perhaps pointing out to me what's the recommended approach to handle a large number of data series with medium amounts of historical data.

    First of all, is there an undocumented option (or a workaround) to set MaximumBarsLookBack to something other than 256 or Infinite? My system requires about 2000 bars lookback to do some calculations, so don't really need infinite, but 256 does not work either.

    My trading system uses about 60 distinct data streams (stocks added via AddDataSeries in my code), if I run a few instances of the system on different stocks (but the same stock data streams are used internally between all), I end up with 70-80 distinct data streams in total. My memory consumption goes to over 11 GB in just a few days. It usually grows really high on the weekend when there is no incoming data -- could this be an NT8 bug?

    11 GB sounds like too much as even several years worth of data for all the stocks should be under 2 GB.

    Does NT8 automatically recognize that different strategies are requesting the same data and instead of instantiating multiple copies it uses a single one?

    Is there a way to force NT8 to truncate the data series to a value (other than 256)?

    Is this expected behavior or is there something that I've missed out in my own code that's resulting in this massive memory leak?

    Any suggestions and thoughts welcome!

    Thanks!

    #2
    Hello dvance,

    Thank you for your note.

    Does the strategy create new objects such as drawing objects on each new bar or in OnMarketData?

    Is this occurring when running in real-time?

    If you do not need to call more than 256 bars in the past, leave MaximumBarsLookBack set to TwoHundredFiftySix for efficiency. There are no other settings for this.

    Below is a public link to the help guide.
    https://ninjatrader.com/support/help...rslookback.htm


    There is an instrument thread that does subscriptions for real-time data, NinjaScript will be consumers that thread will feed.

    Objects being created may be causing the memory growth.
    Chelsea B.NinjaTrader Customer Service

    Comment


      #3
      Hi Chelsea,

      The strategy does not create new drawing objects at all.

      Yes the problem occurs when running in real time.

      I do need to call more than 256 bars in the past.

      Could you please walk me through how 60 real-time data streams could use up 11+ GB of RAM in less than 48 hours (especially on the weekend when the stock exchange is closed)? Is that something you've tested on your platform -- in case it is and it has behaved differently, then I should spend more time looking through my own code to make sure it's not some other issue. Having said that my code is rather simple in terms of variable generation etc. so the likelihood of a memory leak this large I believe is small.

      Comment


        #4
        Hello dvance,

        60 added instruments means OnBarUpdate triggers for every instrument.
        Logic must be run each time OnBarUpdate (or other data driven methods) runs.
        New objects being created (such as ints, doubles, strings, orders, drawing objects, datetime objects, etc) in OnBarUpdate means those objects will be added to memory.

        A script with no logic at all in OnBarUpdate is unlikely to grow to 11GB in memory.

        I'm happy to create a test script that adds 60 instruments and has no logic in OnBarUpdate or any other methods. I'll let you know the results.
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Hello dvance,

          I've created a script that adds 60 instruments, has no complex logic, does not create new objects, and prints the instrument name on each new session (just to show the script is still working and triggering OnBarUpdate()). No other workspaces, charts, or scripts were running during this test. The connection was Kinetick which did not experience any disconnects.

          I've allowed this to run from Thursday through the weekend. The memory did not grow in size.

          This lets us know that adding 60 instruments to a script in of itself does not cause a memory leak.

          Attached are screenshots of the test when it was enabled, one taken after hours during the weekend, and one after the session start today.
          Attached Files
          Chelsea B.NinjaTrader Customer Service

          Comment


            #6
            Hello dvanc,

            Attached is the script I have tested if you also want to test this with all workspaces closed, all other charts closed, and no other indicators or scripts running to see if your memory grows.
            Attached Files
            Chelsea B.NinjaTrader Customer Service

            Comment

            Latest Posts

            Collapse

            Topics Statistics Last Post
            Started by aligator, Today, 10:11 PM
            0 responses
            1 view
            0 likes
            Last Post aligator  
            Started by psquare, Today, 03:11 PM
            4 responses
            18 views
            0 likes
            Last Post psquare
            by psquare
             
            Started by ballboy11, Today, 03:20 PM
            1 response
            12 views
            0 likes
            Last Post NinjaTrader_ChelseaB  
            Started by tonynt, Today, 02:18 PM
            1 response
            10 views
            0 likes
            Last Post NinjaTrader_JoshG  
            Started by denalek2419, Today, 01:46 PM
            3 responses
            12 views
            0 likes
            Last Post NinjaTrader_ChristopherJ  
            Working...
            X