Abysmal Performance on Large Scans

[Phoenix2518]

My strategy requires large scans in order to find uncommon, short-lived patterns and notify me via email. I ran a test to see how fast NinjaTrader could load 1,000 stocks using the following code:

namespace NinjaTrader.NinjaScript.MarketAnalyzerColumns
{
    public class TestColumn : MarketAnalyzerColumn
    {
        protected override void OnStateChange()
        {
            if (State == State.DataLoaded) {
                CurrentValue = 0;
            }
        }
    }
}

I opened a new Market Analyzer, loaded my imported instrument list of exactly 1,000 NASDAQ stocks, and added just this one column. From the time the column first displayed to the time it was populated with all zeroes and "Loading..." no longer showed in the upper left was approximately 12 minutes. Considering this is the simplest possible script, 12 minutes is pathetic.

I tried the same test back-to-back in Thinkorswim. Standard market hours, using the exact same hardware, with the exact same stocks and columns: Bid, Ask, Last, and the test column (plot x = 0). Thinkorswim took 20 seconds flat, meaning NinjaTrader is 36 times slower.

Would anyone care to explain this difference? Considering I purchased a $180 quarterly license for the sole purpose of running large scans like Thinkorswim, but with email notifications on any stock in the scan? The one thing their platform cannot do?

[NinjaTrader_Jim]

Hello Phoenix2518,

Thanks for your post.

You are not setting IsDataSeriesRequired to false, so you are effectively adding 1000 data series when you add this column which will certainly cause a performance impact.

Please try the following as a test:

namespace NinjaTrader.NinjaScript.MarketAnalyzerColumns
{
    public class TestColumn : MarketAnalyzerColumn
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                IsDataSeriesRequired    = false;
            }
            else if (State == State.DataLoaded)
            {
                CurrentValue = 0;
            }
        }
    }
}

IsDataSeriesRequired documentation is publicly available here - https://ninjatrader.com/support/help...esrequired.htm

Please let me know if you have any additional questions.

[Phoenix2518]

Setting IsDataSeriesRequired to false causes the column to display almost instantly, but it also removes the ability to process historical data using OnBarUpdate(). Displaying a static value was just a test - my actual script requires processing the last 15 minutes or so of historical data, which makes this change effectively useless.

You mentioned 1,000 data series objects being responsible for a performance hit. I decided to perform one more test, creating a column that sums the last 15 1-minute bars to make sure both platforms are processing at least that much data.

Thinkorswim:

plot x = close[0] + close[1] + close[2] + close[3] + close[4] + close[5] + close[6] + close[7] + close[8] + close[9] + close[10] + close[11] + close[12] + close[13] + close[14];

NinjaTrader:

namespace NinjaTrader.NinjaScript.MarketAnalyzerColumns
{
    public class TestColumn : MarketAnalyzerColumn
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Calculate = Calculate.OnBarClose;
                BarsPeriod.BarsPeriodType = BarsPeriodType.Minute;
                BarsToLoad = 18;
            }
        }

        protected override void OnBarUpdate()
        {
            if (State == State.Historical || Bars.Count < 15) return;

            CurrentValue = Close[0] + Close[1] + Close[2] + Close[3] + Close[4] + Close[5] + Close[6] + Close[7] + Close[8] + Close[9] + Close[10] + Close[11] + Close[12] + Close[13] + Close[14];
        }
    }
}

Same circumstances: Back-to-back, with the same hardware, stocks, and columns. Thinkorswim slowed down noticeably, taking 1 minutes 13 seconds to populate the column. NinjaTrader, meanwhile, had only populated 75% of the column after 20 minutes. I gave up and don't know how long it would've taken to actually finish.

This is ridiculous. I guess I'll try accumulating data from OnMarketData and handling it myself, since NinjaTrader is incapable of coming anywhere close to Thinkorswim on its own.

[NinjaTrader_Jim]

Hello Phoenix2518,

NinjaTrader will download a minimum of one day of data with minute bars. (Please see GetInitialLookBackDays for MinuteBarsType.) For a perfect apples to apples comparison, you may need to specify loading 1 day to load with ThinkOrSwim.

As long as we are testing the same amount of data loaded, with the same calculation, we could break down further to the data provider's impact by testing another connection technology. As NinjaTrader offers different connection technologies, and different brokers and data providers offer different bandwidth for historical data requests, the connection we use could have a large factor in this equation. We have tested your example on our end with Kinetick and received a time of 2:08 on 1000 stocks.

You could consider testing your example with a fast exclusive data provider like Kinetick, and noting the behavior where we are actually loading at least 1 day of data per request could help for your comparisons.

If there is any other insight you would like us to offer, please let us know.

[Phoenix2518]

So let me get all this straight:

- In column settings, we have barsToLoad. One would expect this to be the initial number of bars downloaded, and documentation seems to agree. According to your comments, this value is completely ignored.

- We also have MaximumBarsLookBack, which I left at 256. According to the docs,

"Only the last 256 values of the series object will be stored in memory and accessible for reference."

"A MaximumBarsLookBack.TwoHundredFiftySix series works as a circular ring buffer, which will "loop" when the series reaches full capacity. Specifically, once there are 256 entries in the series, new data added to the series overwrite the oldest data."

One would expect Bars.Count to match barsToLoad at startup, then fill up to MaximumBarsLookBack before looping around. This seems to match the docs and observed behavior. However, the number of 1-minute bars in a day would be 1440, so again, this value is blatantly ignored according to your comments.

- As mentioned, Bars.Count first matches barsToLoad, not the 1440 bars that are supposedly loaded for the last 24 hours. Where is all of this other data stored? Why is not accessible from the primary class that handles historical data?


Loading a day's worth of data for charts makes sense, but you're telling me that mountains of data are loaded behind the scenes for every instrument in a Market Analyzer, contradicting highly visible settings that apparently do nothing at all. That is the definition of unpredictable and inefficient.

Whatever, I'm done running comparisons. Your earlier advice about IsDataSeriesRequired ended up being helpful after all, since creating my own small buffers is far more efficient than whatever NinjaTrader was doing. It's not having any trouble at all with 2000 stocks, and I plan on doubling or tripling that next week. Guess I won't be opening that dispute with PayPal after all.

[NinjaTrader_Jim]

Hello Phoenix2518,

We are looking at a few different concepts here.

1. DaysToLoad/BarsToLoad (data requested by the script)
2. Amount of data requested to be downloaded by NinjaTrader (data requested through the data provider.)
3. MaximumBarsLookBack (how far back a Series object can be referenced historically)

Let's use your provided sample code as a case study.

BarsToLoad is set to 18, so the script is requesting 18 bars of data. Within the BarsType, when the request is made the BarsToLoad is converted to a number of DaysToLoad to be requested from the provider. NinjaTrader adds this logic to BarsTypes specifically for cases involving Tick based bars like Renko Bars where we cannot easily request a certain number of bars because their could be a varying number of ticks involved in the creation of each bar.

Historical data is then downloaded and stored locally in the db folder. The extra data is not included in the BarsToLoad request for the script because the script is mentioning it does not need it and wants to start building its Bars object with that number of bars. The data can be found when opening the Historical Data Manager.

Your understanding of MaximumBarsLookBack is correct, but should not be confused with the data requested by the script and the data requested from NinjaTrader.

I have noted your concerns in our internal ticket tracking feedback behind improving historical data loading with the ticket ID is SFT-2291. Impact is tracked before enhancements are considered for implementation so we cannot offer an ETA or promise of fulfillment. Your input is tracked, however.

Let us know if there is anything else that requires clarification.