Add data series dynamically

[7robert]

I am adding a series of instruments to a strategy, more than 100 dynamically, that is, I bring the instruments from my database and with a for loop I add: AddDataSeries (i.Symbol, Data.BarsPeriodType.Minute, 1, Data .MarketDataType.Last); This of course takes time to process, but when entering OnMarketData to obtain the Ask and Bid of each series, it seems to behave differently, sometimes it generates 50 records, maybe 90 records.

So when I add the same strategy with another 100 different instruments, both running at the same time, the two strategies begin to show me fewer records, each of them does not exceed 50 records, even though I am passing 100 different instruments to each one, it should be noted which is the same strategy but each one works with different instruments. I would like to know if there is any limit to the data series that I am trying to add, or if you can guide me on how I should do this.

Thank you so much.

[NinjaTrader_Jesse]

Hello 7robert,

Each data provider has different limits on what data you can access or how many instruments you can subscribe to at once. For this question it would be best to contact your data provider/broker to see what you have access to in regard to your account.

Another factor here is that AddDataSeries is not intended to be used in a dynamic way and will fail in various tools and use cases when used like that. This is documented in the AddDataSeries help guide page. For a real test you would need to hard code the 100 instruments in each strategy and then re try your test to see if that still happens. We cannot assist with dynamically adding data like that but if you find a problem when hard coding the instruments we could further review that.

I look forward to being of further assistance.

[7robert]

Thank you very much, I have already updated my data provider which had a limit of 100, I hired one with a limit of 500, now, I also put each data series manually and the same thing happens when you enter OnMarketData in the first minute out of 100 instruments I get 96 records, and from the second minute it generates much less, I speak of a range between 0 and 20.


In this way, I am adding each series:
AddDataSeries("MMM", Data.BarsPeriodType.Minute, 1, Data.MarketDataType.Last);
AddDataSeries("CVX", Data.BarsPeriodType.Minute, 1, Data.MarketDataType.Last);


In my OnMarketData method something like this:

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
if (BarsInProgress == 0)
return;

if (State == State.Realtime)
{
Print("Instrument FullName: " + Instrument.FullName);

if (marketDataUpdate.MarketDataType == MarketDataType.Ask)
{
ask = marketDataUpdate.Price;
Print("Ask: " + marketDataUpdate.Price.ToString());
}

if (marketDataUpdate.MarketDataType == MarketDataType.Bid)
{
bid = marketDataUpdate.Price;
Print("Bid: " + marketDataUpdate.Price.ToString());
}


Print("Spread: " + Math.Round((ask - bid), 4).ToString());
}
}

[NinjaTrader_Jesse]

Hello 7robert,

Are you able to open a market analyzer will all these instruments and see that they all have active data at the time you are testing? It would not be expected to see all of the instruments providing ticks at the exact same time as they are all individual instruments and may have differences in the frequency of trading.

The market analyzer would likely be the easiest tool to test the subscriptions as you can see the prices eventually populate as new data for the instruments are received in realtime.
if you otherwise leave the script running for a while do you see all instruments report eventually?


I look forward to being of further assistance.

[7robert]

I have 100 instruments in series of minutes, additionally I created a timer in which I collect the data every hour and it happens exactly the same the first time it generates more records between 90 and 100, then the number of records for the next hour is less, I miss it This is why most of the instruments always generate data for me the first time, I would only need to review the market analyzer, but I think that if in 1 minute it generates 96 records, I do not see why in an hour it can not generate even 100.

Thank you very much for your cooperation in helping me with this problem.

[NinjaTrader_Jesse]

Hello 7robert,

I am not certain I understand what you are doing with the timer, the market data won't relate to the timer so that may or may not see some instruments when its run.

Are you able to use the market analyzer to see all instruments populate live data?

Have you also checked the Log tab of the control center to check if any subscriptions failed?

I look forward to being of further assistance.

[7robert]

Hello NinjaTrader_Jesse,

I have reviewed the market analyzer with all 100 instruments and only 1 instrument is not generating data ("MNTA").

On the other hand, every time I turn on these strategies, they take time to activate or even happen that they are not activated, in this case I have a message like this: Default Strategy Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.

Also in the log I always get messages like the following for each instrument added in the strategy:

Default 'ZB ## - ##' market data is delayed since your account does not have a real-time subscription for the exchange this market trades on.


Thanks again for your help.

[NinjaTrader_Jesse]

Hello 7robert,

Thank you for the reply.

With the strategy all instruments would need to be valid and have realtime data for it to work correctly, if you are seeing messages in the log surrounding specific instruments you would need to remove those from the AddDataSeries statements.

The error you are seeing would relate to something being null in the OnStateChange override. It would be difficult to say what that may be without seeing the code used. You can try commenting out the code you have added in OnStateChange to see when that error goes away to get a better idea of the cause.

I look forward to being of further assistance.

[7robert]

Hello Ninja_Trader_Jesse,

I am attaching the file of my strategy, it should be noted that I have two other strategies that are the same but with different instruments, the same thing happens, when I have the 3 in execution among all they do not generate more than 100 records, I have changed the code a little but the results they are the same

Thank you.!

[NinjaTrader_Jesse]

Hello 7robert,

Have you tried a more simple test and printed the BarsInProgress and Instrument without all of the timer logic you have?

protected override void OnBarUpdate()
{
Print(BarsInProgress + " " + Instrument);

While I can't really comment on the list and other custom logic you added I could suggest doing a more simple test to make sure you get a print for each instrument in realtime.

Are you otherwise seeing the same warnings in the log? If so have you tried to remove those instruments from the strategy? Only realtime enabled instruments will drive the strategies logic.

Are you otherwise able to see all instruments being used between all of your strategies work at once in the market analyzer? If you are seeing that a maximum of 100 instruments can be subscribed to at once the market analyzer would easily be able to show that by having only 100 out of the max rows show data.



Please let me know if I may be of further assistance.

[7robert]

Hello NinjaTrader_Jesse,

f I have tried without any of the logic that happens in the strategy only with the AddSeries of the instruments and the Print (BarsInProgress + "" + Instrument); As I was telling you, when the strategy is running alone if it reaches the number of records of 100, it shows 98 or 96 that would be enough, but when I start running another strategy equal to that with 100 different instruments, then that's where they start to generate less records, I have added the instruments to the market analyzer and it's all good some instruments vary from time to time and do not show data but I suppose this is normal since then I see that they do show, the problem occurs when 2 strategies run at the same time I will give you the examples, for this case I have added an additional series of 65 seconds to be able to count the times that the series of 1 minutes passed through the OnBarUpdate.

Thank you very much for helping me with this.

[NinjaTrader_Jesse]

Hello 7robert,

Thank you for the post.

You mentioned that some instruments do not get data in the analyzer or get it very late, are you including those instruments in the strategies? If so that may be part of the problem, if the instrument is unable to load data it will not process and may cause the strategy to have difficulty loading the other data. If you exclude any instruments which is not getting data do you still see the same result?

If they are getting really slow data you also should not expect those to be reported before your BIP 101 condition, they may happen infrequently and not in sync with the other condition.

You are counting for any series calling OnBarUpdate which would very unlikely reach 100 before your other condition resets it.

if(BarsInProgress == 101)
{
Print("Count SPREAD 1: " + countIns);
countIns = 0;
}

if(State == State.Realtime)
{
Print(BarsInProgress + " " + Instrument);
countIns += 1;  // unlikely to count the correct number of series before the other condition resets it

}

Have you tried to just print the BarsInProgress and do no other logic like the print I had suggested?

protected override void OnBarUpdate()
{
Print(BarsInProgress + " " + Instrument);

If any instrument does not call OnBarUpdate before your BIP 101 condition then it would not work to count like this. For this type of count all instruments would need to happen in the exact same order which is not possible in realtime. You would need to run the script for a while and collect the output. You could then review if each BarsInProgress index was reported at some point during the entire output you collected. An example would be if an instrument does not see an update for more than 60 seconds, it would not be included in your count print but would show up in the general BarsInProgress indexes that are being printed.



Please let me know if I may be of further assistance.