OnBarUpdate not called in order for multiple data series?

[ntdev]

I'm seeing a strange behavior which causes my internal Series<double> variable to assert with 'System.ArgumentOutOfRangeException'

Repro:
1. New custom indicator with input series GC 12-20 and the following code

private Series<double> rawCalcs;
...
if (State == State.SetDefaults)
    AddPlot(...
...
if (State == State.Configure)
    AddDataSeries("[B]EURGBP[/B]"

protected override void OnBarUpdate()
    Print(string.Format("OnBarUpdate(Bar#{0}; Current={1}) => Close={2}", BarsInProgress, CurrentBar, Close[0]));

    if (BarsInProgress == 1)  rawCalcs[0] = Close[0];      // *** ERROR - exception, ArgumentOutOfRangeException

Actual:
OnBarUpdate(Bar#1; Current=0) => Close=1.305
OnBarUpdate(Bar#1; Current=1) => Close=1.30508
OnBarUpdate(Bar#1; Current=2) => Close=1.3055
OnBarUpdate(Bar#1; Current=3) => Close=1.30501
OnBarUpdate(Bar#1; Current=4) => Close=1.30501
OnBarUpdate(Bar#1; Current=5) => Close=1.30493
OnBarUpdate(Bar#1; Current=6) => Close=1.30498
OnBarUpdate(Bar#1; Current=7) => Close=1.30576
OnBarUpdate(Bar#1; Current=8) => Close=1.30542
OnBarUpdate(Bar#1; Current=9) => Close=1.30487
OnBarUpdate(Bar#1; Current=10) => Close=1.30477
OnBarUpdate(Bar#1; Current=11) => Close=1.30492
OnBarUpdate(Bar#0; Current=0) => Close=1925.8
OnBarUpdate(Bar#1; Current=12) => Close=1.30554
OnBarUpdate(Bar#0; Current=1) => Close=1928.2
OnBarUpdate(Bar#1; Current=13) => Close=1.30565
OnBarUpdate(Bar#0; Current=2) => Close=1928.2
OnBarUpdate(Bar#1; Current=14) => Close=1.3055
OnBarUpdate(Bar#0; Current=3) => Close=1930.4
OnBarUpdate(Bar#1; Current=15) => Close=1.30567
OnBarUpdate(Bar#0; Current=4) => Close=1928.3
OnBarUpdate(Bar#1; Current=16) => Close=1.30551
OnBarUpdate(Bar#0; Current=5) => Close=1929.7
OnBarUpdate(Bar#1; Current=17) => Close=1.30542

Expected:
I would expect BarsInProgress to be alternated between 0 and 1. I would not expect for Bar#1 to be called with data 12 times before Bar#0 starts getting called with data.


Any idea why this would be happening?

[NinjaTrader_Jim]

Hello ntdev,

Thanks for your post.

We should expect historical data to be played back following timestamps of each bar. This may not necessarily alternate between BarsInProgress 0, BarsInProgress 1, etc.

Something also to note is that plots are synchronized to the primary data series. Plot values should be assigned on BarsInProgress 0, as there would be another slot added to the Plot's series at that point.

I have attached a couple examples for how historical data is unrolled in a Multi Time Frame script, and I have attached an example for assigning plot values in BarsInProgress 0.

We look forward to assisting.

[ntdev]

Thanks, Jim, for your response and thanks for the sample.

But, both the main series and the secondary series have the same time frame in my example and I want to make calculations from multiple values.

Closes[0].Bars.BarsPeriod
{5 Minute}
BarsPeriodType: Minute
BarsPeriodTypeName: null
BarsPeriodTypeSerialize: 4
BaseBarsPeriodType: Minute
BaseBarsPeriodValue: 1
MarketDataType: Last
PointAndFigurePriceType: Close
ReversalType: Tick
Value: 5
Value2: 1
VolumetricDeltaType: BidAsk

Closes[1].Bars.BarsPeriod
{5 Minute}
BarsPeriodType: Minute
BarsPeriodTypeName: null
BarsPeriodTypeSerialize: 4
BaseBarsPeriodType: Minute
BaseBarsPeriodValue: 1
MarketDataType: Last
PointAndFigurePriceType: Close
ReversalType: Tick
Value: 5
Value2: 1
VolumetricDeltaType: BidAsk

Another note is that using a different secondary instrument like ES 09-20 does not have the same problem -- the problem (for me) being that CurrentBar get's way off between the 2 different series. It would be nice if CurrentBar stayed in sync for date/time for the 2 different series.

While looking at the data I noticed that
TradingHoursArray[0] = Nymex Metals - Energy ETH
TradingHoursArray[1] = Forex
from my first example.

Closes[0].Bars.BarsSeries
{Instrument='GC 12-20' From='2020-08-12' To='2020-08-17' period='5 Minute' Count='1031' firstTime=8/11/2020 6:05:00.000 PM lastTime=8/17/2020 10:55:00.000 AM}
BarsPeriod: {5 Minute}
BarsType: {NinjaTrader.NinjaScript.BarsTypes.MinuteBarsType}
Count: 1031
DayCount: 0
EstimatedFutureBarTimeDiff: 0
FromDate: {8/12/2020 12:00:00 AM}
Id: "3a4cbce632fb428983d43bd8b048fa40"
Instrument: {GC 12-20 Nymex}
IsDisposed: false
...
IsRolloverAdjusted: true
...
LastBarTime: {8/17/2020 10:55:00 AM}
LastIncludesEndTimeStamp: true
LastPrice: 1995
PercentComplete: 0
SyncRoot: {System.Threading.ReaderWriterLockSlim}
TickCount: 0
TicksOnLastSecond: 0
ToDate: {8/17/2020 12:00:00 AM}
TotalTicks: 63452
TradingHours: {Nymex Metals - Energy ETH}

Closes[1].Bars.BarsSeries
{Instrument='GBPUSD' From='2020-08-12' To='2020-08-17' period='5 Minute' Count='1004' firstTime=8/11/2020 5:05:00.000 PM lastTime=8/17/2020 4:40:00.000 AM}
BarsPeriod: {5 Minute}
BarsType: {NinjaTrader.NinjaScript.BarsTypes.MinuteBarsType}
Count: 1004
DayCount: 0
EstimatedFutureBarTimeDiff: 0
FromDate: {8/12/2020 12:00:00 AM}
Id: "0e1367b599cd45c1819224a876039b18"
Instrument: {GBPUSD Default}
IsDisposed: false
...
IsRolloverAdjusted: false
...
LastBarTime: {8/17/2020 4:40:00 AM}
LastIncludesEndTimeStamp: true
LastPrice: 1.30875
PercentComplete: 0
SyncRoot: {System.Threading.ReaderWriterLockSlim}
TickCount: 0
TicksOnLastSecond: 0
ToDate: {8/17/2020 12:00:00 AM}
TotalTicks: 2122
TradingHours: {Forex}

I guess that my solution will be to put everything into a has table (by time stamp) and then do all my own painting in OnRender.

[NinjaTrader_Jim]

Hello ntdev,

The timestamps for the bars will make the difference for how the historical data gets unrolled. Using the same time frame on a different instrument does not necessarily mean there are bars in that data series at that time.

Please reference the bar timestamps when comparing how historical data gets unrolled.

Please write back and let us know if you are seeing bar timestamps that are coming out of order.

[ntdev]

Thanks Jim. The timestamp part clears up everything.

Adding the timestamp to my log shows...

OnBarUpdate(Bar#1; Current=0) => Close=1.30289 || time=8/12/2020 5:15:00 PM
OnBarUpdate(Bar#1; Current=1) => Close=1.30291 || time=8/12/2020 5:30:00 PM
OnBarUpdate(Bar#1; Current=2) => Close=1.3031 || time=8/12/2020 5:45:00 PM
OnBarUpdate(Bar#1; Current=3) => Close=1.30297 || time=8/12/2020 6:00:00 PM
OnBarUpdate(Bar#0; Current=0) => Close=1929.3 || time=8/12/2020 6:15:00 PM
OnBarUpdate(Bar#1; Current=4) => Close=1.30322 || time=8/12/2020 6:15:00 PM
OnBarUpdate(Bar#0; Current=1) => Close=1931.4 || time=8/12/2020 6:30:00 PM
OnBarUpdate(Bar#1; Current=5) => Close=1.30342 || time=8/12/2020 6:30:00 PM
OnBarUpdate(Bar#0; Current=2) => Close=1932.1 || time=8/12/2020 6:45:00 PM
OnBarUpdate(Bar#1; Current=6) => Close=1.30351 || time=8/12/2020 6:45:00 PM
OnBarUpdate(Bar#0; Current=3) => Close=1926 || time=8/12/2020 7:00:00 PM
OnBarUpdate(Bar#1; Current=7) => Close=1.30362 || time=8/12/2020 7:00:00 PM

So, now I get it. Even though CurrentBar is not in sync for each series, the timestamps come perfectly in order.

I'll just handle the part about different data turning on and off at different times by doing everything with time and creating my own "Series" which can be a dictionary to accumulate the values that I'll paint in OnRender().