BarsInProgress dilema

[fx_pete]

I have coded an indicator that loads a number of data series into BarsArray and performs some calculations. When the indicator is loaded the user can select a date for which the calculations will be done. The indicator finds the bar corresponding to the date and performs the calculations on all the series in BarsArray. Once all the series have been processed (BarsInProgress == BarsArray.Length-1) the result is displayed.

This works fine under most circumstances however there is one problem:

For example I have an open chart with one data series displayed. The indicator is then loaded. (there are 10 series in BarsArray). The data series displayed on the chart (BarsInProgress=0, primary) has 5000 days (bars) loaded with bars visible going back to 1999. However some of the other data series start at later dates.

If I select a date when all data series have bars on the screen then everything works. However when the date is such that some of the data series do not have any bars on the screen yet then my check (BarsInProgress == BarsArray.Length-1) fails.

Basically there are 10 series in BarsArray but OnBarUpdate() will be called for example with BarsInProgress == 0,1,4,7,8 corresponding to the series with bars actually present by the chosen date.

I need to run the calculations for the series in BarsArray (in this case 0,1,4,7,8) and once the last series has been processed, display the output. That is what I was trying to achieve by BarsInProgress == BarsArray.Length-1.

Can someone please help me figure out how to do this with BarsInProgress not incrementing by 1 each time until it equals the BarsArray Length?

Any help would be greatly appreciated.

Thanks,

Peter

[NinjaTrader_Matthew]

Peter,

To better assist, would you be able to provide us with a sample script of the check that you're doing? I'd like to run this on our end to ensure we're correctly addressing the problem you're facing and not making any assumptions in your logic.

[fx_pete]

I appreciate your willingness to help but I don't want to post it because it may generate other questions, unrelated to the problem I have. If there is anything unclear about my description of the issue please feel free to ask. I believe that problem can be understood from my description and looking at the code at this point will only help to confuse. I will post the code but only as the last option.

Basically I need a way of knowing when I have reached the last series in the BarsArray for which bars exist on the chart ... remember I have 10 series added but only 4 of them go back far enough for my date selection. So when OnBarUpdate() runs & I print out BarsInProgress it will give something like 0,1,4,7,8 ... obviously OnBarUpdate() does't run when there are no bars displayed for the series, or does it?

[koganam]

I appreciate your willingness to help but I don't want to post it because it may generate other questions, unrelated to the problem I have. If there is anything unclear about my description of the issue please feel free to ask. I believe that problem can be understood from my description and looking at the code at this point will only help to confuse. I will post the code but only as the last option.

Basically I need a way of knowing when I have reached the last series in the BarsArray for which bars exist on the chart ... remember I have 10 series added but only 4 of them go back far enough for my date selection. So when OnBarUpdate() runs & I print out BarsInProgress it will give something like 0,1,4,7,8 ... obviously OnBarUpdate() does't run when there are no bars displayed for the series, or does it?

Sounds like you need to use a manually incremented loop counter indexed to process every BarsArray element. Skip those where the BarsArray element is null. At the end, check that the loop count matches the number of elements that you are processing.

[sledge]

I have coded an indicator that loads a number of data series into BarsArray and performs some calculations. When the indicator is loaded the user can select a date for which the calculations will be done. The indicator finds the bar corresponding to the date and performs the calculations on all the series in BarsArray. Once all the series have been processed (BarsInProgress == BarsArray.Length-1) the result is displayed.

This works fine under most circumstances however there is one problem:

For example I have an open chart with one data series displayed. The indicator is then loaded. (there are 10 series in BarsArray). The data series displayed on the chart (BarsInProgress=0, primary) has 5000 days (bars) loaded with bars visible going back to 1999. However some of the other data series start at later dates.

If I select a date when all data series have bars on the screen then everything works. However when the date is such that some of the data series do not have any bars on the screen yet then my check (BarsInProgress == BarsArray.Length-1) fails.

Basically there are 10 series in BarsArray but OnBarUpdate() will be called for example with BarsInProgress == 0,1,4,7,8 corresponding to the series with bars actually present by the chosen date.

I need to run the calculations for the series in BarsArray (in this case 0,1,4,7,8) and once the last series has been processed, display the output. That is what I was trying to achieve by BarsInProgress == BarsArray.Length-1.

Can someone please help me figure out how to do this with BarsInProgress not incrementing by 1 each time until it equals the BarsArray Length?

Any help would be greatly appreciated.

Thanks,

Peter

BarsArray[0].Count ??
BarsArray[1].Count ??
BarsArray[4].Count ??
BarsArray[7].Count ??
BarsArray[8].Count ??

[fx_pete]

screen shots might help ...

thanks for the suggestions everyone but it seems like some screen shots are in order...

Screen 1 is a capture of my indicator working.
- Chart of SPY is loaded (5000 days) along with my indicator
- My indicator loads other data series into BarsArray, 10 total so in effect you would get something like screen 2.
-My indicator works by selecting a date; then returns, among other things, of 63 days and 20 days ago (from the selected date) are computed for each data series in the BarsArray.
-My indicator then checks if BarsInProgress == BarsArray.Length-1, ie that all the series have been processed and then displays the results in the indicator pane.
-This works for dates going back a few years, however I start encountering problems when I select a date for which only some of the dataseries (ETFs) have bars (ie ETFs don't exist yet)

This picture is shown in screen 3. Under this condition, the BarsArray still has all 10 dataseries, however since there are no actual bars around this date, OnBarUpdate() is only called for the series that have bars plotted.

So in screen 3 the BarsInProgress would assume the following values: 0,3,5,6. Now with the final value being 6 my check BarsInProgress == BarsArray.Length-1 fails.

Any suggestions on how work around this? May be I am missing something simple but this is the 1st time I have encountered this problem and NT doesn't make it easy by not providing any documentation...

[NinjaTrader_Matthew]

Hello,

I'm not sure what documentation you're referring to, but we have a plenty of resource on the following archive:

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?alphabetical_reference.htm

You can read more about mulit-instrumet handling from the following link:

http://www.ninjatrader.com/support/helpGuides/nt7/multi_time_frame__instruments.htm

You can use a CurrentBars[x] check to see if the bars array you're testing on contains data or not:

http://www.ninjatrader.com/support/helpGuides/nt7/currentbars.htm

I'm not quite sure what you're checking with BarsInProgress == BarsArray.Length-1. Can you elaborate on this?

[koganam]

thanks for the suggestions everyone but it seems like some screen shots are in order...

Screen 1 is a capture of my indicator working.
- Chart of SPY is loaded (5000 days) along with my indicator
- My indicator loads other data series into BarsArray, 10 total so in effect you would get something like screen 2.
-My indicator works by selecting a date; then returns, among other things, of 63 days and 20 days ago (from the selected date) are computed for each data series in the BarsArray.
-My indicator then checks if BarsInProgress == BarsArray.Length-1, ie that all the series have been processed and then displays the results in the indicator pane.
-This works for dates going back a few years, however I start encountering problems when I select a date for which only some of the dataseries (ETFs) have bars (ie ETFs don't exist yet)

This picture is shown in screen 3. Under this condition, the BarsArray still has all 10 dataseries, however since there are no actual bars around this date, OnBarUpdate() is only called for the series that have bars plotted.

So in screen 3 the BarsInProgress would assume the following values: 0,3,5,6. Now with the final value being 6 my check BarsInProgress == BarsArray.Length-1 fails.

Any suggestions on how work around this? May be I am missing something simple but this is the 1st time I have encountered this problem and NT doesn't make it easy by not providing any documentation...

This is the code that implements my suggestion. Let me know what gives.

[FONT=Courier New]
[/FONT][FONT=Courier New][COLOR=#0000ff][FONT=Courier New][COLOR=#0000ff]for[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New] ([/FONT][FONT=Courier New][COLOR=#0000ff][FONT=Courier New][COLOR=#0000ff]int[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New] index = [/FONT][FONT=Courier New][COLOR=#800080][FONT=Courier New][COLOR=#800080]0[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New]; index < BarsArray.Length; index++) 
{
[/FONT][FONT=Courier New][COLOR=#0000ff][FONT=Courier New][COLOR=#0000ff]if[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New] (CurrentBars[index] < [/FONT][FONT=Courier New][COLOR=#800080][FONT=Courier New][COLOR=#800080]0[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New]) [/FONT][FONT=Courier New][COLOR=#0000ff][FONT=Courier New][COLOR=#0000ff]continue[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New];
[/FONT][FONT=Courier New][COLOR=#0000ff][FONT=Courier New][COLOR=#0000ff]else
[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New]{
[/FONT][FONT=Courier New][COLOR=#008000][FONT=Courier New][COLOR=#008000]//Do the calculations here
[/COLOR][/FONT][/COLOR][/FONT][FONT=Courier New]}
}
[/FONT]

[fx_pete]

OK, first off I am not referring to the docs for supported functionality. I am referring to docs for unsupported functionality. It would have been beneficial to get docs. for the entire object model with descriptions of functionality and usage. NT can still choose not to support questions about it but having that document would make it possible to try to do a lot more. Anyway as for the docs. you listed it actually helped me to read some of them again, so I will see if I can now solve my problem. As for not understanding why I use BarsInProgress == BarsArray.Length-1 .... as I already wrote twice before, I was operating under the assumption that BarsInProgress would be incremented by 1 for each data series in BarsArray. Since I have 10 series loaded once BarsInProgress == 9 I would know that all data series have been processed; exactly what I wrote in this thread 2x before. In other words when 1st I began writing the indicator I did not imagine a scenario where a date would be selected for which bars do not exist yet for some of the series. needles to say I did not realize that BarsInProgress will stay at 0 until the it encounters a date for which bars from series at index 1 start at which point BarsInProgress will go thru 0,1 for each bar.... I assumed that BarsInProgress will go 0 thru 9 for every bar because there are 10 series in BarsArray ....

So do I need to be any clearer? Is there any one else that finds my explanation confusing or is it just Mathew... I don't know may it is me

[fx_pete]

yes that is what I was thinking after reading some of the docs suggested by Mathew .... I will give it a shot and hopefully that will be it

thanks kogaman!

[fx_pete]

still can't get it to work ...

so as my last option I am including the entire indicator with instructions. I will also include a code snippet below and highlight the section that is causing me problems. May be this time Mathew will be able to help.

I have attached the script and the data file I use for it. Before installing this indicator you must have Excel Primary Interop Assemblies installed

go HERE and read post #1. Follow instructions to install the assemblies before installing the indicator (I have included the dll in the DataFiles attachment in case you are having problems installing it). Or you can just look at the cs file without running the indicator. If you choose to install it then read on:

Install the script as usual. Then create a data directory under bin/custom/indicators and place the excel file there. When running the script you must tell the indicator the path to the data file. You can do that each time manually when you run the indicator in the path parameter or a much better way would be if you would load the indicator code and go to the variables section and redefine the path there. Compile the indicator again and this time it should show as default under the indicator parameters.


Once again guys I thank you in advance for any help you can provide. Somehow I thought that the solution provided by koganam would have worked but I had problems with implementing it.

        protected override void OnBarUpdate()
        {		
			
			// make sure we have the right bars period type
			if (BarsPeriod.Id != PeriodType.Day && BarsPeriod.Value < 1)
				return;	
			
			// make sure there are enough bars
			if (CurrentBar <= GetMaxPeriod()) // in this case <= because [barsago+1] below
				return;

                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]

				double distribA = etf[BarsInProgress].ComputeDistributions(Time[barsagoA], Time[0]);
				double distribB = etf[BarsInProgress].ComputeDistributions(Time[barsagoB], Time[0]);
				
				// get the 1st return
				double rA = etf[BarsInProgress].GetReturn(Close[barsago], Close[barsagoA]) + distribA;
				
				// get the 2nd return
				double rB = etf[BarsInProgress].GetReturn(Close[barsago], Close[barsagoB]) + distribB;

                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]

				// volatility calculation ...
				// a) get the individual returns for the defined period and 
				// b) store them in the data set
				double distribV = 0.0;
				double[] vreturns = new double[periodV];
				for (int i = barsago + (periodV-1); i >= 0; i--) // itterate down from 19 to 0 values ago from barsago (20 values)
				{
					distribV = 0.0;//etf[BarsInProgress].ComputeDistributions(Time[barsagoV]);
					vreturns[i] = ((Close[i] - Close[i+1]) / Close[i+1]) + distribV;
				}

                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]

				// compute  volatility
				// a) using standard deviation and 
				// b) anualize it by multipling it by 252 
				//    where 252 = trading days in a year 
				double v = etf[BarsInProgress].GetVolatiltiy(vreturns, 252);
							
				// save results to etf data
				etf[BarsInProgress].SaveResults(rA, rB, v);

			
				// with all series saved, go ahead computing the rank by ...
			
             [SIZE="4"][B]if (BarsInProgress == BarsArray.Length-1)[/B][/SIZE]
				{
				
					// remove any invalid ETFs
					etf.RemoveAll( delegate (EtfData item) 
						{ 	// by checking if the values are 0
							return item.ReturnA < Double.Epsilon && item.ReturnB < Double.Epsilon && item.Volatility < Double.Epsilon;
						}
					);
							
					// create copies of data for sorting
					List<EtfData> sortedByA = new List<EtfData>(etf);
					List<EtfData> sortedByB = new List<EtfData>(etf);
					List<EtfData> sortedByV = new List<EtfData>(etf);
					
					/// sort by returnsA, returnsB and then by volatility
					/// - for returns need high to low, hence need to reverse default sort order
				
					sortedByA.Sort( delegate (EtfData item1, EtfData item2) 
									{ 
										return item1.ReturnA.CompareTo(item2.ReturnA); 
									}
					);
                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]
                       [SIZE="7"]  .[/SIZE]

[koganam]

so as my last option I am including the entire indicator with instructions. I will also include a code snippet below and highlight the section that is causing me problems. May be this time Mathew will be able to help.

I have attached the script and the data file I use for it. Before installing this indicator you must have Excel Primary Interop Assemblies installed

go HERE and read post #1. Follow instructions to install the assemblies before installing the indicator (I have included the dll in the DataFiles attachment in case you are having problems installing it). Or you can just look at the cs file without running the indicator. If you choose to install it then read on:

Install the script as usual. Then create a data directory under bin/custom/indicators and place the excel file there. When running the script you must tell the indicator the path to the data file. You can do that each time manually when you run the indicator in the path parameter or a much better way would be if you would load the indicator code and go to the variables section and redefine the path there. Compile the indicator again and this time it should show as default under the indicator parameters.


Once again guys I thank you in advance for any help you can provide. Somehow I thought that the solution provided by koganam would have worked but I had problems with implementing it.

        protected override void OnBarUpdate()
        {        
 
            // make sure we have the right bars period type
            if (BarsPeriod.Id != PeriodType.Day && BarsPeriod.Value < 1)
                return;    
 
            // make sure there are enough bars
            if (CurrentBar <= GetMaxPeriod()) // in this case <= because [barsago+1] below
                return;
 
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
 
[COLOR=red]              double distribA = etf[BarsInProgress].ComputeDistributions(Time[barsagoA], Time[0]);[/COLOR]
[COLOR=red]              double distribB = etf[BarsInProgress].ComputeDistributions(Time[barsagoB], Time[0]);[/COLOR]
 
[COLOR=red]              // get the 1st return[/COLOR]
[COLOR=red]              double rA = etf[BarsInProgress].GetReturn(Close[barsago], Close[barsagoA]) + distribA;[/COLOR]
 
[COLOR=red]              // get the 2nd return[/COLOR]
[COLOR=red]              double rB = etf[BarsInProgress].GetReturn(Close[barsago], Close[barsagoB]) + distribB;[/COLOR]
 
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
[COLOR=red]                     [SIZE=7].[/SIZE][/COLOR]
 
[COLOR=red]              // volatility calculation ...[/COLOR]
[COLOR=red]              // a) get the individual returns for the defined period and [/COLOR]
[COLOR=red]              // b) store them in the data set[/COLOR]
[COLOR=red]              double distribV = 0.0;[/COLOR]
[COLOR=red]              double[] vreturns = new double[periodV];[/COLOR]
[COLOR=red]              for (int i = barsago + (periodV-1); i >= 0; i--) // itterate down from 19 to 0 values ago from barsago (20 values)[/COLOR]
[COLOR=red]              {[/COLOR]
[COLOR=red]                  distribV = 0.0;//etf[BarsInProgress].ComputeDistributions(Time[barsagoV]);[/COLOR]
[COLOR=red]                  vreturns[I] = ((Close[I] - Close[i+1]) / Close[i+1]) + distribV;[/I][/I][/COLOR]
[I][I][COLOR=red]              }[/COLOR][/I][/I]
 
[I][I][COLOR=red]                     [SIZE=7].[/SIZE][/COLOR][/I][/I]
[I][I][COLOR=red]                     [SIZE=7].[/SIZE][/COLOR][/I][/I]
[I][I][COLOR=red]                     [SIZE=7].[/SIZE][/COLOR][/I][/I]
 
[I][I][COLOR=red]              // compute  volatility[/COLOR][/I][/I]
[I][I][COLOR=red]              // a) using standard deviation and [/COLOR][/I][/I]
[I][I][COLOR=red]              // b) anualize it by multipling it by 252 [/COLOR][/I][/I]
[I][I][COLOR=red]              //    where 252 = trading days in a year [/COLOR][/I][/I]
[I][I][COLOR=red]              double v = etf[BarsInProgress].GetVolatiltiy(vreturns, 252);[/COLOR][/I][/I]
 
[I][I][COLOR=red]              // save results to etf data[/COLOR][/I][/I]
[I][I][COLOR=red]              etf[BarsInProgress].SaveResults(rA, rB, v);[/COLOR][/I][/I]
 
 
[I][I][COLOR=red]              // with all series saved, go ahead computing the rank by ...[/COLOR][/I][/I]
 
[I][I]           [SIZE=4][B]if (BarsInProgress == BarsArray.Length-1)[/B][/SIZE][/I][/I]
[I][I]              {[/I][/I]
 
[I][I]                  // remove any invalid ETFs[/I][/I]
[I][I]                  etf.RemoveAll( delegate (EtfData item) [/I][/I]
[I][I]                      {     // by checking if the values are 0[/I][/I]
[I][I]                          return item.ReturnA < Double.Epsilon && item.ReturnB < Double.Epsilon && item.Volatility < Double.Epsilon;[/I][/I]
[I][I]                      }[/I][/I]
[I][I]                  );[/I][/I]
 
[I][I]                  // create copies of data for sorting[/I][/I]
[I][I]                  List<EtfData> sortedByA = new List<EtfData>(etf);[/I][/I]
[I][I]                  List<EtfData> sortedByB = new List<EtfData>(etf);[/I][/I]
[I][I]                  List<EtfData> sortedByV = new List<EtfData>(etf);[/I][/I]
 
[I][I]                  /// sort by returnsA, returnsB and then by volatility[/I][/I]
[I][I]                  /// - for returns need high to low, hence need to reverse default sort order[/I][/I]
 
[I][I]                  sortedByA.Sort( delegate (EtfData item1, EtfData item2) [/I][/I]
[I][I]                                  { [/I][/I]
[I][I]                                      return item1.ReturnA.CompareTo(item2.ReturnA); [/I][/I]
[I][I]                                  }[/I][/I]
[I][I]                  );[/I][/I]
[I][I]                     [SIZE=7].[/SIZE][/I][/I]
[I][I]                     [SIZE=7].[/SIZE][/I][/I]
[I][I]                     [SIZE=7].[/SIZE][/I][/I]

I see at least one logic error. Your code seems to be assuming that there is a tick coming in for every bar series on each bar of the primary bar series. Not true. Therefore, your last filtered code block's code only executes whenever a tick comes in on the last bar series that you added, which tick has nothing to do with ticks that come in on the primary bar series. You need to use a filter that actually mirrors where you want to process data.

Better yet, as that block removes invalid data, you should probably be processing it on every bar. Therefore, I would use FirstTickOfBar as the filter.

The other data, I would simply iterate through with a for loop as I described, just to be sure that on each pass, I processed all valid barSeries. All the code in red would be in the for loop, indexed by a counter that loops through the data (per the construct that I already gave), not directly by BarsInProgress, as data may not exist for some BarsInProgress.

[fx_pete]

Big thanks goes to ...

KOGANAM

I have to admit for some reason I found this difficult to understand but with koganam's help I think I finally got it

here is what my OBU() looks like now ...

        protected override void OnBarUpdate()
        {
			// make sure we have the right bars period type
			if (BarsPeriod.Id != PeriodType.Day && BarsPeriod.Value < 1)
				return;	
	
			if (!excelDataLoaded) // first bar of primary series
			{
				
				// Read Distributions Data		
				if (!ReadExcel())
				{
					Alert("openError", NinjaTrader.Cbi.Priority.High, "Error opening spreadsheet - check indicator parameters",
						"Alert1.wav", 10, Color.Black, Color.Yellow);				
					
					Print("!!!!!!!!!!!!!!!!!!!!!");
					Print("CAN'T READ EXCEL FILE");
					Print("!!!!!!!!!!!!!!!!!!!!!");
					Print("");
				}
				else
				{
					excelDataLoaded = true;
				}				
			}		
			
			if (!FirstTickOfBar) 
				return; // process data only bar by bar
			
			if (CurrentBar <= GetMaxPeriod())
				return; // proceed further only if enough bars are present
	
	[COLOR="Blue"]		if (!finished)
			{
				int endbar = Bars.GetBar(date);
			
				if (CurrentBar == endbar)
				{// perform computation once CurrentBar # and the date bar # match	
					
					// process all data series added to BarsArray
					[COLOR="DarkRed"]for (int index = 0; index < BarsArray.Length; index++)
					{
						if (CurrentBars[index] < 0) // no bars exist
						{
							continue; // skip to next series
						}	
					
						[I][B]>>> proceed with calculations ...[/B][/I]

					
					} // endfor processing data series[/COLOR]

					finished = true;
					
				} // endif CurrentBar == endbar
			
			} // endif !finished[/COLOR]
				
        } // end OnBarUpdate()

[fx_pete]

ETF Replay - Final Version

Ok I am going to post the final version since some people have requested it. There are some differences when compare to numbers from ETF Replay but the ranking for top 3 is not effected in 98% of the cases. There may be times where close rankings such as 4 and 5 or 7 and 8 would be switched because of the slight differences. Return A and Return B #s are matching in most cases at least to the 1st decimal place. Volatility #s are a little off. I could never figure out why; perhaps someone will find an error with the way I calculate it. I will also attach a spreadsheet that compares the accuracy between ETF Replay and the script for year 2008 & 2009.

All the installation instructions in post #12 still apply (make sure you delete any old versions first).

Also there is now the posibility to disable adding the distributions to the calculation (see indicator params)

Note: this indicator was never meant for public consumption so don't email me complaining that there is no error checking

Attachment 1) Indicator
Attachment 2) Interopp assem. for those who have problem installing it
- Excell distributions data
- Excell file with results comparison to ETF Replay