Correlation indicator: instrument vs. itself

[digibob]

Hi. Please see example of correlation between SPY and SPY. I would expect it to be 1 (continuous. irrespective of period). What am I missing?

[NinjaTrader_Jim]

Hello digibob,

Thanks for your post.

The Correlation indicator adds a data series for the secondary instrument and then compares this with the primary. Since the data series are not processed at the same time, comparing the same instrument will result in the added data series being a bar behind.

We can observe if we modify the indicator so the previous bar value of the primary series is used instead of what is read from the secondary, we will see the same results as the Correlation indicator when comparing the same data series.

We can also observe if we modify the indicator so that the current bar value of the secondary series is used and the previous value of the primary data series is used, we will get the flat line which you expect.

We look forward to assisting.

[digibob]

Hi Jim. Thanks for explaining and providing examples. It's clear now, I'm just not sure the implementation of the indicator is what one would expect. At least I didn't. You could argue for leading-lagging correlation but then how is the user expected to know there is leading-lagging here and which of the two instruments is which without looking in the code, and why one bar and not two bars or three, etc. it's all rather arbitrary. Is there a doc page explaining this?

More to the point, is there a way to compare the current bars of both series without having to modify the indicator? I mean via the way of passing in the series parameters. I'm guessing there is no way.

[NinjaTrader_Jim]

Hello digibob,

There wouldn't be documentation specifically warning users of this, but this would be a general understanding involving Multi Time Frame or Multi Instrument scripts where additional data series are processed in a script sequentially and are not processed simultaneously.

As indicators work in general, we should expect that plots are synchronized to the primary data series, and each data series is processed in a separate update in OnBarUpdate. If we involve another data series for comparisons and assign plots on the primary data series, one data series will have a bar before the other.

More to the point, is there a way to compare the current bars of both series without having to modify the indicator? I mean via the way of passing in the series parameters. I'm guessing there is no way.

No, there is not. Since each data series is processed separately, you would have to modify it like I have if you are comparing the same data series. However, comparing the primary data series to the same data series wouldn't give us anything useful, and with the way the indicator works now, it compares the primary data series value with the most recent value processed from the secondary series.

Let us know if you have any additional questions.

[digibob]

Hi Jim. From experience working with multi instruments and time frames I know it's possible to compare bars across different series when all the bars involved are current, same timestamp, none is prior. I simply do all the work on the last BarsInProgress and if needed I use SessionIterator. The point is it's doable, but it's too small a point for discussion. All I'm saying is the Correlation indicator's current implementation may be lost on some users. More often than not correlation is used without leading-lagging, meaning each pair of price values share the same timestamp. This is what people expect by default from a Correlation indicator.

Anyway I appreciate your help. Thanks.

[johnnymustard]

Hello all,

Is it possible to modify the NT8 correlation to compare a security to an indicator rather than another security?
For example, compare CL to a 20 period ema. I'd expect them to be positively correlated most of the time.
When they are negatively correlated (or no correlation) that is at a minimum a pause in the market.
No correlation or negative values are what reversals will look like too though.

The price will reverse before the correlation plot catches up. It tends to show up glaringly on sharp reversals.

I had something that did this in NT7 and it was effective at finding reversals. I haven't been able to find an NT8 version.
Seems to me to be an easy upgrade, the code is already there. It just needs to be able
to add an indicator instead of a security. Easy fix, right? Or is it more complicated?

I'd appreciate any guidance on this point.
I thank you & good day, JM

[NinjaTrader_Jim]

Hello johnnymustard,

The indicator does compare the two data series using an SMA. You could change Each SMA so an EMA is used if you like, or use another indicator all together.

The SMA is called on line 66 and 76.

EMA syntax - https://ninjatrader.com/support/help...onential_e.htm

If you would like to change it so the first SMA uses the primary data series, you can change the BarsArray it uses to BarsArray[0] and have it processed on BarsInProgress 0.

Multi Time Frame and Instruments - https://ninjatrader.com/support/help...nstruments.htm

We look forward to assisting.

[KirkHammett]

Hello digibob,

Thanks for your post.

The Correlation indicator adds a data series for the secondary instrument and then compares this with the primary. Since the data series are not processed at the same time, comparing the same instrument will result in the added data series being a bar behind.

We can observe if we modify the indicator so the previous bar value of the primary series is used instead of what is read from the secondary, we will see the same results as the Correlation indicator when comparing the same data series.

We can also observe if we modify the indicator so that the current bar value of the secondary series is used and the previous value of the primary data series is used, we will get the flat line which you expect.

We look forward to assisting.

Hi Jim!

Could you please share the code on the modifications you did to the Correlation Indicator to get the flat line when comparing the same instrument?

Do I understand it correctly that also when comparing two different instruments, the Correlation Indicator lags 1 bar due to the multi timeframe behavior? There is no difference if it is the same or two different instruments right?

Thanks!
Kirk

[NinjaTrader_Jim]

Hello Kirk,

I had just modified line 76: avg0 = SMA(BarsArray[0], Period)[1];

This was just a demonstration, a simple condition that checks if Instruments[0] and Instruments[1] match to set the plot to 1 could also be considered.

In Multi Time Frame scripts, OnBarUpdate is called separately for each data series. We would need to wait for the OnBarUpdate iterations of the other data series for that value to be current. Since plots are synchronized to the primary data series, the next most up to date plot seen would come after the next OnBarUpdate iteration for the primary series. This would be 1 bar if using Calcvulate.OnBarClose, but can still give more up to date values with less lag using Calculate.OnPriceChange.

Let me know if you have any additional questions.

[cliffhu]

Hi Jim,

If you modified avg0 = SMA(BarsArray[0], Period)[1] to use the previous primary bar, shouldn't you modify the following part as well? so SMA[1] is compared with Inputs[0][1] rather than Inputs[0][0]

for (int i = 0; i < Period; i++)
{

nominator += (avg0 - Inputs[0][i+1]) * (avg1 - Inputs[1][i]);
denominator1 += (avg0 - Inputs[0][i+1]) * (avg0 - Inputs[0][i+1]);
denominator2 += (avg1 - Inputs[1][i]) * (avg1 - Inputs[1][i]);

}

I know in fact this giving wrong result, but where is problem?

[NinjaTrader_Jim]

Hello cliffhu,

The modification I posted were simply to demonstrate that since multi time frame scripts do not process OnBarUpdate at the same time for multiple data series, we would not see a flat line unless we modified the indicator to compare the previous value of the primary data series to the secondary.

The changes were not made to "fix" anything. Instead of checking if you have a correlation of 1 to see if you are comparing the same data series, I suggest comparing BarsArray[X].Instrument.FullName and BarsArray[X].BarsPeriod to see if the data series you want to compare are the same.

Instrument - https://ninjatrader.com/support/help...instrument.htm

BarsPeriod - https://ninjatrader.com/support/help...barsperiod.htm