RemoveLastBar() question

[Harry]

There has been a thread in the Historical Beta section on this issue, but it is closed. Therefore I am opening a new one.

I understand that there are bar types such as Renko bars that use "RemoveLastBar()" and which cannot be used with tick replay. I further understand that I am able to check the current bar type via "BarsArray[0].BarsType.IsRemoveLastBarSupported" for this property.

Following questions:

(1) The description in the help guide is rather meagre. It does not explain what this property is all about.

(2) If I look at a few system indicators, they check for the property IsRemoveLastBarSupported().

I just had a look at the WMA which uses two different algorithms depending on whether the bar type holds that property "IsRemoveLastBarSupported". This is confusing to me.

As far as I can see

- all bar types will work with both algorithms
- the first algorithm is a slow algorithm that loops through the last elements of the Input series for every single tick
- the second algorithm is a recursive algorithm which is faster and does not use any loops
- the second algorithm is also inefficient, because it performs a few unnecessary calculations with every tick which can be done for the first tick only

Why would I use the first algorithm at all? The code does not make any sense to me.

Or is there anything that I did not understand?

[NinjaTrader_ChelseaB]

Hello Harry,

Thank you for your note.

I will forward your inquiry to the NinjaTrader Development and I will let you know what I am able to find.

We appreciate your patience.

[NinjaTrader_ChelseaB]

Hello Harry,

I am still waiting for discussion with our development on this.

However, I've asked why this is used before and got some information back.

Please see the following.

http://ninjatrader.com/support/forum/showthread.php?p=501367#post501367

The idea is that there is "faster" logic that interferes with bar types that use RemoveLastBar().

[Harry]

Hello Harry,

I am still waiting for discussion with our development on this.

However, I've asked why this is used before and got some information back.

Please see the following.

http://ninjatrader.com/support/forum/showthread.php?p=501367#post501367

The idea is that there is "faster" logic that interferes with bar types that use RemoveLastBar().

This is exactly the point. I do not understand why the logic would be faster. If I look at the code of the WMA, the supposedly fast logic looks much slower than the standard logic which is used for regular bar types.

The "fast logic" adds an unnecessary iteration over N = period bars, where the "slow logic" just uses a few simple calculations.

Also I do not understand how the logic interferes with the bar types that use RemoveLastBar() such as Renko bars.

The help guide does not give any explanations either.


Thank you for your answer. I am patientlly waiting for an explanation.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your response.

The code is considered more efficient when ran over Tick Replay. However if a bars type removes the last bar via RemoveLastBar() it is technically not supported in Tick Replay and therefore results in the need for different logic, that being the logic that is similar to NinjaTrader 7. (Renko, LineBreak, PointAndFigure bars all use RemoveLastBar())

Essentially, you don't want to add to a sum on each tick if the bar is going to change it's Open, High, Low, or Close at some point. The check is there to ensure the bars types that re-draw do not utilize logic that is optimized for Tick Replay.

This is listed in the Notes for the "Developing for Tick Replay" section of the Help Guide: http://ninjatrader.com/support/helpG...ick_replay.htm

Please let me know if you have any questions.

[Harry]

Essentially, you don't want to add to a sum on each tick if the bar is going to change it's Open, High, Low, or Close at some point. The check is there to ensure the bars types that re-draw do not utilize logic that is optimized for Tick Replay.

Please let me know if you have any questions.

Thank you for your explanations. This was helpful as I think I have a better understanding now of what is RemoveLastBar(). If I look at a Renko bar it can build below the prior bar or above the prior bar. As the change between above and below may happen intra-bar, the bar needs to be reset because

- there may be a different opening price (changed intra-bar)
- there may be a higher low (changed intra-bar)
- there may be a lower high (changed intra-bar)

This also means that any indicator which uses intra-bar logic such as

- determinig the opening price with the first tick
- relying on the low being lower than the original open

will fail

- and either produce false results
- or produce different results when used with Calculate.OnEachTick compared to Calculate.OnBarClose

Also Renko bars cannot be backtested because the backtested fills are nonsense. All this is known and can been addressed by separating OHLC from the visuals. OHLC should represent the actual price ticks during the life of the bar, while Renko is just a visual style. The solution was BetterRenko and all the problems were gone.


Coming back to WMA

If I look at the code of the WMA, I still do not see any reason why the first code section below would be necessary:

if (BarsArray[0].BarsType.IsRemoveLastBarSupported)
{
    if (CurrentBar == 0)
        Value[0] = Input[0];
    else
    {
        int back = Math.Min(Period - 1, CurrentBar);
        double val = 0;
        int weight = 0;
        for (int idx = back; idx >= 0; idx--)
        {
            val += (idx + 1) * Input[back - idx];
            weight += (idx + 1);
        }
        Value[0] = val / weight;
    }
}

The second bracket should also work with bars that support RemoveLastBar(). It is my understand that the second section of the code works much faster. So why has this bracket been introduced into the code. Sorry, I don't get it.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience.

IsRemoveLastBarSupported == true Bars Types, like Renko, can update data between bars (and thus can't be used in Tick Replay). In the SMA indicator, when IsRemoveLastBarSupported == false and we have a strong guarantee that data between bars can not change, once Period bars have elapsed, we see this code:

double last = Value[1] * Math.Min(CurrentBar, Period);
Value[0] = ((last + Input[0]) / (Math.Min(CurrentBar, Period) + 1));

When IsRemoveLastBarSupported is false, and we might be able to modify building or previous bars, we find the following:

sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
Value[0] = sum / (CurrentBar < Period ? CurrentBar + 1 : Period);

In other words, when our old bars can be trusted never to change, we calculate everything on the spot with bar data that remains reliable intra-bar, and no information needs to be saved between bars inside the sum or priorSum variables. When bars themselves can change as we're building a new bar however, we store data externally in the SMA indicator itself between bars in sum and priorSum. We then reset this data every new bar with the following:

if (IsFirstTickOfBar)
    priorSum = sum;

To sum up everything simply :

• When IsRemoveLastBarSupported == true, intra-bar data can be modified
• When it's false, it can't be, and is instead "write once, read forever" like a burned CD

Please let me know if you have any questions.

[Harry]

Hello Patrick,

Thank you for your detailed explanations. Much appreciated!

I now applied both algorithms to Renko charts, and as you stated the first algorithm works with Renko bars, while the second does not work.

However, I was able to see something strange.

When I apply the second algorithm

if (IsFirstTickOfBar)     
          priorSum = sum;
sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0); 
Value[0] = sum / (CurrentBar < Period ? CurrentBar + 1 : Period);

to Renko bars, it shows a very odd behaviour:

Let us call the current bar X, for example X = 283. When the current bar closes, NinjaTrader recalculates the value for X. With the first tick of the bar X + 1 = 284, the bar X = 283 is being recalculated and the indicator value for X = 283 is being changed.

So far, so good. What comes totally unexpected is that when the bar X = 283 is closed, NinjaTrader also changes the value for bar X - 1 = 282.


This looks like a mega bug!

Why does NinjaTrader recalculate the indicator values for two consecutive bars? It is completely unnecessary to recalculate two indicator values in a row, when a new Renko bar opens.I know that values for Renko bars may be repainted, but I do not expect that indicator values are being repainted two bars.

This behavior makes Renko bars totally unuseable!

[Harry]

IsFirstTickOfBar not working with Renko bars

I have now run a test on my chart to find out more about the problem.

You can do this by just looking which bars are being processed when FirstTickOfBar is used.

This is what I found:

FirstTickOfBar! ... CB = 76257 06.07.2017 18:07:41
FirstTickOfBar! ... CB = 76258 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76259 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76258 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76259 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76260 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76259 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76260 06.07.2017 18:07:50
FirstTickOfBar! ... CB = 76261 06.07.2017 18:07:50

This means that each Renko bar has three first ticks!

If you have six bars A B C D E F, they are processed

A B C B C D C D E D E F etc.

and all of those are first ticks. The first tick of each bar being processed three different times.

This is ridiculous!

( I do not want to offend anyone, I just cannot believe it.)

Obviously, if IsFirstTickOfBar is true several times for each bar, then it cannot work the way it is supposed to do.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience.

This is expected behavior as RemoveLastBar() will cause any indicator calling the bars type to trigger OnBarUpdate to run again for the prior bar. You would need to be aware of this for your script when calling a bars type that re-draws it's bars to ensure you track the CurrentBar and do not re-calculate in OnBarUpdate.

Please let me know if you have any questions.

[Harry]

Hello Patrick,

Thank you very much for your patience and the support you have provided.

This discussion has helped me a lot to get a better understanding how exotic bar types are being processed. I will review everything that I have coded in order to make sure that it will work with exotic bar types that support RemoveLastBar().

This is quite a good summary:

This is expected behavior as RemoveLastBar() will cause any indicator calling the bars type to trigger OnBarUpdate to run again for the prior bar.

There is one question that remains:

I do understand that OnBarUpdate() is running again for the prior bar. However, from my tests running any indicator on Renko bars with Calculate.OnEachTick I have observed that Renko bars trigger OnBarUpdate() for the two preceding bars - not just the prior bar.

This is what I am astonished about. Why is it necessary to run OnBarUpdate() for the two prior bars again?

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your patience.

Upon further research we have found that this behavior is very deep into the core logic of NinjaTrader. As this is the case, it is not something that we are going to re-visit in the foreseeable future. It is expected behavior for bar types that remote that last bar in order to rebuild the current bar.

Please let me know if I may be of further assistance.

[Harry]

Primarily I did not want to change that behavior. I just wanted to understand what is happening.

Thanks to your help and this thread I have now gained a better understanding.


If I look at the documentation for RemoveLastBar() and IsRemoveLastBarSupported() the implications of this feature are not explained. There are no examples. If I look at the section on best practices, it is not particularly helpful!


Best practice or worst practice?

Here is something that I have taken from the NinjaTrader 8 help guide.

Best practice

// dedicated logic to cache the prior sum on each tick of bar
// While it is a good practice, this can cause problems for bar types which may remove last bar (see below)
if(IsFirstTickOfBar)
priorSum=sum;

sum=priorSum+Input[0]-(CurrentBar>=Period?Input[Period]:0);
Value[0]=sum/(CurrentBar<Period?CurrentBar+1:Period);

Unfortunately, this is worst practice and misleading. This approach does not work with some bar types any more. This was best practice for NinjaTrader 7 and the example needs to be updated.

A correct implementation for NinjaTrader 8 would be:

// Variables
private double priorSum;
private Series<double> sum

// State == State.DataLoaded
sum = new Series<double>(this);

// OnBarUpdate
if(CurrentBar == 0)
     priorSum = 0;
else if(IsFirstTickOfBar)
     priorSum = sum[1];
 
sum[0] = priorSum + Input[0] - (CurrentBar>=Period? Input[Period] : 0);
Value[0] = sum[0] / (CurrentBar < Period? CurrentBar + 1 : Period);

As far as my understanding goes, this example should also work with Renko bars.

Could you please confirm that the code above works with bar types that support RemoveLastBar()? Or please correct me, if I am wrong!


Please update the documentation

Also it would be pretty helpful to update the sections on best practices and RemoveLastBar() in the help guide. I was pretty much trapped by this behavior and adapting my NinjaScript code to this behavior is a lot of work, as I have extensively used IsFirstTickOfBar to reduce CPU load during live trading - in particular during news releases.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your response.

// Variables
private double priorSum;
private Series<double> sum

// State == State.DataLoaded
sum = new Series<double>(this);

// OnBarUpdate
if(CurrentBar == 0)
     priorSum = 0;
else if(IsFirstTickOfBar)
     priorSum = sum[1];
 
sum[0] = priorSum + Input[0] - (CurrentBar>=Period? Input[Period] : 0);
Value[0] = sum[0] / (CurrentBar < Period? CurrentBar + 1 : Period);

I am not understanding how you see this to be the correct implementation. I can see why it might work as you hold the prior value in a series, but then again you have created a new series just to perform what one double was already doing. This would result in more memory being used then the current implementation.

The Best Practices are updated as needed. Can you provide the other items you feel need to be updated?

I look forward to your response.

[Harry]

Patrick,

Thank you again for your answer. I basically agree with your reasoning.

It is correct that I create a series to perform what a double was doing. However, the double does not work with Renko bars and other bar types that support RemoveLastBar() - because OnBarUpdate() processes those bars back and forth and therefore invalidates any approach to work with a double.

In order to find a solution that works with both standard and exotic bar types I am suggesting thr code, because I suppose that it works with all bar types.

The solution adopted takes a little bit more memory, but not much, because the newly added Series<double> can be set to MaximumBarsLookBack.TwoHundredFiftySix. After all only the two prior values need to be accessed inside OnBarUpdate(). In terms of CPU load the solution is nearly equivalent, as the prior value of the series is only called once for the first tick of each bar.

Therefore the solution is clean and fast and better than the approach that you have followed for the default SMA and WMA. In that approach everything needs to be coded twice - once for standard bar types and once for exotic bar types that support RemoveLastBar().

Hello Harry,

Thank you for your response.

I am not understanding how you see this to be the correct implementation. I can see why it might work as you hold the prior value in a series, but then again you have created a new series just to perform what one double was already doing. This would result in more memory being used then the current implementation.

The Best Practices are updated as needed. Can you provide the other items you feel need to be updated?

I look forward to your response.