Playback -> Calculate.OnEachTick -> OnBarUpdate called before transition to Realtime

[bltdavid]

One more thing.

In NT7 strategies, I also monitored using OnConnectionStatus, and when both connections
first showed ConnectionStatus.Connected, it triggered my OnFirstConnection, which
helped to find the FirstRealTimeBar. When FirstRealTimeBar and IsZombieLHB were true
at the same time, then I termed this state 'Petrified Data' and waited for 1 or more additional
bars before signal processing was allowed ... worked great on strategies w/Futures.

But, I hear you. Such research on trying to solve LHB -> FRB issues should never have been
necessary, and NT support, IMHO, never quite understood the depth and seriousness of this
issue, which they probably assumed has only ever been noticed by a handful of hardcore
NinjaScript users.

First off, it's a very difficult problem to explain. Why? Well, the lack of a common vocabulary
in the product to describe the events in play(*) tended to muddy the explanations of the issue.

For ex, in NT8, the state of the tick just processed -- is that reflected by State, or is State
more of the current processing state of the framework? I suppose the answer is: it depends.

Specifically, for the LHB -> FRB transition, I always wished to see something like this added
to the framework,

public enum TickState { Historical, RealTime, FirstRealTime }
public TickState BarState = TickState.Historical;

which works for every BarType, even a 1-Tick DataSeries, and works for every tick, even
for OnEachTick/OnPriceChange.

When OnBarUpdate is called, BarState would be set appropriately to represent the tick
state of the current bar (aka, clump of 1 or more ticks) being processed.

TickState Meanings:

• Historical - all ticks in the bar being processed by this OnBarUpdate are historical.
• Realtime - all ticks in the bar being processed by this OnBarUpdate are realtime.
• FirstRealTime - 1 or more of the ticks in this bar are realtime.

How would BarState work?

OnBarClose: This one is easy. If BarState == TickState.FirstRealTime then this closed
bar contains a combination of both Historical and RealTime ticks, which obviously means
somewhere inside this bar was the first realtime tick from the connected data feed. This
is the "hybrid" bar. (Well, maybe. Actually, BarState.FirstRealTime doesn't say anything
about how many historical ticks are in the bar, which could be zero, it only means one or
more ticks in the bar are realtime.)

if (IsFirstTickOfBar && BarState == TickState.FirstRealTime)
{
    ... first closed bar containing realtime ticks being processed by this OnBarUpdate ...
}

OnEveryTick: Also easy. If IsFirstTickOfBar is true then BarState would have the same
meaning and follow the same semantics as OnBarClose. Otherwise, BarState would
reflect the actual state of the specific tick(s) be processed. In other words,

if (!IsFirstTickOfBar && BarState == TickState.FirstRealTime)
{
    ... OnEveryTick: first realtime tick being processing by this OnBarUpdate
    ... OnPriceChange: first price change resulting from a realtime tick
}

The second code snippet could only occur for OnEveryTick/OnPriceChange. Both would
require surfacing the NinjaScript framework concept of a "hybrid" bar (which may or may
not occur) -- a hybrid bar contains both historical and realtime ticks -- and may occur on
the primary bar series or any of the secondary data series added via NinjaScript.

Obviously, a 1-Tick data series would have no hybrid bars, since each bar contains just
one tick. And if the boundaries of the LHB -> FRB transition coincide perfectly between
the boundaries of 2 bars (which should be possible but relatively rare), the FRB could
conceivably consist of all realtime ticks. The LHB is 100% always historical ticks only.

I know I know, this is all just wishful thinking, but these design ideas could be useful to
help illustrate what the current framework is lacking. Again, just my 2˘.

(*) - For me, in NT7, this whole issue started because I saw CurrentBar have the same
value for 2 closed bars -- which (when it did happen ) only happened w/COBC=False.
My "zombie bar" and "petrified data" ideas were my attempts to isolate this occurrence,
so that I could ignore that first bar. But none of these ideas are even necessary if the
framework would suppress that "zombie" bar and/or make sure all CurrentBars[n] were
unique for both bars during the LHB -> FRB transitions of each bar series.

[NinjaTrader_ChrisL]

Hi digibob,

It seems you would need to work out something else for a large backtest. Tick Replay is the only solution to get both the correct first tick of the daily bar and the first real-time tick. I am able to get both of these with the code below:

private bool realTimeFlag = false;
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                ...
            }
            else if (State == State.Configure)
            {
                AddDataSeries(BarsPeriodType.Minute, 1440);
                AddDataSeries(BarsPeriodType.Second, 1);


            }
            else if (State == State.Transition)
            {
                Print("Transition");
                realTimeFlag = true;
            }
            else if (State == State.Realtime)
            {
                Print("Realtime");
            }
        }

        protected override void OnBarUpdate()
        {

            if (CurrentBars[0] < 0 || CurrentBars[1] < 0 || CurrentBars[2] < 0)
                return;

            if (BarsInProgress == 1 && IsFirstTickOfBar)

            {
                Draw.Dot(this, "OPENBAR"+CurrentBar, false, 0, Close[0], Brushes.Blue);
                Print(Times[1][0]+" | "+Times[2][0]+" | State="+State);
            }

            if(realTimeFlag)
            {
                Print("First tick in real time");
                realTimeFlag = false;
            }
        }

[digibob]

Hi all. I appreciate everyone's responses from start to end of this thread. I have no more power left to try to get the point across. This is me giving up.

Chris, thank you for the script. As you said, tick replay is the only solution to achieve all the goals here, and tick replay is not realistic for me for the reasons mentioned before. The large backtest is just one of them.

It is frustrating that tick replay is taken as a fitting solution and thus serves as a way to disregard/overlook the actual issue. The no-tick-replay behavior on transition is out of sync, buggy, lacking, whatever word fits. Tick replay is not the alternative to no tick replay.

Unfortunately things will stay the way they are and my sympathy for the next poor user who is going to stumble on this issue only to face a wall when trying to explain and get things changed.

bltdavid, I agree with- and relate to everything you said.
- This is more of an edge case that not many users come across, but it doesn't mean it should not be addressed and fixed.
- There is a lack of common vocabulary, and might I add, there is no lack of confusing terminology.
- TickState - that could be exactly the solution here and, in general, could help the strategy gain more control and the user better understanding of what is happening underneath. I see no reason why a strategy should not receive events/have access to info on this deeper layer.
But what do I know.

Hopefully someone is listening.

I'd like to end by saying all this talk, sometimes criticizing, doesn't make a dent in my sentiment towards NT. I love using NT and I find support dedicated and great in general. Simply (newsflash) not everything in life is the way you want it to be.

Thank you all again.