Playback -> Calculate.OnEachTick -> OnBarUpdate called before transition to Realtime

[digibob]

Chris, I removed old market replay data, cleared the cache, re-downloaded market replay data, re-ran the script with OnEachTick - same output. The dates on each line are the same (in line with what I expect and with what I've seen ever since start using NT, but different from your print), and, there is the strange tick before the strategy is enabled. Notice it comes in before midnight. The line in bold.

Output of running the indicator on the chart without a strategy:

Indicator: 13-Feb-20 17:00:00 | 13-Feb-20 17:00:00 | State=Historical
Indicator: 14-Feb-20 17:00:00 | 14-Feb-20 17:00:00 | State=Historical
Indicator: 17-Feb-20 13:00:00 | 17-Feb-20 12:59:59 | State=Historical
Indicator: 18-Feb-20 17:00:00 | 18-Feb-20 16:59:59 | State=Historical
Indicator: Transition
Indicator: Realtime

Output of running the strategy with the indicator (AddChartIndicator):

Strategy: 13-Feb-20 17:00:00 | 13-Feb-20 17:00:00 | State=Historical
Strategy: 14-Feb-20 17:00:00 | 14-Feb-20 17:00:00 | State=Historical
Strategy: 17-Feb-20 13:00:00 | 17-Feb-20 12:59:59 | State=Historical
Strategy: 18-Feb-20 17:00:00 | 18-Feb-20 16:59:59 | State=Historical
Strategy: 19-Feb-20 17:00:00 | 18-Feb-20 23:59:58 | State=Historical
Strategy: Transition
Enabling NinjaScript strategy 'Test17/192771183' ...
Indicator: Transition
Strategy: Realtime
Indicator: Realtime

As I mentioned earlier it's the same on live run, not exclusive to market replay.

[NinjaTrader_ChrisL]

Hello digibob, thanks for your reply.

I am getting prints like so on my end with NinjaTrader Continuum data feed:

2/24/2020 3:00:00 PM | 2/24/2020 2:59:59 PM | State=Historical
2/25/2020 3:00:00 PM | 2/25/2020 2:59:59 PM | State=Historical
2/26/2020 3:00:00 PM | 2/26/2020 2:59:59 PM | State=Historical
Transition
Realtime

Do you get the same if you run the script I attached in my previous post?

[digibob]

Hi Chris. Yes, I get the same. The print you post now is different from the ones you posted before. Please see your earlier posts #9 and #12 where the dates on each line are not the same, followed by an explanation why it is so, which took me off course, made me doubt my understanding of the mechanism. If having same dates on each line is 'correct' after all, then happily we can leave that and go back to the original issue. The reason for my opening the thread.

There is a tick that comes in too early. It's not about the dates in the lines in the print. It's not about the print at all. The print is just meant to show the tick but I see that it's there also without the print. It comes in before the strategy is enabled, before Calculate changes from OnBarClose to OnEachTick, before the state changes from historical to realtime. At least before the strategy is notified of the change.

The chronological sequence of events appears to be this:

First, the state changes from historical to realtime BEHIND THE SCENES. OnBarClose changes to OnEachTick. The strategy at this point is not aware of that.
Then, the first realtime tick comes in and is being read by the strategy. As far as the strategy is concerned, the state is still historical.
Then, and only then. the strategy is notified of the state change. ("Enable strategy")

Either some setting or definition at my end is messing things up or it's an undesired behavior of the platform.

[NinjaTrader_ChrisL]

Hello digibob, thanks for the follow up.

The strategy does not go from OnBarClose to OnEachTick, it will stay on OnEachTick throughout the entire historical run. The way the code is written makes it run on the open of every day bar, so your code is running only on the open of the daily bars. As for the second series, it will take all of the seconds from the beginning of the chart to the end second, then switch to real time mode. This might be easier to see if you add a 1 tick series to the chart and examine what is happening from the historical to realtime switch.

e.g.

{
                AddDataSeries(BarsPeriodType.Tick, 1);
            }
            else if (State == State.Transition)
            {
                Print("Transition");
            }
            else if (State == State.Realtime)
            {
                Print("Realtime");
            }
        }

        protected override void OnBarUpdate()
        {
            if(BarsInProgress == 1)
                Print(Times[1][0]);
        }

[digibob]

Hi Chris. I appreciate your patience with my poor wording I'm obviously failing to explain and putting focus on the wrong things. With respect I try another way, setting aside the timestamps, I will use characters to illustrate.

If the strategy were to run without ever transitioning to realtime, we'd see a pattern in the printed lines, the result of the same call to OnBarUpdate on every historical daily bar, something like this:

AAA AAA AAA
AAA AAA AAA
AAA AAA AAA

When the strategy runs and does transition to realtime we get this:

AAA AAA AAA
AAA AAA AAA
AAA AAA AAA
BBB BBB BBB
Enabling NinjaScript strategy

Whatever is causing the BBB print, be it the opening or closing of this and that bar of this and that series whatever, before considering anything, just the simple fact that the historical-state pattern suddenly changes, how can *anything* change before the strategy is enabled.

The correct sequence should be:

AAA AAA AAA
AAA AAA AAA
AAA AAA AAA
Enabling NinjaScript strategy
BBB BBB BBB

ie. the change, BBB, should come after the strategy is enabled.

To translate pseudo output to actual, please see the following script and its output:

public class Test19 : Strategy
{
    private bool isSessionJustClosed;

    protected override void OnStateChange()
    {
        if (State == State.SetDefaults)
        {
            Name                                = "Test19";
            Calculate                           = Calculate.OnEachTick;
        }
        else if (State == State.Configure)
        {
            AddDataSeries(Data.BarsPeriodType.Second, 86400);
            AddDataSeries(Data.BarsPeriodType.Second, 1);

            isSessionJustClosed = false;
        }
        else if (State == State.Transition)
        {
            Print("Transition");
        }
        else if (State == State.Realtime)
        {
            Print("Realtime");
        }
    }

    protected override void OnBarUpdate()
    {
        if (CurrentBars[0] < 0 || CurrentBars[1] < 0 || CurrentBars[2] < 0)
            return;

        if (State == State.Historical)    // historical implementation
        {
            if (BarsInProgress == 1)
            {
                Print("Task (1st of 2) performed on historical session close trading day " + Times[1][0].ToShortDateString());
                isSessionJustClosed = true;
            }
            else if (BarsInProgress == 2)
            {
                if (isSessionJustClosed)
                {
                    Print("Task (2nd of 2) performed on 1-second series after historical session close trading day " + Times[2][0].ToShortDateString());
                    Print("---------------------------");
                    isSessionJustClosed = false;
                }
            }
        }
        else if (State == State.Realtime)    // realtime implementation
        {
            if (BarsInProgress == 1 && IsFirstTickOfBar)
                Print("First realtime tick trading day " + Times[1][0].ToShortDateString());
        }
    }
}

output:

Task (1st of 2) performed on historical session close trading day 14-Feb-20
Task (2nd of 2) performed on 1-second series after historical session close trading day 14-Feb-20
---------------------------
Task (1st of 2) performed on historical session close trading day 17-Feb-20
Task (2nd of 2) performed on 1-second series after historical session close trading day 17-Feb-20
---------------------------
Task (1st of 2) performed on historical session close trading day 18-Feb-20
Task (2nd of 2) performed on 1-second series after historical session close trading day 18-Feb-20
---------------------------
Task (1st of 2) performed on historical session close trading day 19-Feb-20
Transition
Enabling NinjaScript strategy
Realtime
First realtime tick trading day 20-Feb-20


The strategy is enabled mid day 19-Feb. This is important.

The line printed just above Transition, that's the 'BBB' line in my above illustration. The call to OnBarUpdate that ends up printing this line is the change in the historical-state pattern. It is misplaced. If the state is still historical, this call can't possibly be there. It creates a blind spot. The strategy is not psychic. It doesn't know it's about to be enabled. There is no event callback prior to that. All the strategy sees is a call outside the historical-state pattern (the timing) but when it checks the state it's still historical. So "what gives?" the strategy asks.

The right place to trigger this call to OnBarUpdate is immediately after the strategy is enabled, so the strategy can capture and react to it in realtime. Where this call belongs.

More clarification:

When checking the time signature of that misplaced 'BBB' call, it is 17:00 (EST), which is to be expected. The actual time when this call is made, as seen by checking the 1-second series time signature, is when the strategy is enabled. This too is to be expected. The time signatures are ok. If you check just the time signatures you don't see the problem because that's not the problem. I focused on that earlier and confused you, my apologies. The problem is that this call is triggered a split-second before the strategy is enabled instead of a split-second after. It's a mid day 19-Feb historical-state call 'closing' the 19-Feb session before the strategy is enabled for the 19-Feb session. Surely that's weird.

I would like to stress that the point here is not to modify the script I provided in attempt to find ways around the issue. The script is just an example. The code flow in my actual strategy is a lot more complex and I try to work around this issue with session iterator and other means, not always successful. I provide the example to show that said call to OnBarUpdate is triggered prematurely, in historical state instead of in realtime state, creating a blind spot for the strategy.

I hope I'm making sense finally.

[NinjaTrader_ChrisL]

Hi digibob,

I wanted to let you know I am researching your question. I will reply here once I have a definitive answer.

Thank in advance for your patience.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your patience.

I asked my colleague on the issue and the fact of the matter is the "Enabling strategy" message is not synchronous with data processing and is not an indication of historical to real time transition. A print from OnStateChange() of the State would tell when the strategy transitions to real time.

[digibob]

Hi Chris. Thank you, but it's not about the print. It's about the state. That call to OnBarUpdate should trigger in realtime. Instead it's triggered in historical, outside the normal expected historical timing, right before transition to realtime. Why is it there? Can you explain the event that triggers a call in historical state right before transition to realtime?

If the state is historical, there is no reason for anything to happen there.

Like I said it's a blind spot for the strategy and the print I provided shows it. Not the "enable strategy" but the word "historical" in the line, as in, printed via the historical implementation.

As for OnStateChange(), there IS a print from OnStateChange() in the print I provided. That shows it too.

[NinjaTrader_ChelseaB]

Hello digibob,

Historical data is processed before real-time data.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/95921-auto-trail-code-for-strat-that-is-backtestable?p=782442#post782442

Are you printing the State in OnBarUpdate() and seeing that this is printing Historical when a real-time bar is updating?

I've tested this and I am not able to reproduce. When a real-time bar is updating, I am seeing the State is Real-time. Let me know if you would like a video demonstrating this.

[digibob]

Hi Chelsea. Thank you for your reply. Let me first say I stumbled on the problem thru the logic in the code before ever printing anything. I'm using the print in here just in support of what I say, to show the problem. The print in itself is not the problem. I'm trying to show a call to OnBarUpdate() that should not be where it is.

Historical data is processed before real-time data - no problem there. Perfectly understood. How about future data processed as historical? Before the strategy is enabled for today's session, there is already a call closing today's session, in historical state.

Monday, historical processing, OnBarUpdate() is called for a daily series at 17:00 EST.
Tuesday, historical processing, OnBarUpdate() is called for a daily series at 17:00 EST.
Wednesday, strategy enabled at 10:45am, OnBarUpdate() is called for a daily series at 10:45am.

The only way this can be, is if that call at 10:45am is in realtime. But that is not what I'm seeing. I'm seeing a mid-session call, for a daily series, in historical state.

There are two options here:

Option #1 - this is happening only at my end. If so, I would be interested to see the printed output (possibly with a video as you suggest) of your running the script I provided. Please, not a different script that you use as a way to explain to me the mechanism. We are past that. Using any script other than the one I'm referring to would not be apples to apples for me and we'd be possibly missing the point again. I would like to see if at your end the said call is triggered right before the strategy is enabled as it does here. I'm referring to my post #20, the call that prints the line right above Transition. It starts with the words "Task (1st of 2)".
Of course if that call were to come after the strategy is enabled rather than before (as it should be) then the contents of this line would be different.

More importantly, if you do get a different output, then what am I supposed to do with it. You guys keep saying you don't see a problem and you just leave it at that. It's not helping. Why am I seeing something different than you do and how can I fix it.

Option #2 - this is happening not only at my end, it is actually a bug and can be described in the following way: the state underneath, in the underlying mechanism, has already changed to realtime by the time this call is triggered, just the strategy is not yet notified of the state change.

In chronological order--
1. The state in the underlying mechanism changes to realtime. The strategy is unaware.
Then-
2. OnBarUpdate() is called. The strategy thinks the state is still historical.
Then-
3. The strategy is notified of the state change. A fraction of a second too late.

I currently see no other way to explain it. Whether it's only at my end or not, it's a strange behavior. To repeat again, a call is triggered mid-session, for a daily series, in historical state.

Thank you for your time.

[NinjaTrader_ChrisL]

Hi digibob, thanks for your patience. I am understanding the question now, apologies for the confusion.

Right now I am getting more information on the first tick behavior. It looks like we have a related discussion here which I will be referencing:

https://ninjatrader.com/support/foru...essing?t=95419

This resulted in the feature request SFT-1892 being created which is described as:

The client would like to see the 'IsFirstTickOfBar' not return true until the first Full Real-time bar is hit in OBU() processing.

Right now I am seeing IsFirstTickOfBar become true right in the middle of today's bar, which one wouldn't expect with Tick Replay enabled. I'll report to my supervisor and update you shortly.

[NinjaTrader_ChrisL]

Hi digibob,

I ran a test with Tick Replay on and was able to get around the issue (I didn't have Tick Replay enabled initially). If you have Tick Replay enabled the OnBarUpdate method will be called exactly when today's daily bar opens. I proved it by drawing a dot when the condition is true:

OnEachTick

else if (State == State.Configure)
            {
                AddDataSeries(BarsPeriodType.Minute, 1440);
            }

protected override void OnBarUpdate()
        {

            if (CurrentBars[0] < 0 || CurrentBars[1] < 0)
                return;

            if (BarsInProgress == 1 && IsFirstTickOfBar)

            {
                Draw.Dot(this, "OPENBAR"+CurrentBar, false, 0, Close[0], Brushes.Blue);
            }
        }

Remember the time stamp for the daily bar will be at EOD.

[digibob]

Hi Chris. We are getting somewhere. Thank you. Close, not exactly there though. I've not used Tick Replay before. With this enabled, the undesired call is eliminated, which is good, but the goals are not yet achieved. Let me explain by breaking it down to historical and realtime.

About historical, enabling tick replay would require overhauling the entire historical implementation and structure that is already built and complete and does not require tick granularity. The challenge is just the point of transition, not the whole historical processing. Also, in backtest I run tests that span 20 years. If I were to rewrite the historical segment and enable tick replay I'd be old before the test finished.

About realtime, using tick replay still doesn't help capture the first realtime tick. IsFirstTickOfBar, as you said and showed, captures the first tick when today's daily bar opens. There is also a slight confusion going on in the thread you linked about the difference between the two .

So of these two the one I need to capture (tick replay aside) is the first tick immediately after the strategy is enabled. That 'premature' call I kept talking about, to my understanding that was the call that was supposed to capture the first realtime tick, and instead it was triggered before transition. I still contend that this call has no place in historical state. By all logic it should trigger in realtime after transition. If we stop for a moment to think what actually triggers this call, it IS the first realtime tick. That's the event. If the first realtime tick is not there, that call is not there either. And yet when the call is triggered the state is historical. Go figure.
To me this is undesired behavior, a bug, call it what you will. It's unrelated to tick replay.

So with your help, please, how can I achieve the following fairly simple goals? --

if (historical)
- perform task every day on session close (daily series)
- and make sure to not perform this task mid-bar just before transition
else if (realtime)
- capture the first realtime tick immediately after transition (same daily series)

Thank you.

[bltdavid]

- capture the first realtime tick immediately after transition (same daily series)

FWIW, in NT7, this was indeed a problem with COBC=False processing.

I was able to filter out the "zombie" bar which marked the LHB -> FRB transition,

LHB = Last Historical Bar
FRB = First Realtime Bar

protected override void OnBarUpdate()
{
    if (FirstTickOfBar)
    {
        if (Historical && !CalculateOnBarClose && CurrentBar == LastHistoricalBar)
        {
            // found transition: last historical bar -> first realtime bar
            Print("Found ZombieBar");
            IsZombieLHB = true;
            return;
         }
        ....
    }
    ....
}

and inside OnStartUp,

LastHistoricalBar = Count - 1;

I have not yet tried to filter the dreaded "zombie" bar in NT8. But perhaps this NT7 code
snippet will give you some ideas ...

[digibob]

Hi bltdavid, thanks for your input. Yes, I tried and am trying different workarounds, some based on conditionals similar to what you posted, and as I said in an earlier post not always successful due to the complexity of the code flow in my strategy. For example to find the "zombie" bar is one thing, problem is that when you find it, that's when IsFirstTickOfBar is true, and it's not where you need it to be true, because then you can't use it to capture the first realtime tick, you have to use more flag variables and mess up the code further. It's just one example.

I'm trying to show there is something fundamentally incorrect in the way transition is implemented by the system, creating what I called a "blind spot", which is the equivalent of your "zombie bar" () (obviously not a new issue if you mention NT7), and hoping to get either a "yes, it's confirmed as undesired behavior and will be fixed in the next update/release" (wishful thinking), or alternatively get an elegant solution from Support, if there is one. Because non-elegant workarounds I have many and they don't handle 100% of the cases.

(eg. they may break depending on the time of day when the strategy is enabled. Or they may rely on the session of one instrument type (futures) to always end after that of another instrument type (equities) and may break in the rare case when these two sessions end at the same time. And so on. But this is beyond the scope of this thread. I'm looking for a solution as opposed to a workaround. There must be a simple fit-all solution to achieve the goals I mentioned in the previous post. If Support can help find one - great, mission accomplished. If not, I suppose that would be a sign that the developers should be involved.)