The OnMarketData() method is guaranteed to always be called before OnBarUpdate()

[NinjaTrader_JessicaP]

We do not, tracking IDs are provided only to aid users in forums searches.

I have some feedback from a member of our QA team. It does seem that this is the expected behavior in this particular instance. I had come across the documentation in OnMarketData this morning, and I am following up to see if we can word this more clearly. The reason I believed this to be the expected behavior yesterday is due to this table in the Developing for Tick Replay page I mentioned, and this turns out to be the canonical behavior :

Calculate.OnEachTick

OnMarketData and OnBarUpdate call for each tick used to build the bar

• 1 OnBarUpdate event on CurrentBar 0
• Followed by 1 OnMarketData event
• 1 OnBarUpdate event on CurrentBar 0
• Followed by 1 OnMarketData event
• 1 OnBarUpdate event on CurrentBar 0
• Followed by 1 OnMarketData event
• 1 OnBarUpdate event on CurrentBar 0
• Followed by 1 OnMarketData event
• ...etc.

I will be following up with this request.

With Tick Replay enabled and Calculate.OnEachTick, a documented OnBarUpdate event precedes an OnMarketData event. This user would like OnMarketData's documentation changed to mention this exception to OMD's guarantee that OMD precede OBU, or would like Ninja's behavior changed so that OMD precedes OBU in the TR + C.OET event, with the Developing for Tick Replay page adjusted instead.

I will follow up with more information as soon as it's available. If the feature requests already exists, a vote will be added to it.


Please let us know if there are any other ways we can help.

[joanrocagas]

Come on!

The problem is not that the documentation is wrong, which it is. The problem is that the whole NT8 system is doing wrong calculations on indicators!!!!

How can it be the expected behavior, to have the indicators miscalculated all the time!!??

I've been trying to explain it in different ways. It's just conceptually wrong. Indicators are being updated BEFORE the MarketDataEventArgs object is available in the onMarketData !!!!

Could an experienced developer review this issue, please?? or point me what I am missing here!

[NinjaTrader_JessicaP]

The product management and development teams are reviewing this thread, and any information we add to it will be made available to them. In the feature request I sent them I mentioned the resolution where Ninja's behavior is changed so that OMD precedes OBU in this corner case, and the TR page's documentation is then updated to reflect the new behavior. I am sending another reply directly to let them know that this is your preferred solution. Please let us know if there are any other ways we can help.

[joanrocagas]

That is ok Jessica.

I just trying to say that I don't see it as a corner case. Tick Replay doesn't have anything to do with this issue. It happens with or without TR. It's not a feature request or my "preferred way" just because it suits me. Keeping the call order like it is now it just causes NT8 to miscalculate Indicators, and probably all Strategies too. It happens too with Calculate = onPriceChange.

Market data should be made available first, then update the Indicators, then update the Strategies. I don't understand how the other way around could be "expected behaviour". It's not that I want to change the canonical behaviour of NT8. I'm trying to point out a fundamental design error in the NT8 architecture.

Hope to get it sorted soon, NT looks very promising to me.

[tradesmart]

My workaround goes something like this:

protected override void OnMarketData(MarketDataEventArgs e)
{
   if (Calculate == Calculate.OnBarClose) 
      CalcSomething(CurrentBar, e.Time) ;
}

protected override void OnBarUpdate()
{
   if (Calculate != Calculate.OnBarClose) 
      CalcSomething(CurrentBar, Time[0]) ; 
}

[NinjaTrader_Matthew]

Thanks for providing your comments.

There was a typo in the OnMarketData help guide page which is now correct. Sorry for any confusion that may have caused

The designed behavior is for OnBarUpdate to come before OnMarketData, allowing for the internal indexes to be updated correctly for the upcoming OnMarketData calls. This also ensures that you receive an OnMarketData update in sync with the tick that closed the bar.

Please consider the following output, assuming your indicator starts processing in real-time on primary current bar #50.

[B]5-Tick series running OnBarClose[/B]
OBU CurrentBar 50
    OMD #256=CurrentBar 50
    OMD #257=CurrentBar 50
    OMD #258=CurrentBar 50
    OMD #259=CurrentBar 50
    OMD #260=CurrentBar 50     <- tick that closed bar #50
OBU CurrentBar 51            <- current bar is updated
    OMD #261=CurrentBar 51    <- values for omd are in sync for next set of bars
    OMD #262=CurrentBar 51
    OMD #263=CurrentBar 51
    OMD #264=CurrentBar 51
    OMD #265=CurrentBar 51

On the other hand, if we updated OnMarketData first (which we experimented with in early design), there are issues with OBU and OMD indexers being out of sync

[B]5-Tick series running OnBarClose[/B]
    OMD #256=CurrentBar 49   <- values for omd are now behind
    OMD #257=CurrentBar 49
    OMD #258=CurrentBar 49
    OMD #259=CurrentBar 49
    OMD #260=CurrentBar 49     <- tick that closed bar #50
OBU CurrentBar 50                  <- current bar is updated too late
    OMD #261=CurrentBar 50    <- values continue to be out of sync
    OMD #262=CurrentBar 50
    OMD #263=CurrentBar 50
    OMD #264=CurrentBar 50
    OMD #265=CurrentBar 50    
OBU CurrentBar 51

I can sympathize that this is a confusing subject and understand that it may be more intuitive to think of "Market Data Events" driving the indicator "OnBarUpdate" event. But in reality, it is better to think of the Bars being the primary input to the indicator, and the OnMarketData event to be a sub-set of the bar event. (There is also a core market data engine which distributes the ticks to the bars, which is not to be confused with the OnMarketData() event method exposed to NinjaScript)

We appreciate you sharing your thoughts on the matter as this is something we've been back and forth with internally, but have ultimately decided the current implementation is the best of a 'pick your poison' scenario.

To accommodate this design decision, you can track the current bar as we have done in the BuySellPressure and BuySellVolume indicators.

We'll also be reviewing and updating our documentation to see how this information can be more clearly reflected in the near future.

[joanrocagas]

[QUOTE=NinjaTrader_Matthew;490085]
Please consider the following output, assuming your indicator starts processing in real-time on primary current bar #50.

[B]5-Tick series running OnBarClose[/B]
OBU CurrentBar 50
    OMD #256=CurrentBar 50
    OMD #257=CurrentBar 50
    OMD #258=CurrentBar 50
    OMD #259=CurrentBar 50
    OMD #260=CurrentBar 50     <- tick that closed bar #50
OBU CurrentBar 51            <- current bar is updated
    OMD #261=CurrentBar 51    <- values for omd are in sync for next set of bars
    OMD #262=CurrentBar 51
    OMD #263=CurrentBar 51
    OMD #264=CurrentBar 51
    OMD #265=CurrentBar 51

The problem is still there.

Let's say that on OMD #260 my indicator would reach a threshold that would activate a strategy action to enter a long position. But because the indicator will not be updated until OBU CurrentBar 51, the strategy action will not be triggered until OBU CurrentBar 51, so that is OMD 261, which is one tick late.

If you look at the source code of BuySellVolume indicators, you'll see that is what would happen since OMD events don't really update the indicator values.

This forces to move all the update logic to the OMD method, which makes the OBU method rather useless except for the fact that is the method that updates the Bar index.

The problem is that this design makes us detect the end of the bar on the first tick of the next bar, which is one tick late.

[joanrocagas]

The designed behavior is for OnBarUpdate to come before OnMarketData, allowing for the internal indexes to be updated correctly for the upcoming OnMarketData calls.

If you look at the output files posted on reply #10 you'll see that

Calculate = onBarClose -->> OMD executes before OBU
Calculate = onEachTick --->> OBU executes before OMD

so, NT8 is not behaving as you say it is...

[NinjaTrader_Brett]

Hopefully this will clarify using your example:

5-Tick series running OnBarClose
OBU CurrentBar 50
OMD #256=CurrentBar 50
OMD #257=CurrentBar 50
OMD #258=CurrentBar 50
OMD #259=CurrentBar 50
OMD #260=CurrentBar 50

{
OBU CurrentBar 51
OMD #261=CurrentBar 51 <-- NinjaTrader doesn't close the last bar until we get the next tick of the next bar. Therefore this is the actual tick that closed the previous bar and triggered OBU. Hence OBU is called before OMD and everything is kept in sync.
}

OMD #262=CurrentBar 51
OMD #263=CurrentBar 51
OMD #264=CurrentBar 51
OMD #265=CurrentBar 51

[joanrocagas]

{
OBU CurrentBar 51
OMD #261=CurrentBar 51 <-- NinjaTrader doesn't close the last bar until we get the next tick of the next bar. Therefore this is the actual tick that closed the previous bar and triggered OBU. Hence OBU is called before OMD and everything is kept in sync.
}

That is exactly where it lies the problem. If OMD 260 carried some data that made me enter a position that would not be actually noticed by the indicator until OMD 261.

It's not that hard to see. Look at the source code of your tick indicators, they are all wrong calculated in that case.


You guys all the time ignore the problem I'm pointing out post after post. None of you have actually dig deep into thread and actually understood the problem.

You just keep saying that "everything is sync", that is "expected behaviour", and things like that. Those don't address at all the problem I'm pointing.

If you actually look at the source code and test the indicators you would see what I trying to explain you all the time. It seems nobody here is really concerned with the actual underlying mechanics.

I really want to use NT, and I really want to like it. But you are making it difficult.

[NinjaTrader_Brett]

Would be happy to check into if something is wrong, which indicator do you reference specifically and I'll take a look into and get back.

M concern is we're not on the same page around NinjaTrader using the event of the tick of the next bar to close the previous bar. All our existing indicators should be working with this concept.

My definition of 'working' being:

* As the next tick of the next bar comes in and closes the previous bar.
* Then an 'on bar close' calculated indicator would have its OnBarUpdate() method triggered.
* All OMD events that factored into building the bar have been called prior to the closing OnBarUpdate() method was triggered.
* The OMD event that triggered closing the bar is called in the context of the next CurrentBar. (After OBU was called closing the previous bar)

Perhaps a phone call would be necessary to rip through, let me know if you would help or if you simply had an indicator for me to analyze which I could return with my analysis.

-Brett

[joanrocagas]

I don't have NT in this computer now, but I'll try to recall the source code.

Look at the BuySell Volume indicator. This is a Calculate onEachTick indicator.

In the OMD method is where it updates two private variables: buys and sells.
The OBU method is where the indicator is actually updated.

Let's say I have an strategy that executes an action when Buys == Sells + 1000

Let's simulate three market ticks with volume
1st tick 500 buys
2nd tick 500 buys
3rd tick 1000 sells.

OBU Bar 1 (Buys[0] = 0, Sells[0] = 0, buys = 0, sells = 0) OMD 1 (buys += 500, sells += 0)
OBU Bar 1 (Buys[0] = 500, Sells[0] = 0, buys = 500, sells = 0) - OMD 2 (buys += 500, sells += 0 )   <<<< HERE My strategy should execute because actual market buys == actual market sells + 1000, but Indicator Buys is still not updated!!! It will be on the next OMD!!

OBU Bar 1 (Buys[0] = 1000, Sells[0] = 0) - OMD 3 (buys += 0, sells +=1000)  <<<<---- NT will execute strategy action here, which is wrong!!! market buys != market sells +1000 (actual market buys = 1000, actual market sells = 1000)

This problem happens in all Calculate onEachTick indicators. And if OBU is always executed before OMD it will happen always in all indicators.

[joanrocagas]

Did my last example clarified the issue, Brett?

[NinjaTrader_Brett]

**** Edited to correct slide, check slide in post 34

Yes it did, thanks for taking the time to prepare that. I took some time to start working on some additional documentation images which I attached a preview of here to help explain the sequence of events. Hopefully it helps answer some questions then it does create more questions but if it confuses you don't worry i'll be added some context and a help guide article in the coming weeks.

To answer your question: You are correct in your example of how the sequence of events would unfold. This is actually expected from our side and there are various reasons why this is how it is. Regardless its not your expectation when working on OnEachTick and is admittedly confusing which is why I started up on additional documentation.

The key takeaways of the additional documentation which should help you head in the right direction will be:

- In principle ALL NinjaScript OnBarUpdate logic triggered 'on bar close' will not trigger until the 'next' tick of the next bar is actually received (Since we don't know to close the current bar until we get the first tick of the next bar)
- NinjaScript objects which have OnBarUpdate logic triggered 'on each tick' is subject to the same rules above since you want the same actual ticks reflected in the same bar.
- If you have code which you do not want to wait for the next tick to trigger, then you must alter your code accordingly. Specifically in Volume analysis you would trigger all actions directly in OnMarketData and basically not use OnBarUpdate. In your sample I would move the order methods down into OnMarketData in that case.

I will look into altering the BuySellVolume default implementation so that it follows this principle which would resolve a current problem it has which is that its always 1 tick behind. Which would simply be moving all the OnBarUpdate logic down into OnMarketData. Leaving OnBarUpdate essentially empty and not used, which I think aligns with the point your trying to make which is there is little point to utilizing OnBarUpdate if you plan to utilize OnMarketData on Calculate On Each Tick.

[roman_ch]

Brett,

First of all, thank you for taking time to respond. It is truly much appreciated. Second, there are probably several interesting sub-topics that could be developed, but I'd like to specifically focus only on one in this post.

Let me start by using CurrentBar as example:

• according to the helpguide CurrentBar is officially available in OBU. The guide, however, does not state that CurrentBar is NOT available in OMD. From experience, we know that CurrentBar is "available" in OMD;
• testing shows that the value of CurrentBar is only updated when in OBU. With Calculate on Each Tick and following the slide from your post, will tick No.6 in OMD still have CurrentBar set at 0? Seems that, yes, it will, although it truly belongs to bar 1. If so, there is a catch: CurrentBar will be "working" in OMD, except for the first tick of the next bar and this can cause issues that are not obvious to debug; and
• we seem to accept that certain user logic must be properly coded from OMD rather than OBU. For purposes of that kind of logic, knowing when CurrentBar is (i) available and/or (ii) updated could be quite important.

I must be clear in that I am not interested in a discussion of CurrentBar logic as such, although it is important. Instead, what I have in mind is in the context of the documentation you mentioned you might be working on.

It would be of great help to have a clear indication in the helpguide of when / where user-exposed NT variables are updated / are current. Ideally, this could be done specifically on each relevant "page" in the guide, or, in the alternative, a general article could be the solution at first instance. Please could NT team consider improving the helpguide in this regard.

Regards,
Roman