Market replay OnBarUpdate for multi - series order

[habibalex]

I have an indicator where I add a 1 - tick data series as a secondary series.

When I am in market replay mode, it appears like the primary series is processed first and then the secondary series is processed.

In regular mode, it looks like the secondary series is processed while the primary series is processed.

How do I emulate this parallel processing during in market replay mode?

[sledge]

This is a problem in NT7??? hmmmm

I haven't touched NT7 replay in over a year. Post some sample code that shows the problem. Thanks.

[NinjaTrader_JessicaP]

Hello habibalex, and thank you for your question.

NinjaTrader is event driven and asynchronous. There are no guarantees in terms of event ordering. It is likely that you are using a different data feed provider from that which powers Market Replay data. If you need to ensure ordering, I would advise you to use a C# Queue class or another similar data storage class, documented at this publicly available MSDN link,

https://msdn.microsoft.com/en-us/library/system.collections.queue.aspx

coupled with a user class which contains all the data you would like available in-order.

Here is just one example showing how you can gain more control over the order events are replayed in.

[FONT=Courier New][FONT=Courier New]///<summary>
/// Stores all the information you would like to process from the secondary series
[/FONT][/FONT][FONT=Courier New][FONT=Courier New][FONT=Courier New]///</summary>
[/FONT][/FONT]private class Node
{
  public double highPrice;
  public DateTime when;
}
private System.Collections.Queue mostRecentEvents;

protected override void OnBarUpdate()
{
  switch (BarsInProgress)
  {

    // secondary series, just store stuff to replay during the primary series here
    case 1 :
      if (mostRecentEvents == null)
      {
        mostRecentEvents = new System.Collections.Queue();
      }
      Node newNode = new Node();
      newNode.highPrice = Highs[BarsInProgress][0];
      newNode.when = Times[BarsInProgress][0];
      mostRecentEvents.Enqueue(newNode);
      break;

    // primary series, process secondary series events when we want to
    case 0 :

      // no secondary series events have played here

      if (mostRecentEvents != null)
      {
        // if you would like these replayed most-to-least recent use Dequeue instead

        // replay least-to-most recent
        foreach(Node curNode in mostRecentEvents)
        {
          // this will replay
          Print("this node's when: " + curNode.when);
          Print("this node's price: " + curNode.highPrice);
        }
      }

      // now all secondary series events have played

      break;
  }
}[/FONT]

Please let us know if there are any other ways we can help.

[habibalex]

yes NT7. I have something like this in the code. When not connected to market replay, this runs fine when loading the chart. In replay cBarNumber = lastBarNumber because it processes BarsInProgress == 0 first then BarsInProgress == 1

if(BarsInProgress == 1){
    if(cBarNumber != lastBarNumber){
        lastBarNumber = cBarNumber;
    }
}else{
 cBarNumber  = CurrentBar;
}

[NinjaTrader_JessicaP]

Hello again habiblax, and thank you for providing that code. Upon reviewing it, it looks like my reply, which must have posted simultaneously with yours, still applies to your situation. Please review it and let us know if there are any other questions we may answer.

[habibalex]

Hi Jessica,

I'm not connected to any feed when I just start up ninjatrader. My indicator works the same way when Ninja starts up and when I'm connected to my CQG datafeed. It seems like the market replay does something different.

[habibalex]

Hi Jessica,

After reviewing your code I still don't see how that will fix my issue, the queue will not record the currentbar of the primary series when iterating through the secondary series.

I can work around this issue by using something like this I think.

cBarNumber = BarsArray[0].GetBar(Time[0]);

However, it seems like Market Replay has a bug since the both CQG, and NT7 chart loading both work the same way, and only Market Replay breaks the indicator.

[NinjaTrader_JessicaP]

Hello habiblax,

Just to clarify, when you say this breaks your indicator, are you saying that cBarNumber is not initialized because BarsInProgress == 1 occurs before BarsInProgress == 0 ?

If this is the case, this looks like NinjaTrader 7 is working as designed. As OnBarUpdate is asynchronous, there are no guaranteed BarsInProgress orderings. You will need to use a strategy such as the one I suggested if you need things to occur in a specified order.

The strategy I suggested was not a literal example, rather, it was a demonstration of this concept :

• When BarsInProgress is not set to your primary series' bars, you can store any values you will want available to your script
• When BarsInProgress is set to your primary series' bars, you can retrieve the stored information, either before or after your primary series' bars are processed, as you see fit, in either most-to-least recent order or least-to-most recent order

You can modify the strategy I posted to store any value, including the cBarNumber value, from any event at any point in time.


Please review the help guide page on multiple data series available here for more information.

http://ninjatrader.com/support/helpGuides/nt7/?multi_time_frame__instruments.htm

Please let us know if there are any other ways we can help.

[habibalex]

Let me explain what I am doing. I am adding a single tick secondary series to a chart. Every time a new bar is formed on the primary series I start recalculating things. It's not a problem if it doesn't exactly match up every time, however I do need it to approximately start the recalculation around when the new bar starts.

When loading a chart during Market Replay, all the Bars from the primary series are processed first, then all the Bars in the secondary series are processed. This order is completely different than how ninjatrader loads charts when not connected to anything or when connected to a broker.

So to summarize: I believe my problem is that when ninjatrader loads charts in Market Replay, it loads them differently than when it's not connected to a broker or when connected to a live data feed.

[NinjaTrader_JessicaP]

I believe you are correct about Market Replay data loading OnBarUpdate events with a different BarsInProgress ordering than occurs with live data.

What I was trying to draw your attention to, is the fact that Ninja was designed in such a way that this ordering, itself, should be treated as random. As a programmer, you should not be making assumptions about the ordering of these events. If you would like to make such assumptions, you will need to add support so that these assumptions become safe, as a developer.

If you need events to occur chronologically, you can use the Time dataseries and a C# Dictionary object together. Should you not have milisecond data - which is typically unique - to hash with, I would recommend simulating this manually, by

• detecting if an existing timestamp key exists
• adding 1 milisecond to get a new timestamp key if this is the case

I am including publicly available documentation to the MSDN Dictionary class.

https://msdn.microsoft.com/en-us/library/xfhwa508.aspx

Time dataseries,

http://ninjatrader.com/support/helpGuides/nt7/time.htm

[habibalex]

I'm able to get around the issue by testing for when I'm in market replay mode, so that's fine. But I still think something is wrong.

Chart loading when Unconnected works in one way, Connected to a broker works similarly, Live data from a broker works similarly, Connected to market replay and playing back the data works similarly. Yet while Connected to Market Replay and loading chart data it doesn't work similarly.

[NinjaTrader_JessicaP]

Thank you for this additional information. I understand that your strategy has a different event ordering when loading historical data while connected to Market Replay. Please review this excerpt from the help guide.

1. With multi-time frame and instrument strategies, OnMarketData() will be called for all unique instruments in your strategy. Use the BarsInProgress to filter the OnMarketData() method for a specific instrument. (BarsInProgress will return the first BarsInProgress series that matches the instrument for the event)
2. Do not leave an unused OnMarketData() method declared in your NinjaScript object. This will unnecessarily attach a data stream to your strategy which uses unnecessary CPU cycles.
3. Should you wish to run comparisons against prior values you will need to store and update local variables to track the relevant values.
4. With NinjaTrader being multi-threaded, you should not rely on any particular sequence of events like OnMarketData() always being called before OnBarUpdate() or vice versa.

This excerpt I believe also applies to our situation,

1. A multi-series script only processes bar update events from the primary Bars (the series the script is applied to) and any additional Bars objects the script adds itself. Additional Bars objects from a multi-series chart or from other multi-series scripts that may be running concurrently will not be processed by this multi-series script.
2. If a multi-series script adds an additional Bars object that already exists on the chart, the script will use the preexisting series instead of creating a new one to conserve memory. This includes that series' session template as applied from the chart. If the Bars object does not exist on the chart, the session template of the added Bars object will be the session template of the primary Bars object. If the primary Bars object is using the "<Use instrument settings>" session template then the additional Bars objects will use the default session templates as defined for their particular instruments in the Instrument Manager.
3. In a multi-series script, real-time bar update events for a particular Bars object are only received when that Bars object has satisfied the BarsRequired requirement. To ensure you have satisfied the BarsRequired requirement on all your Bars objects it is recommend you start your OnBarUpdate() method with CurrentBars checks.
4. A multi-series indicator will hold the same number of data points for plots as the primary series. Setting values to plots should be done in the primary series in OnBarUpdate(). If you are using calculations based off of a larger secondary series, it may plot like a step ladder because there are more data points available than there are actual meaningful data values.

Should neither of these apply to your situation, please send a stripped down copy of your strategy to platformsupport[at]ninjatrader[dot]com along with your most recent (My) Documents\NinjaTrader 7\logs and (My) Documents\NinjaTrader 7\traces files so that we can look into this more carefully. Please also export the historical data you are playing over through the historical data manager. Instructions on doing so are here,

http://ninjatrader.com/support/helpGuides/nt7/exporting.htm

Thank you for your report and for using NinjaTrader.