• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Understanding Instrument.MarketData.Update

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Understanding Instrument.MarketData.Update

    First I'd like to express gratitude that the NT8 developers have provided us with both synchronized and unsynchronized market data events.
    Being able to access the unsynchronized data is important to us developers who want to do our job in advanced scenarios without creating deadlocks.
    So thank you.

    In order to understand the unsynchronized Instrument.MarketData.Update event a little better, I wrote the code that is attached and saw the output that is shown in the image below.
    To understand it better, I'd like to make some observations and ask some questions:

    Observation 1:
    There are two threads used to raise the events.
    Each thread seems to raise its own events in the correct chronological sequence.
    Combined, the two threads raise duplicate copies of each event, and the combined chronological sequence will vary depending on each thread's progress.

    Question 1:
    Why is a single event subscription receiving events from both threads and thereby receiving duplicate copies of each event?

    Observation 2:
    There seems to be a pattern with the threads that can be more easily explained if we name the threads:
    Let the thread with the first event, in this case having ThreadId=49, be called the "primary thread".
    Let the thread with the second event, i this case having ThreadId=57, be called the "secondary thread".
    When the indicator is replaced, it will again subscribe and receive events from two threads.
    The primary thread (id=49) will still be there. The secondary thread will have a different id.

    It seems that the "primary thread" is like a "constant" thread that is always there and always pumping events.
    It seems that the "secondary thread" is more like a temporary thread that was created just for this one particular subscription and a different thread will be used for the next subscription.

    Question 2:
    Can I rely on these observations about the threads to be consistent?

    Question 3:
    It seems like the best way to use the event data is to choose a thread id and only accept events from that thread id.
    Do you agree?

    NB: Ignore the '0' price in the first entry in the image below. It happened because of a minor code issue at the time I took the screenshot and does not happen with the code now posted below.

    Click image for larger version  Name:	output.jpg Views:	0 Size:	182.1 KB ID:	1088746


    Here's the code I used to generate the view above:

    Code:
    using NinjaTrader.Core;
    using NinjaTrader.Data;
    using NinjaTrader.Gui;
    using System;
    using System.Globalization;
    using System.Runtime.CompilerServices;
    using System.Runtime.ExceptionServices;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using System.Windows.Media;
    
    namespace NinjaTrader.NinjaScript.Indicators {
    
        public class MarketDataSynchronizationTests : Indicator {
    
            #region Static stuff related to the unique indicator id
    
            private static int __id = 0; // The source of the unique indicator ids.
    
            private static int GetUniqueId() {
                return Interlocked.Increment(ref __id);
            }
    
            #endregion
    
            private readonly object Mutex = new object();
            private readonly int Id = GetUniqueId(); // A unique id assigned only to this indicator instance.
    
            private bool _subscribed = false; // True when this indicator has subscribed to market data updates.
    
            private TickSequencePosition sequencePosition = new TickSequencePosition();
    
            protected override void OnStateChange() {
                switch (State) {
    
                    case State.SetDefaults:
                        IsOverlay = false;
                        Name = "_Market Data Synchronization Tests";
                        break;
    
                    case State.DataLoaded: {
                        if (ChartControl is object) {
                            Instrument.MarketData.Update += MarketData_Update;
                            _subscribed = true;
                        }
                        break;
                    }
    
                    case State.Terminated:
                        if (_subscribed) {
                            try { Instrument.MarketData.Update -= MarketData_Update; } catch { }
                        }
                        break;
                }
            }
    
            private void MarketData_Update(object sender, MarketDataEventArgs e) {
                lock (Mutex) {
                    if (e.Instrument != Instrument) return;
                    if (e.MarketDataType != MarketDataType.Last) return;
                    var sb = new StringBuilder();
    
                    /// Figure out if the tick time is out of sequence, so we can flag it in the output window with a "*" character.
                    var isBehind = false;
                    try { sequencePosition.OnTick(e.Time); } catch { isBehind = true; }
                    sb.Append(isBehind ? "* " : "  ");
    
                    sb.Append(e.Time.ToString("HH:mm:ss.fffffff"));
                    sb.Append(' ');
    
                    sb.Append(e.Volume.ToString().PadLeft(3));
                    sb.Append('@');
                    sb.Append(Instrument.MasterInstrument.FormatPrice(e.Price));
    
                    sb.Append("  Id: ");
                    sb.Append(Id.ToString().PadLeft(3));
    
                    sb.Append("  ThreadId: ");
                    sb.Append(Thread.CurrentThread.ManagedThreadId.ToString().PadLeft(3));
    
                    Print(sb.ToString());
                }
            }
    
    
            /// <summary>
            /// This object keeps track of a tick's position within a sequence of ticks, and can be used to compare that tick's position
            /// with another tick's position.
            /// A tick's position in the sequence of ticks is defined firstly by its time. But because multiple ticks can occur at the same
            /// time, we also need to keep track of the tick's position within the group of ticks that occured at the same time.
            /// </summary>
            public struct TickSequencePosition : IComparable<TickSequencePosition>, IEquatable<TickSequencePosition> {
    
                /// <summary>
                /// The tick's time.
                /// </summary>
                public DateTime Time { get; private set; }
    
                /// <summary>
                /// The tick's position within the group of ticks that occurred at the same time.
                /// </summary>
                public int CountAtTime { get; private set; }
    
    
                /// <summary>
                /// Use this to advance the position counter.
                /// </summary>
                /// <exception cref="Exception">Thrown when time has gone backward. When thrown, it shows that there is an error in your code.</exception>
                public void OnTick(DateTime time) {
                    if (time > Time) {
                        Time = time;
                        CountAtTime = 0;
                    } else if (time == Time) {
                        CountAtTime++;
                    } else {
                        throw new Exception("Tick time went backwards");
                    }
                }
    
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public override int GetHashCode() { return  (Time.GetHashCode() * (-1521134295)) + CountAtTime.GetHashCode(); }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public override bool Equals(object obj) {
                    if (obj is TickSequencePosition) return Equals((TickSequencePosition)obj);
                    return false;
                }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public bool Equals(TickSequencePosition other) { return this == other; }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public int CompareTo(TickSequencePosition other) {
                    var timeCompare = Time.CompareTo(other.Time);
                    return timeCompare == 0 ? CountAtTime.CompareTo(other.CountAtTime) : timeCompare;
                }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator ==(TickSequencePosition left, TickSequencePosition right) { return left.Time == right.Time && left.CountAtTime == right.CountAtTime; }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator !=(TickSequencePosition left, TickSequencePosition right) { return !(left == right); }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator >(TickSequencePosition left, TickSequencePosition right) { return left.CompareTo(right) == 1; }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator <(TickSequencePosition left, TickSequencePosition right) { return left.CompareTo(right) == -1; }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator >=(TickSequencePosition left, TickSequencePosition right) { return left.CompareTo(right) >= 0; }
    
                [MethodImpl(MethodImplOptions.AggressiveInlining)]
                public static bool operator <=(TickSequencePosition left, TickSequencePosition right) { return left.CompareTo(right) <= 0; }
            }
        }
    }
    Last edited by bboyle1234; 02-27-2020, 09:37 PM. Reason: Fixed an error

    #2
    Update:
    I have noticed that if the indicator OnBarUpdate method is blocked with Thread.Sleep, one of the threads discussed above will stop raising events until the indicator has stopped blocking its OnBarUpdate thread. Just another observation along the way to understanding how this works.

    Comment


      #3
      Hello bboyle1234,

      I wanted to start off by saying that this is an unexpected way for an Indicator to observe market data, I likely will not be able to answer these questions accurately based on the syntax you used and the context it is used from.

      The indicator has an override for market data already called OnMarketData, had you tried using that override and observed the same results? The syntax you have shown would be used from an Addon context so I would not necessarily have any specific expectations or details for using this from an indicator.


      Observation 1:
      There are two threads used to raise the events.
      Each thread seems to raise its own events in the correct chronological sequence.
      Combined, the two threads raise duplicate copies of each event, and the combined chronological sequence will vary depending on each thread's progress.

      Question 1:
      Why is a single event subscription receiving events from both threads and thereby receiving duplicate copies of each event?

      Observation 2:
      There seems to be a pattern with the threads that can be more easily explained if we name the threads:
      Let the thread with the first event, in this case having ThreadId=49, be called the "primary thread".
      Let the thread with the second event, i this case having ThreadId=57, be called the "secondary thread".
      When the indicator is replaced, it will again subscribe and receive events from two threads.
      The primary thread (id=49) will still be there. The secondary thread will have a different id.

      It seems that the "primary thread" is like a "constant" thread that is always there and always pumping events.
      It seems that the "secondary thread" is more like a temporary thread that was created just for this one particular subscription and a different thread will be used for the next subscription.

      Question 2:
      Can I rely on these observations about the threads to be consistent?

      Question 3:
      It seems like the best way to use the event data is to choose a thread id and only accept events from that thread id.
      Do you agree?

      NB: Ignore the '0' price in the first entry in the image below. It happened because of a minor code issue at the time I took the screenshot and does not happen with the code now posted below.

      I believe in this situation we would need to see if your observations apply toward the indicators OnMarketData override and if so continue the question from that point using the override instead. The Addon event subscription is intended to be used in combination with other addon items to subscribe to data from an addon, I am not certain if that will affect this event being subscribed from an indicator which already has a market data subscription.

      We additionally won't have any specific info on the threading aspect of your questions, if you are encountering some kind of problem because of threading we could try to explore that with additional specific details for that specific problem but we would not be able to look into what is calling the thread/subscription or why it is called multiple times here as that would be happening internally.

      Is there a greater concern past finding out about the internal threads and how they are used such as a problem you encountered or task you are trying to do?



      I have noticed that if the indicator OnBarUpdate method is blocked with Thread.Sleep,
      Right, you are blocking the execution by issuing a Thread.Sleep. You would need to avoid doing any type of blocking such as Thread.Sleep or while loops while working in NinjaScript.



      Please let me know if I may be of additional assistance.
      JesseNinjaTrader Customer Service

      Comment


        #4
        Dear Jesse,
        Thank you for taking a thorough look at my questions.

        You're quite right that this code would be "unusual in an indicator". In my real world the code lives inside an addon, in a much bigger context, of course.
        I've put the demo code here in an indicator because it's simple and easy to understand. And unlike an addon, you can easily restart the test by pressing F5 on the chart.

        Yes, I've tried all the ways I could find of obtaining real-time market data from NT8 inside an add-on. Here are the ways I found. Of course they don't include the Indicator.OnMarketDataUpdate override because that doesn't exist in an addon.

        BarsRequest.Update += xxx (synchronous)
        Instrument.MarketDataUpdate+= xxx (synchronous)
        new MarketData(Instrument).Update+= xxx (synchronous)

        and finally:

        Instrument.MarketData.Update+= xxx (Asynchronous) (hooray!!)

        The add-on really needed a way to obtain data asynchronously, and by that, I mean, outside the single-threaded context of the indicator's OnBarUpdate method.



        My questions are about looking "under the covers" at how Instrument.MarketData.Update works, because I want to understand it and prevent myself writing bugs in my addon through simply not understanding how and why it works.

        My add-on needs to request historical data, which it does via the BarsRequest facility. Then it must correctly append real-time data, but it must do so OUTSIDE the single-threaded context of the OnBarUpdate method, which means it unfortunately cannot use the BarsRequest.Update event. The only way I have found to get this data is via the asynchronous Instrument.MarketData.Update event, which seems to raise duplicate events, depending on how many threads are involved in raising events there.

        I'm hoping to learn more about the origin, lifespan, and purpose of those threads, or whatever it is that drives the event so that I can write code that filters duplicate ticks, and appends new ticks in correct sequential order. I think I've got some pretty good code running and it seems to be yielding pretty much perfect results. But I want to learn more. Hence the questions:


        Observation 1:
        There are two threads used to raise the events.
        Each thread seems to raise its own events in the correct chronological sequence.
        Combined, the two threads raise duplicate copies of each event, and the combined chronological sequence will vary depending on each thread's progress.

        Question 1:
        Why is a single event subscription receiving events from both threads and thereby receiving duplicate copies of each event?
        This question is made with the hope of understanding more about what's going on under the hood so I can anticipate a mistake I could be making.

        Observation 2:
        There seems to be a pattern with the threads that can be more easily explained if we name the threads:
        Let the thread with the first event, in this case having ThreadId=49, be called the "primary thread".
        Let the thread with the second event, i this case having ThreadId=57, be called the "secondary thread".
        When the indicator is replaced, it will again subscribe and receive events from two threads.
        The primary thread (id=49) will still be there. The secondary thread will have a different id.

        It seems that the "primary thread" is like a "constant" thread that is always there and always pumping events.
        It seems that the "secondary thread" is more like a temporary thread that was created just for this one particular subscription and a different thread will be used for the next subscription.

        Question 2:
        Can I rely on these observations about the threads to be consistent?
        This question was made with the thought in mind that perhaps I should try to identify the "primary thread", if such a thing exists, and accept events only from that thread, ignoring others, and thereby receiving a "perfect stream".

        Question 3:
        It seems like the best way to use the event data is to choose a thread id and only accept events from that thread id.
        Do you agree?
        This question was made with exactly the same motivation as the previous question

        But here are newer, more important questions:
        Can I rely on all events from a single thread id being in correct sequential order?
        Can I rely that a single thread id will supply all events?


        Thanks again Jesse!

        Comment


          #5
          Hello bboyle1234 ,

          Thank you for the reply.

          In this situation there is not really anything additional that I could provide as this type of information is not documented or available. Understanding how the internal events/threads work is not something which our support could accurately detail or confirm/deny expectations because we have no guidelines/documentation there. I can put in a feature request to try and have additional documentation added for the events being used but for the time being that information is not present for me to provide it.

          In general as an end user you should not need to know anything about the underlying threads or their ID's. If you are seeing something unexpected happening with a NinjaScript event/method/object and you are getting that far into debugging you likely should instead extract a small sample of the overall problem (not your thread diagnosis but instead the reason you created that) so that we can get a idea of how that is affecting your code and then forward to development for further review.


          Please let me know if I may be of additional assistance.
          JesseNinjaTrader Customer Service

          Comment

          Latest Posts

          Collapse

          Topics Statistics Last Post
          Started by MercuryRed, Today, 03:24 PM
          0 responses
          5 views
          0 likes
          Last Post MercuryRed  
          Started by itrader46, Today, 03:24 PM
          0 responses
          2 views
          0 likes
          Last Post itrader46  
          Started by beobast, Today, 01:56 PM
          1 response
          3 views
          0 likes
          Last Post NinjaTrader_Jesse  
          Started by daigo1, Today, 02:21 PM
          2 responses
          6 views
          0 likes
          Last Post daigo1
          by daigo1
           
          Started by Zapzap, Today, 02:24 PM
          1 response
          10 views
          0 likes
          Last Post NinjaTrader_ChelseaB  
          Working...
          X