Announcement

Collapse

Looking for a User App or Add-On built by the NinjaTrader community?

Visit NinjaTrader EcoSystem and our free User App Share!

Have a question for the NinjaScript developer community? Open a new thread in our NinjaScript File Sharing Discussion Forum!
See more
See less

Partner 728x90

Collapse

^TICK index tick data processed asynchronous

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    ^TICK index tick data processed asynchronous

    Hopefully you can explain why the following occurs.
    I have a simple indicator that plots the ^Tick high, low, and close values.
    BarsInProgress 0 is a 30 second chart.
    BarsInProgress 1 is a 1 tick ^Tick data series.
    Calculate = On bar close.
    Every once in a while I notice the last tick from BarsInProgress 1 is processed after BiP 0 has processed.
    Here is what I'm seeing in the NS Output windows...
    BiP-1 26660 : 08:24:22 | Closes[0]: 136 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26661 : 08:24:23 | Closes[0]: 108 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26662 : 08:24:24 | Closes[0]: 110 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26663 : 08:24:25 | Closes[0]: 80 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26664 : 08:24:26 | Closes[0]: 80 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26665 : 08:24:27 | Closes[0]: 72 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    BiP-1 26666 : 08:24:28 | Closes[0]: 32 tickHigh: 148 tickLow: -41 (OBC) .Realtime
    # BiP--0 888 : 08:24:30 | Closes[1]: 32 (OBC) .Realtime
    BiP-1 26667 : 08:24:29 | Closes[0]: 30 tickHigh: 30 tickLow: 30 (OBC) .Realtime
    !! This tick was processed late! Primary bar time: 08:24:30 Secondary bar time:08:24:29
    BiP-1 26668 : 08:24:30 | Closes[0]: 38 tickHigh: 38 tickLow: 30 (OBC) .Realtime
    BiP-1 26669 : 08:24:31 | Closes[0]: 30 tickHigh: 38 tickLow: 30 (OBC) .Realtime
    BiP-1 26670 : 08:24:32.003 | Closes[0]: 22 tickHigh: 38 tickLow: 22 (OBC) .Realtime
    BiP-1 26671 : 08:24:33 | Closes[0]: 22 tickHigh: 38 tickLow: 22 (OBC) .Realtime


    and

    BiP-1 26903 : 08:28:25 | Closes[0]: -195 tickHigh: -153 tickLow: -243 (OBC) .Realtime
    BiP-1 26904 : 08:28:26 | Closes[0]: -187 tickHigh: -153 tickLow: -243 (OBC) .Realtime
    BiP-1 26905 : 08:28:27 | Closes[0]: -219 tickHigh: -153 tickLow: -243 (OBC) .Realtime
    BiP-1 26906 : 08:28:28 | Closes[0]: -189 tickHigh: -153 tickLow: -243 (OBC) .Realtime
    # BiP--0 896 : 08:28:30 | Closes[1]: -189 (OBC) .Realtime
    BiP-1 26907 : 08:28:29 | Closes[0]: -201 tickHigh: -201 tickLow: -201 (OBC) .Realtime
    !! This tick was processed late! Primary bar time: 08:28:30 Secondary bar time:08:28:29
    BiP-1 26908 : 08:28:30 | Closes[0]: -207 tickHigh: -201 tickLow: -207 (OBC) .Realtime
    BiP-1 26909 : 08:28:31 | Closes[0]: -153 tickHigh: -153 tickLow: -207 (OBC) .Realtime
    BiP-1 26910 : 08:28:32 | Closes[0]: -171 tickHigh: -153 tickLow: -207 (OBC) .Realtime
    BiP-1 26911 : 08:28:33 | Closes[0]: -157 tickHigh: -153 tickLow: -207 (OBC) .Realtime
    BiP-1 26912 : 08:28:34 | Closes[0]: -143 tickHigh: -143 tickLow: -207 (OBC) .Realtime
    BiP-1 26913 : 08:28:35 | Closes[0]: -161 tickHigh: -143 tickLow: -207 (OBC) .Realtime


    or sometimes BiP 0 processes to late, like this...
    BiP-1 16825 : 12:10:26 | Closes[0]: 111 tickHigh: 209 tickLow: 103 (OBC) .Realtime
    BiP-1 16826 : 12:10:27 | Closes[0]: 81 tickHigh: 209 tickLow: 81 (OBC) .Realtime
    BiP-1 16827 : 12:10:28 | Closes[0]: 55 tickHigh: 209 tickLow: 55 (OBC) .Realtime
    BiP-1 16828 : 12:10:29 | Closes[0]: 47 tickHigh: 209 tickLow: 47 (OBC) .Realtime
    BiP-1 16829 : 12:10:30 | Closes[0]: 37 tickHigh: 209 tickLow: 37 (OBC) .Realtime
    BiP-1 16830 : 12:10:31 | Closes[0]: 43 tickHigh: 209 tickLow: 37 (OBC) .Realtime
    BiP-1 16831 : 12:10:32 | Closes[0]: 33 tickHigh: 209 tickLow: 33 (OBC) .Realtime
    BiP-1 16832 : 12:10:33 | Closes[0]: 7 tickHigh: 209 tickLow: 7 (OBC) .Realtime
    BiP-1 16833 : 12:10:34 | Closes[0]: 15 tickHigh: 209 tickLow: 7 (OBC) .Realtime
    BiP-1 16834 : 12:10:35 | Closes[0]: 9 tickHigh: 209 tickLow: 7 (OBC) .Realtime
    # BiP--0 1121 : 12:10:30 | Closes[1]: 9 (OBC) .Realtime
    BiP-1 16835 : 12:10:36 | Closes[0]: -7 tickHigh: -7 tickLow: -7 (OBC) .Realtime
    BiP-1 16836 : 12:10:37 | Closes[0]: -53 tickHigh: -7 tickLow: -53 (OBC) .Realtime
    BiP-1 16837 : 12:10:38 | Closes[0]: -55 tickHigh: -7 tickLow: -55 (OBC) .Realtime


    When that occurs the wrong Close value is plotted.


    90% of the time the BiP is processed synchronously like this...
    BiP-1 26933 : 08:28:55 | Closes[0]: -149 tickHigh: -141 tickLow: -251 (OBC) .Realtime
    BiP-1 26934 : 08:28:56 | Closes[0]: -167 tickHigh: -141 tickLow: -251 (OBC) .Realtime
    BiP-1 26935 : 08:28:57 | Closes[0]: -141 tickHigh: -141 tickLow: -251 (OBC) .Realtime
    BiP-1 26936 : 08:28:58 | Closes[0]: -159 tickHigh: -141 tickLow: -251 (OBC) .Realtime
    BiP-1 26937 : 08:28:59 | Closes[0]: -147 tickHigh: -141 tickLow: -251 (OBC) .Realtime
    # BiP--0 897 : 08:29:00 | Closes[1]: -147 (OBC) .Realtime
    BiP-1 26938 : 08:29:00 | Closes[0]: -131 tickHigh: -131 tickLow: -131 (OBC) .Realtime
    BiP-1 26939 : 08:29:01 | Closes[0]: -79 tickHigh: -79 tickLow: -131 (OBC) .Realtime
    BiP-1 26940 : 08:29:02 | Closes[0]: -47 tickHigh: -47 tickLow: -131 (OBC) .Realtime
    BiP-1 26941 : 08:29:03 | Closes[0]: 1 tickHigh: 1 tickLow: -131 (OBC) .Realtime
    BiP-1 26942 : 08:29:04 | Closes[0]: 11 tickHigh: 11 tickLow: -131 (OBC) .Realtime

    Please find attached the indicator and a chart image, since chart templates can't be attached.

    Click image for larger version  Name:	Chart and Data box.png Views:	0 Size:	68.2 KB ID:	1076920Click image for larger version  Name:	Chart NQ and TICK.png Views:	0 Size:	50.0 KB ID:	1076917

    Is the indicator coded wrong? Does the ^Tick index 1-tick data series not synchronize correctly?
    Bottom line is how do we get the indicator to match the ^TICK bars OHLC values on the chart?
    Attached Files
    Last edited by zacwhitesi; 11-06-2019, 01:37 PM.
    zacwhitesi
    NinjaTrader Ecosystem Vendor - Shark Indicators

    #2
    Hello zacwhitesi,

    Thank you for your post.

    So that I may try to reproduce this on my end as accurately as possible, what data provider are you connected to? I'd first like to rule out that it's something with the data itself rather than a code issue first.

    Do you see this occur if you download Market Replay data for today and run it over the same time?

    Thanks in advance; I look forward to assisting you further
    Kate W.NinjaTrader Customer Service

    Comment


      #3
      Hi there, it's CQG.

      DEBUG [2019-11-06 07:02:01,357] - Certificate with subject [CN=www.cqgtrader.com, OU=Gateway, O=CQG Inc., L=Denver, S=Colorado, C=US] has passed verification at CertificatePolicy
      DEBUG [2019-11-06 07:02:01,364] - Certificate with subject [CN=www.cqgtrader.com, OU=Gateway, O=CQG Inc., L=Denver, S=Colorado, C=US] has passed verification at CertificatePolicy
      DEBUG [2019-11-06 07:02:01,367] - Certificate with subject [CN=www.cqgtrader.com, OU=Gateway, O=CQG Inc., L=Denver, S=Colorado, C=US] has passed verification at ServerCertificateValidationCallback
      INFO [2019-11-06 07:02:01,535] - ASLU: ReportClientActivity(******,*****) method calling at https://login.cqgnet.com/AUTHENTICAT...THSERVICE.ASMX auth service
      INFO [2019-11-06 07:02:01,699] - ASLU: ReportClientActivity method at https://login.cqgnet.com/AUTHENTICAT...THSERVICE.ASMX returned request status: Success
      Last edited by zacwhitesi; 11-06-2019, 03:14 PM.
      zacwhitesi
      NinjaTrader Ecosystem Vendor - Shark Indicators

      Comment


        #4
        Hello zacwhitesi,

        Thank you for your reply.

        I've run this for awhile today, and didn't see this occurring; however much of the day I wasn't running it on the same data feed.

        I'm going to run this for awhile longer tomorrow and see if I can get this to trigger.

        Thanks in advance; I look forward to resolving this for you.
        Kate W.NinjaTrader Customer Service

        Comment


          #5
          Is CQG sending the ^TICK index with a timestamp, or is Ninja assigning a timestamp when the tick is received?

          FYI, the ^TICK bars, as far as I can tell in my spot checking, always close on the correct tick. i.e. No extra tick or missing tick, like the indicator incurs.
          zacwhitesi
          NinjaTrader Ecosystem Vendor - Shark Indicators

          Comment


            #6
            Hello zacwhitesi,

            Thank you for your reply.

            CQG timestamps are native to their data. This makes me wonder if perhaps your computer clock is off just enough to cause an occasional discrepancy between the time a tick is stamped for and NinjaTrader's time functions which run off your computer clock in real time. If you look at historical data or run back over that time period in Market Replay, does the same issue occur?

            If it doesn't, it's likely the PC clock being off just a tiny little bit.

            Please use the steps below to sync the PC clock:
            • Shutdown NinjaTrader
            • Right-click the clock in the lower right corner of your desktop
            • Select Adjust date/time
            • For Windows 10 click 'Additional date, time, & regional settings' -> then click 'Set the time and date'
            • Select the 'Internet Time' tab at the top
            • Set the server to time.nist.gov and then click Update.
            • If the message that appears says successful your PC clock should now be updated.
            • If not, select a different server from the Server: drop-down and try again (repeat until one of the servers is successful)
            After updating the PC clock, please restart NinjaTrader, then right-click the chart and select Reload All Historical Data. Then test the indicator again on real time data.

            Please let us know if we may be of further assistance.
            Kate W.NinjaTrader Customer Service

            Comment


              #7
              I use a program called Dimension 4, which synchronizes the clock every time Windows starts up. The clock is usually adjusted by a few tenths of a second.
              I'll get back to you about the market playback test.
              zacwhitesi
              NinjaTrader Ecosystem Vendor - Shark Indicators

              Comment


                #8
                Yes, ^TICK ticks are also out of sync with Playback Connection.

                BiP-1 25194 : 07:59:56 | Closes[0]: 109 tickHigh: 115 tickLow: 86 (OBC) .Realtime
                BiP-1 25195 : 07:59:57 | Closes[0]: 99 tickHigh: 115 tickLow: 86 (OBC) .Realtime
                BiP-1 25196 : 07:59:58 | Closes[0]: 87 tickHigh: 115 tickLow: 86 (OBC) .Realtime
                # BiP--0 1679 : 08:00:00 | Closes[1]: 87 (OBC) .Realtime
                BiP-1 25197 : 07:59:59 | Closes[0]: 95 tickHigh: 95 tickLow: 95 (OBC) .Realtime
                !! This tick was processed late! Primary bar time: 08:00:00 Secondary bar time:07:59:59
                BiP-1 25198 : 08:00:00.023 | Closes[0]: 116 tickHigh: 116 tickLow: 95 (OBC) .Realtime
                BiP-1 25199 : 08:00:01 | Closes[0]: 76 tickHigh: 116 tickLow: 76 (OBC) .Realtime
                BiP-1 25200 : 08:00:02 | Closes[0]: 46 tickHigh: 116 tickLow: 46 (OBC) .Realtime

                BiP-1 25554 : 08:05:56 | Closes[0]: 231 tickHigh: 285 tickLow: 216 (OBC) .Realtime
                BiP-1 25555 : 08:05:57 | Closes[0]: 261 tickHigh: 285 tickLow: 216 (OBC) .Realtime
                BiP-1 25556 : 08:05:58 | Closes[0]: 243 tickHigh: 285 tickLow: 216 (OBC) .Realtime
                # BiP--0 1703 : 08:06:00 | Closes[1]: 243 (OBC) .Realtime
                BiP-1 25557 : 08:05:59 | Closes[0]: 213 tickHigh: 213 tickLow: 213 (OBC) .Realtime
                !! This tick was processed late! Primary bar time: 08:06:00 Secondary bar time:08:05:59
                BiP-1 25558 : 08:06:00.003 | Closes[0]: 189 tickHigh: 213 tickLow: 189 (OBC) .Realtime
                BiP-1 25559 : 08:06:01 | Closes[0]: 213 tickHigh: 213 tickLow: 189 (OBC) .Realtime
                BiP-1 25560 : 08:06:02 | Closes[0]: 205 tickHigh: 213 tickLow: 189 (OBC) .Realtime


                ect.
                zacwhitesi
                NinjaTrader Ecosystem Vendor - Shark Indicators

                Comment


                  #9
                  Hello zacwhitesi,

                  Thank you for your reply.

                  Just so I can make sure whether we're testing on identical data or not, did you download the market replay data from the Historical Data window, or do you have your NinjaTrader installation set up to record data for Market Replay?

                  Thanks in advance; I look forward to resolving this for you.
                  Kate W.NinjaTrader Customer Service

                  Comment


                    #10
                    It was downloaded. I do not record it.
                    zacwhitesi
                    NinjaTrader Ecosystem Vendor - Shark Indicators

                    Comment


                      #11
                      Hello zacwhitesi,

                      Thank you for your patience.

                      I've noticed this occurring as well and after taking another look at the code and data, the issue is that the secondary series isn't of the same instrument so ticks are coming in at different times.

                      Basically, you have your primary 15 second NQ series, and you've got a 1 tick ^TICK series. Let's look closely at a spot where we had an out of order print:

                      Click image for larger version

Name:	2019-11-11_1216.png
Views:	321
Size:	29.4 KB
ID:	1077364
                      Here we've got an "out of order" looking print. We can see that the 14:10:30 bar for BIP == 0 isn't printing until after what you'd think should be the first two seconds of that bar.

                      The thing is that until the first tick of that bar comes in, OnBarUpdate isn't going to be triggered because it doesn't have anything coming in to tell it it should be making a new bar.

                      If we pull up an NQ 15 second chart and then add in a single tick chart, we can see that there was a tick recorded at 14:10:29.219, but the next tick didn't come in until 14:10:32.059, triggering OnBarUpdate and printing what appears to be out of order.

                      Click image for larger version

Name:	2019-11-11_1246.png
Views:	295
Size:	68.5 KB
ID:	1077365Click image for larger version

Name:	2019-11-11_1248.png
Views:	301
Size:	72.4 KB
ID:	1077366Click image for larger version

Name:	2019-11-11_1248_001.png
Views:	322
Size:	76.4 KB
ID:	1077367

                      If you were using the same instrument and expiry for both the primary and the secondary series, you wouldn't have to worry about this, but because you're using two different instruments and mixing time based and non-time based intervals, you can't guarantee that ticks will come in exactly at the same times.

                      Please let us know if we may be of further assistance to you.
                      Kate W.NinjaTrader Customer Service

                      Comment


                        #12
                        Hi Kate,
                        Yeah, that makes sense. I realized that after posting that scenario, and should have removed it.
                        My main concern is when the ^TICK data is processed late, after the primary bar closes. See the first two Output window text from post #1, and post #8.
                        Thanks.
                        zacwhitesi
                        NinjaTrader Ecosystem Vendor - Shark Indicators

                        Comment


                          #13
                          Hello zacwhitesi,

                          Thank you for your patience.

                          I've been testing this further, and while I've encountered the situation I went over in my last post several times, I've yet to encounter what you're seeing.

                          I'm wondering if now if you may be experiencing some mild connection quality issues from time to time as that can cause quite a few different issues - if ticks are being slightly delayed in getting to you that could certainly cause the behavior you're seeing.

                          Would you be able to send your log and trace files in to us directly so I can see if there's any quality issues being reported? You can do so in the platform by going to Help > Mail to Support. Log and Trace should be checked automatically. Please put 2327531 ATTN Kate W. in the subject line and a link to this forum thread in the message body.

                          Thanks in advance; I look forward to assisting you further.
                          Kate W.NinjaTrader Customer Service

                          Comment


                            #14
                            Hello Zachary,

                            Thank you for your patience.

                            I just wanted to let you know that I am still awaiting word from our QA team on the issue of out of order ticks. It's fairly doubtful they'll get to this prior to the Thanksgiving holiday tomorrow, so I will continue to check with them and let you know anything I find out.

                            Thanks in advance; I look forward to resolving this item for you.

                            Sincerely,
                            Kate W
                            NinjaTrader Platform Customer Service
                            I hope you guys had a good Thanksgiving break.

                            Now that you've witnessed this issue, I don't need anymore interim updates. Please just post the work item number here, so other NinjaScripters can keep an eye on it, when the QA team creates one. Thanks team.
                            zacwhitesi
                            NinjaTrader Ecosystem Vendor - Shark Indicators

                            Comment


                              #15
                              Hello zacwhitesi,

                              Thank you for your patience.

                              I've already also sent this information below to your email, but I am also posting it here in case other users run into this.

                              I've heard back from our development team and unfortunately this is expected.

                              Data processing occurs on Instrument threads. The order of events will be guaranteed to be in order for a given instrument, but we cannot make the same guarantee between separate instruments/separate threads.

                              Data providers may provide a tick with an earlier time stamp for Instrument B after a tick with a later timestamp for Instrument A.

                              Observing the order of events that show the data entering the BarsType, we can see that the same order is kept in OnBarUpdate in an indicator. NinjaTrader receives these ticks in that order, and outputs them in the same order:

                              Click image for larger version  Name:	RealtimeBIPorder.png Views:	0 Size:	62.5 KB ID:	1079937

                              Please also keep in mind that with the Same Instrument/Same timestamp, we will see the primary bar close before the secondary bar. For example, if we have a 5 second bar as primary and a single tick bar as secondary, the single tick bar that closes the primary bar will be seen after the primary bar closes. This is documented in the Multi Time Frame and Instruments page of the Help Guide under Shared TimeStamps:

                              https://ninjatrader.com/support/help...meFrameObjects



                              Please let us know if we may be of further assistance to you.
                              Kate W.NinjaTrader Customer Service

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by PaulMohn, Today, 05:00 AM
                              0 responses
                              8 views
                              0 likes
                              Last Post PaulMohn  
                              Started by ZenCortexAuCost, Today, 04:24 AM
                              0 responses
                              6 views
                              0 likes
                              Last Post ZenCortexAuCost  
                              Started by ZenCortexAuCost, Today, 04:22 AM
                              0 responses
                              3 views
                              0 likes
                              Last Post ZenCortexAuCost  
                              Started by SantoshXX, Today, 03:09 AM
                              0 responses
                              16 views
                              0 likes
                              Last Post SantoshXX  
                              Started by DanielTynera, Today, 01:14 AM
                              0 responses
                              5 views
                              0 likes
                              Last Post DanielTynera  
                              Working...
                              X