• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

OnMarketData getting wrong/too many ticks for tick replay

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    OnMarketData getting wrong/too many ticks for tick replay

    I was having strange issues with getting wrong data when using tick replay. I wrote an isolated indicator to identify some of the problems.

    The main issue I was having was that at the end of a data load CurrentBar*Bars.BarsPeriod.Value+Bars.TickCount would be different from Bars.TotalTicks

    I was not able to reproduce that problem with this indicator but it does show another strangeness which is if you load it on a 10 tick chart with only 10 bars it gets over 50000 bars rather than 10.

    This is meant to be used when tick replay is turned on and Calculate==Calculate.OnEachTick

    You will probably not be able to reproduce this bug immediately, like many NT bugs it only happens after a chart has been used for a while and some of the internal data structures have somehow gotten corrupted.

    In the screenshot it shows the 10 bar 10 tick chart settings and the log lines that show it got the wrong data, such as "Error: [CurrentBar>expectedMaxCurrentBar] [54592>11] CurrentBar=54592 State=Historical"

    I am attaching the source below. It may be good to have it in the background and see if it detects any bugs after doing other things in NT for a few hours.

    Code:
    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.ComponentModel.DataAnnotations;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Windows;
    using System.Windows.Input;
    using System.Windows.Media;
    using System.Xml.Serialization;
    using NinjaTrader.Cbi;
    using NinjaTrader.Gui;
    using NinjaTrader.Gui.Chart;
    using NinjaTrader.Gui.SuperDom;
    using NinjaTrader.Data;
    using NinjaTrader.NinjaScript;
    using NinjaTrader.Core.FloatingPoint;
    using NinjaTrader.NinjaScript.DrawingTools;
    
    namespace NinjaTrader.NinjaScript.Indicators
    {
        public class TickReplayTest : Indicator
        {
            protected override void OnStateChange()
            {
                if(State==State.SetDefaults)
                {
                    Description=Name="TickReplayTest";
                    IsSuspendedWhileInactive=false;
                    Calculate=Calculate.OnEachTick;
                }
                else if(State==State.DataLoaded)
                {
                    if(!Bars.IsTickReplay) throw new Exception("!Bars.IsTickReplay");
                        
                    if(Calculate!=Calculate.OnEachTick) throw new Exception("Calculate!=Calculate.OnEachTick");
                }
            }
            
            bool threwException=false;
            
            int numInputTicksRawCount;
            
            protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
            {
                try
                {
                    // 10 tick 10 bars causes problems
                    int expectedMaxCurrentBar=11;
                    
                    //if(CurrentBar>9000) return; //!!
                    
                    if(threwException) return;
                    
                    if(//State!=State.Historical || //!!
                        marketDataUpdate.MarketDataType!=MarketDataType.Last)
                    {
                        return;
                    }
                    
                    numInputTicksRawCount++;
                        
                    int numInputTicksRawCalculated=CurrentBar*Bars.BarsPeriod.Value+Bars.TickCount;
                    
                    Print("[numInputTicksRawCount="+numInputTicksRawCount+"] [numInputTicksRawCalculated="+numInputTicksRawCalculated+"] CurrentBar="+CurrentBar);
                    
                    if(numInputTicksRawCount!=numInputTicksRawCalculated)
                        throw new Exception("Error: [numInputTicksRawCount!=numInputTicksRawCalculated] ["+numInputTicksRawCount+"!="+numInputTicksRawCalculated+"] CurrentBar="+CurrentBar);        
    
                    if(Bars.TotalTicks!=numInputTicksRawCalculated)
                        throw new Exception("Error: [Bars.TotalTicks!=numInputTicksRawCalculated] ["+Bars.TotalTicks+"!="+numInputTicksRawCalculated+"] CurrentBar="+CurrentBar);        
                    
                    if(CurrentBar>expectedMaxCurrentBar)
                        Print("Error: [CurrentBar>expectedMaxCurrentBar] ["+CurrentBar+">"+expectedMaxCurrentBar+"] CurrentBar="+CurrentBar+" State="+State);
                }
                catch(Exception e)
                {
                    Print(e.ToString());
                    threwException=true;
                }
            }
        }
    }
    Attached Files

    #2
    Hello meinl,

    Thank you for writing in.

    I am looking into this further and will report back with my findings.

    Thank you for your patience.
    Zachary G.NinjaTrader Customer Service

    Comment


      #3
      Hello meinl,

      I did want to note that if connected to a data feed, OnMarketData() will still be called on every change of level I market data despite it not showing up on your chart. This is why your numInputTicksRawCount variable will still increment despite numInputTicksRawCalculated still staying the same.

      I'm unable to see this behavior with CurrentBar occur on my testing with the code you have provided.

      As a test, can you please remove the historical data that you have pertaining to the CL ##-##? Once you have done this, ensure that you retrieve the data needed once more. Then, test again.

      Additionally, can you verify that you are running Beta 12? Click on Help -> About in the Control Center. The version should show up at 8.0.0.12 if you have Beta 12 installed.
      Zachary G.NinjaTrader Customer Service

      Comment


        #4
        This is on 8.0.0.12 64-bit (Standard) - BETA and was actually a very clean install in which I had wiped out all my old data and was starting from scratch.

        I was getting 50000 bars more than I was expecting and was doing this at night when the markets are mostly asleep. I don't think I even had it connected to the internet, but even so 50000 additional bars sounds to me like data corruption. I can believe a few hundred ticks if I had been doing that during heavy trading hours, but not 50k during the time the script loaded 10 bars of 10 tick data.

        I'll try removing the data once more and will see if it happens but even if it does, what difference will it make? If you can't recreate it, then we'll just be back to me going around in a circle again, deleting my database and redownloading it.

        I appreciate your help, but really what I want more than the suggestion to delete my data is for you to add this to your unit testing.

        I would like to know that the automatic QA process you use has this code as a test, and that in your nightly builds you have test copies of NT that randomly look for scenarios in which things that should be tautologically true are false.

        I just spent about a month emailing bug reports for other problems I've found and I don't need a suggestion to delete my data and download it again, I've been told that dozens of times. What I need is to know you're taking steps to make sure these conditions I've identified are being checked for in your automatic unit testing.

        I appreciate your help, can you let me know that you have a unit test for this?

        Comment


          #5
          Hello meinl,
          What development does for testing is not available. When you send in your log and trace files from your other ticket with me I will look into this also.
          BrandonNinjaTrader Customer Service

          Comment


            #6
            Meinl,

            Thanks for taking the time to write up the bug report and sorry about the delete data request.

            I can let you know that on the development team we use a continuous build process where unit tests are run on each build. These unit tests are traditionally built strictly for catching regressions. They wont actually help us find bug scenarios which we're actually not aware of or haven't run into prior and setup tests.

            In your case below I actually see two separate things, correct me if I'm wrong:

            * TickCounter being a large value when using 'Bars Back' option to load data. This case looks expected but I will check into further. The reason its expected is that when using bars back option to load data we must first build the bars and then count back to only show the requested number of bars on a chart. Therefor an underlying bar series could have a lot more data in it then you actually requested under the hood since we needed to build the bars before we could display them.
            * The primary issue you posted here: "CurrentBar*Bars.BarsPeriod.Value+Bars.TickCou nt would be different from Bars.TotalTicks". If I'm understanding the scenario this one is expected as well depending 'Break At EOD' setting. Since a tick bar at the end of a session could be broken sooner then before it receives all 10 ticks since we break the bars at EOD. If you turn that feature off when creating the chart and you still see this problem then I'll promptly look into further.
            BrettNinjaTrader Product Management

            Comment

            Latest Posts

            Collapse

            Topics Statistics Last Post
            Started by Vincere, Yesterday, 11:20 PM
            0 responses
            5 views
            0 likes
            Last Post Vincere
            by Vincere
             
            Started by lMagi, 05-18-2019, 09:34 AM
            1 response
            4 views
            0 likes
            Last Post NinjaTrader_ChrisSch  
            Started by uday12, Yesterday, 10:26 PM
            0 responses
            3 views
            0 likes
            Last Post uday12
            by uday12
             
            Started by uday12, Yesterday, 10:07 PM
            0 responses
            0 views
            0 likes
            Last Post uday12
            by uday12
             
            Started by Nabil, Yesterday, 07:46 PM
            1 response
            5 views
            0 likes
            Last Post NinjaTrader_ChelseaB  
            Working...
            X