• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

IsFirstTickOfBar vs OnBarClose for backtest & live

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    IsFirstTickOfBar vs OnBarClose for backtest & live

    I have a strategy that runs with calculation mode OnEachTick however in a backtest environment data is only processed OnBarClose.

    What I'm after is how to get an "OnBarClose" in a live OnEachTick call to OnBarUpdate.

    The reason for this is that I have to have separate logic to determine at what point we are looking at price (or any) data series.

    For example if I run Calculate.OnEachTick and I want to look for a simple thing such as the last candle closed UP I would have to do something like this:

    Code:
      if(IsFirstTickOfBar && Close[1] > Open[1])
         Print("Up Candle");
    This is fine and works the same in a back test BUT imagine we now want to take a trade :

    Code:
      if(IsFirstTickOfBar && Close[1] > Open[1])
        EnterLong(...);
    In a live environment this will immediately take the trade on the opening of the current candle. In a back test the trade won't get placed until the close of the current candle we are on. To fix this we need to move our logic to using Calculate.OnBarClose for a backtest which is also fine but then we have to do some silly stuff such as:

    Code:
      ...in State.Configure
      IsBackTest = Account.Name == Account.BackTestAccountName;
      
      if(IsBackTest)
         Calculate = Calculate.OnBarClose();
    
      ...in OnBarUpdate
      if(IsBackTest && Close[0] > Open[0])
         EnterLong(...);
      else if(IsFirstTickOfBar && !IsBackTest && Close[1] > Open[1])
         EnterLong(...);
    See where I'm going with this? I would like to keep processing utilizing the same data series index without having to do a bunch of extra logic throughout the code. Meaning can we still do Calculate.OnEachTick and know when the bar is actually closing to reference the current candle as index 0 vs index 1 that it would be for OnBarClose or during a backtest simply to keep our backtest entries in line with our live entries on the same bar?
    Last edited by fxRichard; 01-16-2017, 02:17 PM.

    #2
    Hello fxRichard,

    Please review the help guide document on the differences on real-time vs backtest (historical).
    http://ninjatrader.com/support/helpG...ime_vs_bac.htm

    When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

    Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick'.

    Below is a link to the help guide on Calculate.
    https://ninjatrader.com/support/help.../calculate.htm


    Adding intra-bar granularity can help with this.

    Intra-bar granularity adds a second data series such as a 1 tick series using AddDataSeries() so that the strategy or indicator has the individual ticks in the historical data in between the High and Low of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with. The orders placed will need to use the BarsInProgress (BarsArray) index of that finer resolution series.
    -Adding a 1 tick series with AddDataSeries() does not enable IsFirstTickOfBar to work correctly in historical data. (IsFirstTickOfBar will always be true)

    Below is a link to an official reference sample that demonstrates how to add intra-bar granularity.
    https://ninjatrader.com/support/helpGuides/nt8/en-us/backtesting_ninjascript_strate.htm

    Also, here is a link to the differences on real-time vs backtest (historical).
    http://ninjatrader.com/support/helpG...ime_vs_bac.htm

    As well as a link to the help guide on the AddDataSeries() method.
    http://ninjatrader.com/support/helpG...dataseries.htm

    A link to the help guide on BarsInProgress.
    http://ninjatrader.com/support/helpG...inprogress.htm

    And a link to the help guide on Multi-Time Frame & Instruments. Please see the section 'How Bar Data is Referenced', and 'Accessing the Price Data in a Multi-Bars NinjaScript'.
    http://ninjatrader.com/support/helpG...nstruments.htm

    The Fill limit orders on touch option will also change whether a limit order requires a tick through to fill or if it fills on touch, which can change the price at which a limit order fills and changes the end performance.
    https://ninjatrader.com/support/help...a_strategy.htm
    https://ninjatrader.com/support/help...mitontouch.htm

    Also, below I'm including a link to the help guide on Historical Order Backfill Logic.
    https://ninjatrader.com/support/help...fill_logic.htm



    In NinjaTrader 8, there have been two new enhancements so that programmers may not have to manually add this secondary series and code the script for high accuracy fills (Order Fill Resolution) or for intra-bar actions (TickReplay) depending on the needs of the script.

    The first is TickReplay.
    TickReplay is used for the intra-bar actions. This adds a 1 tick series to the script to be used for triggering OnBarUpdate when Calculate is set to .OnEachTick or .OnPriceChange. This also allows OnMarketData to trigger historically for last market events only (the bid and ask will not trigger).
    -TickReplay will allow for indicators to update for each tick or price change and for scripts to trigger actions intra-bar.
    -TickReplay allows for IsFirstTickOfBar to work correctly in historical data and will be true for the first tick of the bar only
    -TickReplay does not provide fill prices for accurate order fills.

    Below are links to the help guide on TickReplay.
    TickReplay - http://ninjatrader.com/support/helpG...ick_replay.htm
    Developing for Tick Replay - http://ninjatrader.com/support/helpG...ick_replay.htm
    OnMarketData - http://ninjatrader.com/support/helpG...marketdata.htm
    IsFirstTickOfBar - https://ninjatrader.com/support/help...ttickofbar.htm


    The second is Order Fill Resolution.
    This has two settings, Standard and High. When set to standard, the behavior of order fills will be accurate to the close of the bar without any intra-bar information. When set to High, this allow you to select a secondary data series to be used to provide intra-bar pricing information solely for the purpose of order fill accuracy. The highest you could set this to is 1 tick.
    -Order fill resolution does not provide intra-bar granularity to OnBarUpdate for actions to be triggered intra-bar.
    -
    Order fill resolution does not enable IsFirstTickOfBar to work correctly in historical data.

    Below is a link to the help guide on Order Fill Resolution. (Above this you can also read about the Historical Fill Algorithm)
    http://ninjatrader.com/support/helpG...FillResolution


    Currently, at this time, there is a known issue (tracked with ID# NTEIGHT-10782) where the High Fill Order Resolution and TickReplay cannot be used together. These work great when used on their own, however, when used together a complication arises that prevents the script from processing. Our developers are working hard to have this corrected in an upcoming build of NinjaTrader 8.

    In the meantime, if it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself with AddDataSeries() for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically and/or have IsFirstTickOfBar work correctly in historical data.


    It is not advised or necessary to change the Calculate mode during run-time.
    Last edited by NinjaTrader_ChelseaB; 07-31-2019, 07:40 AM.
    Chelsea B.NinjaTrader Customer Service

    Comment


      #3
      Thanks for your response Chelsea. I am familiar with backtesting vs real-time and don't understand why we don't have the capability to determine a bar close when doing Calculate.OnEachTick.

      I understand we don't have intrabar data but I would expect NT to have 2 callbacks something like: OnBarUpdate and OnBarClose etc.

      Not saying this is the solution but there is definitely a better way to handle real-time vs backtesting results and how NT processes data.

      NT has the capability to use Calculate.OnBarClose and reference the current bar at index 0 and evaluate that bar to place a trade at the OPEN of the next bar and get simulated fills at that open price.

      Just by moving to Calculate.OnEachTick we have no real way to know when the current bar closes. So NT has the IsFirstTickOfBar which tells us the last bar just closed but now we are referencing the last bar at index 1 and any calculations we take to place a trade won't happen until the open of the next bar. I understand the reasons why, there is no intrabar data etc, you don't know if the developer technically did that calculation intrabar and would have been filled mid bar etc.

      There is no elegant way in NT to process the current bar upon close with Calculate.OnEachTick to do something like I mentioned above.

      If NT can tell us when a bar closes and call onbarclose I would expect it would give us some way to know the same event when doing calculate.oneachtick.

      Comment


        #4
        Hello fxRichard,

        I'm not able to understand.

        You can tell a bar has closed with Calculate.OnEachTick with IsFirstTickOfBar.
        http://ninjatrader.com/support/helpG...ttickofbar.htm

        Can you clarify what you are not able to do?
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Thanks Chelsea.

          When using Calculate.OnEachTick and using IsFirstTickOfBar the last bar that closed is now index 1 in the Price data series.

          This presents a problem when placing orders after a calculation when a bar closes. Let's look at the two scenarios:

          Scenario 1 - Calculate.OnBarClose

          I want to evaluate the current closing bar, if it closes up I want to immediately take a market entry long, if it closes down I want to immediately go short.

          Code:
          if(Close[0] > Open[0])
              EnterLong();
          else if(Close[0] < Open[0])
              EnterShort();
          If I am running live the order will be immediately placed. If I am backtesting the order will be placed and "simulated filled" immediately on the opening of the next bar. Not a problem.

          Scenario 2 - Calculate.OnEachTick
          In this example we want to do the exact same thing except we don't know when a bar closes unless we use IsFirstTickOfBar. So let's assume the same scenario in which we now have to reference the closing candle at data series index 1 instead of 0 since we are on the first tick of the new bar:

          Code:
          if(Close[1] > Open[1])
              EnterLong();
          else if(Close[1] < Open[1])
              EnterShort();
          If I am running live the order will be immediately placed similarly to using OnBarClose. If I am backtesting the order will be placed and "simulated filled" at the open of the NEXT bar. This is expected and known due to the fact that there is no intrabar data unless you setup one of the scenarios above that you mentioned.


          I would expect there to be some way to handle order placement/fills in backtesting with OnEachTick similar to OnBarClose. To fix this problem we need to be able to do our calculation at the close of a bar while the closing bar is still at data series index 0 and not 1. Once it moves to 1 using "IsFirstTickOfBar" we then have this issue of the order being placed at the open of the following bar.

          Hope this helps clarify.

          Comment


            #6
            Hope this is what you are looking for.

            Code:
            	if (IsFirstTickOfBar)
            	{	
            		if(State == State.Historical)
            		{
            			openValue = Open[0];
            			closeValue = Close[0];					
            		}
            		else
            		if(State == State.Realtime)
            		{
            			openValue = Open[1];
            			closeValue = Close[1];		
            		}					
            	}

            Comment


              #7
              Hello fxRichard,

              Backtesting (historical) and real-time are fundamentally different.
              Below is a link to the help guide that details this.
              http://ninjatrader.com/support/helpG...ime_vs_bac.htm

              If you want intra-bar actions like triggering orders with Calculate.OnEachTick, you must have TickReplay enabled.

              May I confirm you have TickReplay enabled?
              Chelsea B.NinjaTrader Customer Service

              Comment


                #8
                Originally posted by tradertakeo View Post
                Hope this is what you are looking for.

                Code:
                	if (IsFirstTickOfBar)
                	{	
                		if(State == State.Historical)
                		{
                			openValue = Open[0];
                			closeValue = Close[0];					
                		}
                		else
                		if(State == State.Realtime)
                		{
                			openValue = Open[1];
                			closeValue = Close[1];		
                		}					
                	}

                Thanks for your response TraderTakeo, I am aware of this but what needs to be solved in NT is the capability to get a callback when a bar closes when using OnEachTick.

                When checking for IsFirstTickOfBar wich OnEachTick you get a "similar" but not same result as doing your calculation with OnBarClose. The biggest effect is when an actual order is placed when performing backtesting or running on historical data as any orders placed on the current bar will not be "filled" until the open of the next bar (without intrabar data).

                If NT simply had a callback to tell you when a bar is closing then the index value for all data series is still on the closing bar and any orders placed will be more in line with backtesting and real time (for simple order execution).

                This is not so much the issue of the index value but the order handling and processing in NT and when it happens.

                Comment


                  #9
                  Hello fxRichard,

                  To clarify, you are using NinjaTrader 8, you are placing an order after a bar has closed, the Order fill resolution is set to Standard, and the order is showing as being filled on the bar that triggered the condition?

                  With the order fill resolution set to standard this should not be the case.
                  Please let me know if you would like a short sample that demonstrates this.

                  You are using an index of [1] for the most recently closed bar, is this correct? (Using an index of [0] would be for the newly opened and currently building bar.)
                  Chelsea B.NinjaTrader Customer Service

                  Comment


                    #10
                    To piggy back off of fxRichard's frustration. He is right. The Open Price of the next candle when as order is placed is not always the same price as the Candle that just Closed. Actually, it almost never is. This leads to flawed results because the orders are placed at a different price than the Close of that Bar. I wrote a quick method shown below that you can test. You can see from the results I have uploaded that sometimes there is a huge variation in the ticks. If I am running this method on the
                    if(IsFirstTickOfBar
                    , I only expect to see a slight difference in one or two ticks.


                    My test settings.


                    Instrutment: EURUSD
                    Bar Type: Heiken Ashi
                    Base Period: Minute
                    Base Period Value: 5
                    Tick Replay: True



                    PHP Code:
                    if(IsFirstTickOfBarzWriteBarOpenCloseLog(); 
                    PHP Code:
                    public void zWriteBarOpenCloseLog()    
                            {

                                    
                    string barTime string.Format("{0:G}"Time[0]);

                                    
                    string BarOpenCloseLogName =  NinjaTrader.Core.Globals.UserDataDir "\\BarOpenClose.csv";
                                    
                    string BarOpenCloseLogHeader =

                                        
                    "Time"                "," +
                                        
                    "Barof BarOpenClose        "    "," +
                                        
                    "BarsInProgress"            "," +
                                        
                    "Open"                "," +
                                        
                    "Close"                "," +
                                        
                    "Previous Bar Close"    "," +
                                        
                    "Open[0] = Close[1]" "," +
                                        
                    "Difference in Ticks"         "," +
                                        
                    "Bar % Complete";


                                        
                    string diff Math.Round(((Math.Abs(Open[0]-Close[1]))/TickSize),2).ToString();
                                        Print(
                    diff);
                                        
                    string eq = (Open[0] == Close[1]) ? "TRUE" "FALSE";

                                    
                    string BarOpenCloseLogString =
                                        
                    barTime         "," +
                                        
                    CurrentBar     "," +
                                        
                    BarsInProgress         "," +
                                        
                    Open[0]            + "," +
                                        
                    Close[0]        + "," +
                                        
                    Close[1]        + "," +
                                        
                    eq                "," +
                                        
                    diff            "," +
                                        
                    String.Format("Value: {0:P2}."Bars.PercentComplete); // Does not work at all during backtest, only valid during realtime


                                    
                    try
                                    {
                                        if (!
                    File.Exists(BarOpenCloseLogName))
                                        {
                                            
                    using (StreamWriter sw6 = new StreamWriter(BarOpenCloseLogNametrue))
                                               {
                                            
                    sw6.WriteLine(BarOpenCloseLogHeader); // If file doesnt exist, create it and add the Header
                                            
                    sw6.WriteLine(BarOpenCloseLogString); // Append a new line to the file
                                            
                    sw6.Close(); // Close the file to allow future calls to access the file again.
                                            
                    }
                                        }
                                        else
                                        {
                                            if (!
                    zIsFileLocked(BarOpenCloseLogName)) //If file is not locked for editing
                                            
                    {

                                                
                    using (StreamWriter sw6 = new StreamWriter(BarOpenCloseLogNametrue))
                                                   {    
                                                
                    sw6.WriteLine(BarOpenCloseLogString); // Append a new line to the file
                                                
                    sw6.Close(); // Close the file to allow future calls to access the file again.
                                                
                    }
                                            }
                                            else
                                            {

                                            }

                                        }
                                        }
                                        catch (
                    Exception e)
                                        {
                                            
                    // Outputs the error to the log
                                            
                    Log("BarOpenCloseLog - You cannot write and read from the same file at the same time."NinjaTrader.Cbi.LogLevel.Error);
                                            Print(
                    e.ToString());
                                        }

                                } 




                    Attached Files
                    Last edited by cutzpr; 11-25-2018, 06:42 PM.

                    Comment


                      #11
                      Hello cutzpr,

                      Because of how the orders are being placed, this is expected behavior.

                      You would need to place the order before the bar closes with intra-bar granularity if you would an order to be placed (and possibly filled) on the bar where the condition was true.
                      https://ninjatrader.com/support/foru...377#post773377
                      Chelsea B.NinjaTrader Customer Service

                      Comment


                        #12
                        The only way I found to get around this was to add a duplicate data series exactly the same as the primary.
                        Create a bool enter/exit variable that the Primary data series would modify
                        Do no processing on the secondary data series except submit your orders. Since it is immediately processed by the strategy after it is done processing the primary data series.

                        This is the closest thing I can do to mimic a callback that the current bar is closed.
                        Last edited by cutzpr; 11-30-2018, 10:08 PM.

                        Comment


                          #13
                          Hello cutzpr,

                          To confirm, you have tested adding a 1 tick series to the script, and OnBarUpdate did not trigger for each tick?

                          You are wanting to place an order before the bar closes, is this correct?
                          Chelsea B.NinjaTrader Customer Service

                          Comment


                            #14
                            Hi Chelsea,

                            Originally posted by NinjaTrader_ChelseaB View Post
                            In the meantime, if it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically.
                            Can you elaborate on this? Correct me if I'm wrong, but my understanding is this:
                            • If you want accurate fills (i.e. EnterLong() be submitted immediately, not wait until the end of the bar), use Order fill resolution = High
                            • If you want to evaluate and decide on a buy during a bar, not after its closed, use TickReplay
                            • Order fill resolution = High and TickReplay do not work together (due to an issue with the current NT8 software)
                            So not understanding your statement about "if it is necessary to have both"... how do i get my EnterLong() method to execute before the end of the current bar? I don't have any control over that do I?

                            Thanks

                            Comment


                              #15
                              Hello buzzripper,

                              Add a 1 tick series for intra-bar granularity. Submit the order when the tick series is updating before the primary series bar closes. Place the order to the 1 tick series bars in progress so the order fills with accurate price data.

                              Please see the Backtesting NinjaScript Strategies with an intrabar granularity reference sample that demonstrates intra-bar granularity.
                              Chelsea B.NinjaTrader Customer Service

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by RB104, Today, 06:06 PM
                              1 response
                              10 views
                              0 likes
                              Last Post NinjaTrader_ShawnB  
                              Started by jtrading82, 05-25-2018, 09:59 AM
                              9 responses
                              978 views
                              0 likes
                              Last Post NinjaTrader_ShawnB  
                              Started by syxforex, Today, 12:11 AM
                              2 responses
                              15 views
                              0 likes
                              Last Post syxforex  
                              Started by aslane, Today, 02:31 PM
                              1 response
                              10 views
                              0 likes
                              Last Post NinjaTrader_Kate  
                              Started by stoner, 10-16-2019, 01:38 PM
                              5 responses
                              52 views
                              0 likes
                              Last Post sledge
                              by sledge
                               
                              Working...
                              X