Announcement

Collapse

Looking for a User App or Add-On built by the NinjaTrader community?

Visit NinjaTrader EcoSystem and our free User App Share!

Have a question for the NinjaScript developer community? Open a new thread in our NinjaScript File Sharing Discussion Forum!
See more
See less

Partner 728x90

Collapse

Equity backtesting - unrealistic fills

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Equity backtesting - unrealistic fills

    Hello, I'm noticing that when I backtest certain equity strategies on time bars, I'm getting a LOT of totally unrealistic fills because of the following situation:

    1) The strategy trades in the premarket and after hours, when the Bid/Ask will frequently move around a lot with no fills. Then there is a small order of 100 or so shares that all executes at one price, with a significant gap between this execution price and the close of the last bar. No other orders fill within the bar time, so it creates a bar with Open, High, Low, and Close all at the same price.

    2) If my strategy had a limit entry order between the Close of the prior bar and the price of the new bar, backtesting fills the limit order at the price of the new bar, because (if I understand it correctly) the historical fill algo follows its steps, skips over the large price gap, and chooses to fill my order at the only price available in the newest bar.

    3) Then over the next hour or longer, the bid/ask prices of that stock move back the other direction without any fills. Eventually there is a small fill with another large gap in price between the newest bar and the prior bar. Usually taking the stock back to where it started, because whoever placed that random lot order was an idiot. And now the backtest thinks I entered a position at an incredible price, and have a large profit.

    In real life, this would almost never, if ever, happen. If a stock had a last bar close price of 100, and its 4am and I have a live buy limit order in at 99, and the CEO gets caught smuggling a pangolin out of Vietnam, and the bid/ask prices RACE down to 90, it will fill my order on the way. Then say at 90 someone else buys 100 shares, creating a small bar. Then lets say my strategy puts a profit target $1 above the fill price, so now in live trading I am down 10% with an average price around 99, an exit limit order waiting at 100, and the current price at 90. Then time passes and the bid/ask prices move back up to 95 where someone else buys 100 shares. In live trading I'm still very negative.


    BUT if I then ran the same strategy backtested on that same time period (without me in it), the historical fill algo does this:

    Bar 1 Close at 100, Enter Limit Long at 99. (bid/ask moves prices down to 90, where there is a 1 lot fill) --> Bar 2 has Open = High = Low = Close = 90. So the fill algorithm fills my Enter Limit Order at 90 instead of 99. (bid/ask prices move back up to 95 where there is a 1 lot fill) --> Bar 3 has Open = High = Low = Close = 95, so the fill algo fills my Exit limit order at 95.

    Its a HUGE difference, where the back test reports a large unrealistic gain, but live trading would have resulted in a large loss. This situation occurs every day in premarket and after hours, all across the market. SO I have a few questions, if you wouldn't mind.

    - Would it be at all possible to not allow Limit orders to fill at prices better than the limit price? Or at least restrict how extreme it can be? I know limit orders can technically fill at a better price, but in the situation that happens every day in the after hours markets, with bid/ask prices moving all around but few or no fills occurring, they don't just jump over your limit order and hook you up with a great price... They give you the ****ty price and move the market on down. Is there a way to code this, or could this feature be added?

    - Is there a way to have a strategy or a backtest ignore tiny ticks way off of the prior close price? I know you can filter live ticks, but any chance this is coming to back testing?

    - And finally, one other kind of annoying thing is the lack of partial fills in back testing. If I am trying to write a strategy for a large amount of $, and trying to backtest it on a basket of instruments, the results can very unrealistic in thin markets as the fill algo fills my 200,000 share order on a bar with a volume of 100. Is there any way to at least limit my historical fills to something realistic? Like limited by the volume of the bar, or sum of a few recent bars, or anything to make it anywhere close to reality?

    Thanks so much! I always appreciate you help.

    #2
    Hello RobotSyndicate,

    Have you already implemented 1-tick intra-bar granularity for accurate historical order fills?
    https://ninjatrader.com/support/foru...377#post773377

    "Would it be at all possible to not allow Limit orders to fill at prices better than the limit price?"
    It would not be possible to change the rules of order types and how they fill.

    "Is there a way to have a strategy or a backtest ignore tiny ticks way off of the prior close price? "
    It would not be possible to change the backtest fill engine, but you can edit the historical tick data and remove any outlier ticks.
    https://ninjatrader.com/support/help...HistoricalData

    "Is there any way to at least limit my historical fills to something realistic?"
    Unfortunately, backtests cannot simulate part fills. (There fills in backtest are not based on volume like the simulator in realtime)
    I would instead recommend using the Playback connection to test in real-time and enforce partial fills.
    Chelsea B.NinjaTrader Customer Service

    Comment


      #3
      Hi Chelsea, I appreciate your time. I have used 1 tick intra bar granularity, and this actually makes the inaccurate fills even worse.

      I don't want to change the rules of order types and how they fill, I'm hoping that it would be possible to improve your backfill engine to make it MUCH more accurate in low volume markets. The backtest fill engine is fundamentally flawed and inaccurate in low volume markets typically seen in the early premarket and later after hours. It grossly overstates how often a limit order will be filled at a better price than the limit price in these types of markets. For example, if I backtest on the S&P over the past week, the fill engine will say that around 63% of my limit orders were filled at a better price (often significantly) than the limit price. My actual live results with the same data and same strategies (over 2000 orders during the period) was 10%. The end result is grossly overestimated backtest profit, with a significantly lower live trading profit (if any profit at all).

      This significant difference between backtesting and live trading makes NT backtesting in low volume markets dangerously inaccurate. But I would think that a relatively simple fix would be to add a feature where backtest limit orders could only be filled at the limit price. Just keeping everything else the same while preventing the backfill engine from assuming 63% of my orders will be filled at a price better than the limit price would help ensure that a backtested strategy would perform similarly, if not better, with live data.

      Is there any chance I could get a second opinion on this? I truly do love NT, have been using it daily for years with success, and I believe this would improve the program.

      -Best regards

      Comment


        #4
        Hello RobotSyndicate,

        When 1-tick intra-bar granularity is properly implemented, orders fill on the 1 tick series at the specific price and time of an individual tick. Meaning the order would only fill if there was a tick at that price (meaning a trade in real-time occured at that price).

        "I have used 1 tick intra bar granularity, and this actually makes the inaccurate fills even worse."

        This would be unlikely, what was limit price or stop price of the order?
        What was the historical tick data at the time of the fill?

        Please provide output showing the issue.

        Below is a link to a forum post that demonstrates how to use Print() and TraceOrder to understand behavior.
        https://ninjatrader.com/support/foru...121#post791121

        And a link to a forum post that demonstrates writing the information to file if this is extensive.
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Hi Chelsea,

          The issue is that in thin after hours trading, there aren't enough ticks to reflect real life price movement. Bid/Ask prices will move to reflect changes happening in the futures market, or overseas, or whatever, without fills occurring to give your backtesting fill engine anything to work with. I described it above in my first post in detail. Not sure if you would understand it without experience trading thin volume equities from 4am-7am or 6pm-8pm EST. But the result is a highly unrealistic backfill engine with thin equity markets. It doesn't matter what granularity is used, because the issue is that the real life bid/ask movement --between fills-- is not captured in the historical data. Bid/Ask could work its way down 5% without a fill. In real life the limit order would be filled along the way, but NT will only be able to work with the filled ticks rather than the actual real life bid/ask price movement. A simple solution would be to restrict the simulated limit fill price to the desired limit price.

          Thanks.
          Last edited by RobotSyndicate; 10-07-2022, 02:37 PM.

          Comment


            #6
            Hello RobotSyndicate,

            If there are ticks in the historical data, NinjaTrader is going process them in a backtest.

            With 1-tick intra-bar granularity implemented, order fills will only happen if there is a tick at that specific price and time.

            You can edit the historical tick data if you intended to remove these ticks.
            https://ninjatrader.com/support/help...HistoricalData

            Please confirm you have implemented 1-tick intra-bar granularity to increase the accuracy of historical execution fill prices.
            However, I would suggest you use the trading hours template of the exchange <use instrument settings> so that orders are not being filled or submitted outside of market hours.
            Chelsea B.NinjaTrader Customer Service

            Comment


              #7
              Yes I am using 1 tick granularity. I do not need to remove ticks. I understand that if there are ticks in the data, NT will use them. What I'm saying is that your fill engine results in over 6 times more favorable limit order price executions than would be achieved in lived trading. Are you saying that is intentional?
              Last edited by RobotSyndicate; 10-08-2022, 10:58 PM.

              Comment


                #8
                Hello RobotSyndicate,

                The results will follow the data. In historical the results will be as close to real-time if the orders are being submitted to the BarsInProgressIndex of a 1 tick series.

                The output written to text file will show you exactly what is causing differences.
                Below is a link to a forum post that discusses what causes differences and demonstrates how to write the information to file to analyze what is causing differences.
                https://ninjatrader.com/support/foru...nce#post100192

                "6 times more favorable limit order price executions" would be highly unlikely for a single order if the 1-tick intra-bar granularity has been properly implemented. This would imply you have not submitted the order to the BarsInProgressIndex of the 1 tick series in the order method.
                Chelsea B.NinjaTrader Customer Service

                Comment

                Latest Posts

                Collapse

                Topics Statistics Last Post
                Started by bmaltz, Yesterday, 07:15 PM
                2 responses
                13 views
                0 likes
                Last Post bmaltz
                by bmaltz
                 
                Started by iceman600, Yesterday, 01:51 PM
                4 responses
                25 views
                0 likes
                Last Post iceman600  
                Started by NinjaTrader_ChelseaB, 01-08-2017, 06:59 PM
                55 responses
                13,426 views
                3 likes
                Last Post grimes12  
                Started by Mindset, Yesterday, 05:43 AM
                4 responses
                36 views
                0 likes
                Last Post Mindset
                by Mindset
                 
                Started by brenton, Today, 08:31 PM
                0 responses
                15 views
                0 likes
                Last Post brenton
                by brenton
                 
                Working...
                X