Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Are the back tested results accurate for all Chart Styles and Data Series Types?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Are the back tested results accurate for all Chart Styles and Data Series Types?

    I am new to NT and just getting familiar with the product and spend a great deal of time back testing strategies and have a simple and for me critically important for NT support staff.

    Are the the Strategy Analyzer back tested results, except for slippage, accurate for all Chart Styles and Data Series types if Order Fill Resolution = High and Calculate on Tick is specified?

    If the answer is no, then for what Chart Styles and Data Series types are the back tested Strategy Analyzer test results not accurate?

    The same question for candle/bar chart based strategies back testing using multiple time interval data series?

    Before I invest a lot of time and potentially money relying on the Strategy Analyze and the "Optimized" results I would like to know if these results are accurate and is there any product documentation discussing this topic?

    I am so hoping that unlike eSignal where the back testing results for other then bar/candlestick charts and for all chart types using multi time frame results are highly inaccurate that NT would not have the same inaccuracies in the back tester.

    Thank you very much

    Glen
    Last edited by demarcog; 09-05-2020, 09:37 PM.

    #2
    Define 'accurate'.

    Do you mean 'exact'?

    Except maybe for Renko bars, where back-testing problems are well-documented,
    you can't expect your order fills in back-testing to exactly match live order fills on a
    live exchange for every order. (Even if they do match on a particular day, there is no
    guarantee they will continue to match the next day. In fact, there is absolutely no
    guarantee they will ever match, period. End of story. No guarantee, none, zilch.)

    What market are you trading?
    Are you trading 1 contract or 10?
    Are you using Market orders for entry, or Stop, Limit, or StopLimit?
    Will back-testing accurately mimic slippage on every order?
    Will back-testing accurately account for other market participants on every order?

    For example,
    I don't think back-testing has any knowledge of queue depth. Long entry Limit orders
    resting at the exchange may not fill even though it appears Market price has 'touched'
    the Limit price. I mean, it's quite possible for some Limit orders to be so deep in the
    queue that Buy Market orders arrive and the Market price moves higher, away from
    the Limit price, leaving some orders at that Limit price unfilled. If price never falls back
    to that Limit price and never fills those remaining orders, well ... such is life, right?

    But, later, when you back-test using Strategy Analyzer, it may report a filled entry order
    at that Limit price, because price touched the Limit price, but never traded through it,
    but it still touched it, so it records a fill at that price.

    Or, consider the exact opposite scenario, where your order in the order queue is filled
    in the live market, but in Strategy Analyzer because the Limit price was only touched
    and not traded through, the back-tester fails to record a filled order.

    [I'm not sure which algorithm the back-tester uses for fills. I'm just saying there are
    lots of edge cases and that it can be very hard for back-testing results to be exactly
    equivalent to Real-time data results on a live account or Sim101 account. Besides,
    there is never a guarantee of exactness -- even if they do happen to be exact.]

    If you want to understand the 'accuracy' and/or 'exactness' of the Strategy Analyzer,
    I recommend you compare live Sim101 results to the corresponding back-testing
    results for the same period, which means you first need to have the live Sim101
    results as a baseline. Given enough days to compare, you can decide for yourself
    just how well your chosen strategy and chosen bar type(s) perform in back-testing.
    (Market Replay can also help give you an initial baseline of results.)

    IMHO, perhaps accounting for few percentage points difference due to slippage and
    unforeseen edge cases, it's going to depend heavily upon your specific strategy.

    Therefore, the short answer to the question in your title is: No.
    But, perhaps, if you redefine 'accurate' to be 'x% less than 100% exact', you may
    find back-test results satisfactory for certain input bar series -- but this is still highly
    dependent on your strategy. You just need to find what your 'x' is.

    Just my 2˘.

    Good reading here and here.

    Comment


      #3
      no the backest results for anything except minute bars is wrong. You will need to forward test your strategy to see if its viable when using certain

      Comment


        #4
        Hello Glen,

        There are differences on how historical data is processed.

        For more accurate results, I recommend 1 tick granularity and TickReplay. Below is a link to a forum post that details.
        https://ninjatrader.com/support/foru...377#post773377

        That said, some bar types change their values in real-time which prevents an accurate backtest as bltdavid has described for renko bars.
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Originally posted by Teebone21 View Post
          no the backest results for anything except minute bars is wrong. You will need to forward test your strategy to see if its viable when using certain
          Thank you very much Teebone21 I was hoping this was not the case but appreciate you confirming what I suspected.

          Can you suggest the best was to forward test the strategy and excuse the dumb question but I'm new to NT. I

          I must be missing something on how to forward test in real time. The last product I used I setup a Demo account and let it run for whatever period of time. Their equivalent of the Chart Trader would track the PnL in the Chart Trader's PnL box for however many trades, days, or indefinite period the strategy was running.

          I would look at the back test report and compare that to the executions in real time for comparasion purposes. I'm obviously doing something wrong as I load and run the automated strategy in SIM101 account and the PnL get's reset when a trade is closed out, and it looks like at EOD the balances get reset to zero for realized and unrealized.

          I obviously don't know what I'm doing and appreciate the help, from everyone who has been kind enough to assist this basket case.

          Thanks again,

          glen

          Comment


            #6
            Hi Bltdavid,

            Thank you very much for that insightful response. You obviously have tremendous expertise on order processing among may other areas I'm certain and I learned quite a bit reading through the reply. If that is your 2 cents answer I can only imagine the 10 cent response.

            Let me try and clarify some of the issues I was trying to raise and answer a few of the question you posed.

            By "accurate" and I should have made that clearer, apologies. I meant excluding anything that could even remotely be accounted for by "slippage". You raise a good point as it very much depends on what market and what size. I was referring to the ES front month contract and trading a 1 lot. The difference I was asking about goes well beyond any of the comprehensive order queue conditions you raised. I was referring to discrepancies that go well beyond 1% or 2%, winning strategies 75% Win Pct that drops to 35%.

            I'm a bit of a fanatic regarding back testing, been doing it for decades I'm embarrassed to admit. I have tested every conceivable idea published in any trading book or magazine where I could get the code or wrote it myself and shocked how difficult it is to find anything worth trading and the simpler the better.

            You mentioned that the issue of accuracy was well documented for Renko, can you please point out some of that documentation. I did come across some documentation on Point and Figure and back testing inaccuracies and curious about the other Chart types, Unirenko, Point Break, Range bars.

            I used eSignal for years and recently opened an account with NT. I was blown away with how fantastic the product is and very optimistic that with the capabilities of NT unlike other products I've used in the past that perhaps not a Holy Grail system can be found but one where a reasonable living can be made in the markets.

            I know traders aren't "supposed" to help each other as there is the element of a zero sum game obviously. I was hoping however that the NT community could be a place to collaborate on developing a profitable strategy/methodology without giving away the "secret sauce". It would be interesting to share a strategy with a certain degree of profitability (or not) and as a group see if we could improve upon the profitability.

            I've only had NT for a week or so but did download every strategy available and going through the back testing now. Minute charts were not necessarily my first choice for a Data Series which is why before I spend too much time back testing 100 strategies wanted to know if there were back testing accuracy issues on other chart type and very much appreciate you help.

            Thanks Again,

            [email protected]



            Comment


              #7
              Hello Glen,

              Adding 1 tick intra-granularity can improve backtest order fill accuracy for bar types that do not use IsRemoveLastBar support.
              https://ninjatrader.com/support/foru...377#post773377

              With ChartTrader do you have 'Show gross realized PnL when flat' checked and enabled?
              https://ninjatrader.com/support/help...AndDefinitions
              Chelsea B.NinjaTrader Customer Service

              Comment


                #8
                HI Chelsea,

                I am using the 1 tick granularity and Order Fill Resolution = High.

                I will run with Show Gross realized PnL checked but not clear how to enable it beyond checking the box?

                Thank you for your help, it is very much appreciated.

                glen

                Comment


                  #9
                  Hello Glen,

                  When 'Show gross realized PnL when flat'' the pnl will show for the selected account, as shown on the Accounts tab of the control center when the position is flat.
                  Chelsea B.NinjaTrader Customer Service

                  Comment


                    #10
                    Hi Chelsea,

                    I misunderstood when I Check the box Show Gross realized PnL I thought that would be for that particular symbol. I can see the PnL for the account but interested in the PnL for that symbol?

                    thank you.

                    glen

                    Comment


                      #11
                      Hello Glen,

                      Unfortunately, this shows the account pnl and not specific to the symbol, as this is the value being supplied from the connected broker or from the Sim101 account.

                      You might consider creating a custom coded indicator that loops through the executions (<execution>.Order.Instrument) on an account for a specific instrument and adds up the pnl.
                      https://ninjatrader.com/support/help...executions.htm

                      Below are a few scripts that may give you some ideas.
                      https://ninjatrader.com/support/foru...ples#post93881
                      https://ninjatraderecosystem.com/use...lay-indicator/
                      Chelsea B.NinjaTrader Customer Service

                      Comment


                        #12
                        Hi Chelsea,

                        Thank you for the suggestion and example links for a coding solution.

                        Aside from writing code once I have a strategy that back test's well and please excuse my NT ignorance as it is a very rudimentary general question.

                        What is the next step in monitoring the strategy real time, it i suggested to only run one strategy on one symbol at at a time and dedicate one Sim account to that configuration?

                        Normally I would load a strategy into a chart and let it run real time.

                        I would see the strategies executing trades real time where the chart/strategy would have a one to one relationship and at anytime I could go into the Account Manager which would log the trades executed on the screen in the Account Manager which would match exactly with the buys/sells on the screen.

                        In addition I could reload the strategy loaded on the chart and open the back test report at any time and compare the back test reported trades with the real time. The other difference was the Chart Trader as we mentioned would track the PnL for the specific symbol. That is not a big issue as the Account Manager would also keep track on a symbol/strategy basis.

                        This is the functionality I am looking to replicate on NT and sure the answer is right in front of my face but I am looking at the forest and not the trees?

                        thank you

                        glen

                        Comment


                          #13
                          Hello Glen,

                          Below is a link to the Strategy Builder 301 video which demonstrates how to run a NinjaScript Strategy in real-time.
                          https://www.youtube.com/watch?v=HCyt...utu.be&t=1h25s

                          As well as videos on how to backtest and optimize.
                          https://youtu.be/RaAWxf2jxp8
                          https://youtu.be/kjnyYlzzog8
                          https://youtu.be/SxbGhhkKtXk


                          As far as monitoring in real-time, what do you want to monitor?

                          The Strategies tab of the Control Center will show the position and pnl of the strategy.
                          https://ninjatrader.com/support/help...eStrategiesTab

                          Orders and trades can be viewed in the Strategy Performance window.
                          https://ninjatrader.com/support/help...egyPerformance
                          https://ninjatrader.com/support/help...erformance.htm


                          I recommend only running one strategy per instrument and account combination, but this is up to you.
                          https://ninjatrader.com/support/foru...660#post792660

                          Last, below is a link to a forum post with helpful information about getting started with NinjaScript and C#.
                          https://ninjatrader.com/support/foru...040#post786040
                          Chelsea B.NinjaTrader Customer Service

                          Comment


                            #14
                            Chelsea,

                            Thank you for the response that is exactly what I was looking for.

                            I appreciate your patience and assistance in getting me started working with strategies.

                            Glen.

                            Comment

                            Latest Posts

                            Collapse

                            Topics Statistics Last Post
                            Started by NinjaTrader_Brandon, Today, 07:49 AM
                            0 responses
                            3 views
                            0 likes
                            Last Post NinjaTrader_Brandon  
                            Started by pdennis, Today, 07:43 AM
                            0 responses
                            1 view
                            0 likes
                            Last Post pdennis
                            by pdennis
                             
                            Started by b16_aln, Today, 05:15 AM
                            1 response
                            6 views
                            0 likes
                            Last Post NinjaTrader_Jesse  
                            Started by Negustrader, Today, 01:01 AM
                            1 response
                            15 views
                            0 likes
                            Last Post NinjaTrader_Jesse  
                            Started by soon123, Yesterday, 10:50 PM
                            1 response
                            9 views
                            0 likes
                            Last Post NinjaTrader_ChrisL  
                            Working...
                            X