Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Total Net Profit very different- optimiser display vs Trades display

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Total Net Profit very different- optimiser display vs Trades display

    Hi,
    can't explain this one... I am reviewing some optimisations I did recently, and noticed discrepancies... sometimes fairly huge, between the TNP reported by the optimiser and the Cumulative Profit shown in the Trades list. Example shown is fairly extreme...

    I'm inclined to believe the Trades list... which also agrees with the graph of Cumulative Net Profit. It would be reassuring if they both matched the data shown in the top 10 list!

    And yes I am showing the last of the 22 trades made, so cumulative profit should be equal to Total net profit...
    I found a similar post ... but it was way back in 2009... I hope things have changed since then !
    Click image for larger version  Name:	What profit did this strategy make.png Views:	1 Size:	49.1 KB ID:	1061334
    See attached pic

    Emphasize: this occurs when reviewing runs saved in the Log tab. Seems to work fine on a fresh optimisation... so maybe it's in the logic for pulling results out of the database?

    Regards,
    Last edited by tgn55; 06-20-2019, 02:57 AM.

    #2
    Hello tgn55,

    Thank you for the post.

    You said this happens with saved results specifically, can you see this happen with the SampleMACrossOver strategy?

    I tried a few quick tests to see if I could recreate a difference in saved results, but I was unable to see a difference. Does this happen if you open the saved result after another test, for example run a test, run another test and revert back to the saved result?

    Could you provide more specific steps of the process you followed when you saw this?

    I look forward to being of further assistance.
    JesseNinjaTrader Customer Service

    Comment


      #3
      Hi Jesse,
      tried with Sample MA... all looked fine. I then went back to my ZB log entry ... and it also looked fine! But... I then noticed that I had run another optimization on GC in between. The wonky results came from my earlier run...

      So, I have two different optimization runs on ZB in the log, with a GC run in between. But same strategy, same parameters. One is kosher... the other isn't.
      I actually exported the trades showing the $35K trades profit the other day, so I was able to now export the trades with the correct (?) trades, at least what matches the $55K Total Net Profit.
      Same number of trades, but some exited at different times/prices. On examination, I suspect I can partly explain... I refer to IsRising on an SMA, which in all three cases where the trades differ, the SMA is pretty flat. So, maybe a comparison of floating point numbers thing is going on...

      However... that surely does not really explain why the reported trades in both cases show nearly $55K profit in one pane... and $35K in the panes (although that does match the trades). Surely it should be internally consistent? I'm seeing a a strange mix of the two trade collections!

      Can I email you the Excel docs I have? It would make it clearer, I have highlighted the differences. Only 22 trades, and a few differences, so not too onerous to check.. if you wish, although I'm not sure it really helps decipher the mysterious reporting.

      Regards,

      Comment


        #4
        Hello tgn55,

        Thank you for the additional details.

        I don't believe the excel documents would be helpful at this point, the image you provided clearly shows the difference.

        What I can pull from this description is that you ran the ZB -> GC -> ZB, do you always see that happen when doing tests between different instruments in that order?

        You also noted the SampleMACrossOver seemed fine, if that is the case it is possible that the problem is isolated to your custom strategies logic in some way.

        From this point, I would suggest that we try to find a situation where your strategy can more or less reliably fail in reporting the values. Once that situation is found we can confirm if that happens with the Sample again to make sure the problem is or is not isolated to your script. This may be one reason I am unable to see the problem at the moment.

        If this happens to be isolated to your script, you can run some additional tests by reducing the scripts logic. Making the script more simple or temporarily removing parts of its logic can be used to tell if that is part of the problem or not. Another approach is to comment out your logic, and copy the OnBarUpdate/variables from the SampleMaCrossOver to see if the script now works. These type of tests could be used to further extract the problem if it is something in the logic.



        I look forward to being of further assistance.





        JesseNinjaTrader Customer Service

        Comment


          #5
          I'll see what I can do Jesse. I don't really believe it's my strategy logic... which is a kind of crossover, pretty basic, and I have reasonable experience coding. I'm always keen to nail any nasty gremlins hiding in NT, I'll probably return to this next week. Tonight (I'm in Australia, its already after 11 PM ;-) I'm going to be focusing on making the strategy more robust... bottom line, I will come back to follow up... but probably will not be tonight!

          And I'll ensure I capture exactly what I do ... assuming I can reproduce the weirdness!

          Cheers,

          Comment


            #6
            Hey Jesse... me again. Just reviewing things... it's been a couple months since I posted this, I got kinda distracted with other SA issues...

            I have just posted another issue with dodgy results in the SA area,
            https://ninjatrader.com/support/foru...-inconsistency

            Not clear if they might be related, I can't really see how... but thought it might be worth cross-referencing just in case.
            I have not yet got any further with this particular anomaly... I am already spending about half my time submitting bug reports! Will try to return to this issue and see if I can create some further info.

            Meanwhile... try the above link. It's pretty whacko stuff...

            Cheers,
            T.

            Comment


              #7
              So... I have continued to be plagued by these inconsistent reports periodically since I started this thread 18 months ago. Sigh... So this is what I've figured out/observed.
              1. Trades are NOT saved in the sdf file.
              2. The Strategy Analyzer does NOT rerun a strategy necessarily every time you click on a row in the Log pane ... and I suspect this is key to what causes the erroneous reports.
              3. Right-click>Open in New SA Tab DOES rerun the strategy... which is probably why this "workaround" mysteriously "fixes" reports that are just completely out of synch... ... but I digress
              4. I have test cases in which it is abundantly clear that the results such as shown in the pics in my post #1 actually come from running the strategy, seemingly with the same parameters... but, in fact... using a DIFFERENT parameter set. (My strategy has LOTS of parameters, but trust me... the results are extremely obvious that the results shown purport to come from one set, when they really come from a different set... I will in due course reveal all... but, the purpose of this post is to try to zero in on a test case using your beloved Sample MA Crossover ;-)
              OK... sorry for the lengthy preamble.

              My SUSPICION is that when results like pics shown in post #1 arise, the results MUST be pulled from memory, since they are NOT stored on disk AFAIK, and, as I have print statements in my strategy that confirm if/when it is run... I KNOW my strategy is NOT rerun... so... seems clear the numbers must be getting pulled from cache somewhere. Memory, or disk that is NOT the sdf...

              Next step ... and again, I am guessing... and looking for confirmation/denial... I SUSPECT that identifying the results to pull from said cache are keyed based on things like strategy name, start/end dates, and... ??? at least SOME parameters??? I am surmising that, if a match is found in the cache on effectively a SUBSET of the parameters, then that block of results is displayed... albeit possibly NOT the correct results for applying ALL parameters in the Settings pane. At least, that is what I am seeing with my strategy.

              So.. wrapping up... if you can confirm my beliefs (or correct me), I will attempt to create a test, if possible, using the old MA X standard... I say "if possible", as if my guess as to underlying mechanism is somewhere near the money, the vanilla MA X thing may NEVER actually trigger this behaviour. Anyway... I have some ideas on how to address that, if it is the case.

              Apologies for making you read a novel... but if I am right... this might result in fixing a long-standing and VERY significant bug in NT...

              Regards,
              T.

              Comment


                #8
                Hello tgn55,
                My SUSPICION is that when results like pics shown in post #1 arise, the results MUST be pulled from memory, since they are NOT stored on disk AFAIK, and, as I have print statements in my strategy that confirm if/when it is run... I KNOW my strategy is NOT rerun... so... seems clear the numbers must be getting pulled from cache somewhere. Memory, or disk that is NOT the sdf...
                I couldn't really say as we are referring to internal logic there. From your and my perspective where the results come from doesn't matter, we really just need to understand the steps that lead to that result so we can report it.
                Next step ... and again, I am guessing... and looking for confirmation/denial... I SUSPECT that identifying the results to pull from said cache are keyed based on things like strategy name, start/end dates, and... ??? at least SOME parameters??? I am surmising that, if a match is found in the cache on effectively a SUBSET of the parameters, then that block of results is displayed... albeit possibly NOT the correct results for applying ALL parameters in the Settings pane. At least, that is what I am seeing with my strategy.
                Again I couldn't really comment here as that's going to relate to internal logic which we cant view. Knowing where the results come from really does not help to find a solution or to report the problem. If you isolated some steps that reproduce what you had described that is what you should focus on or providing the most simple example of those steps. We don't need to go into actually debugging the why its happening, just how to make it happen so development can review it.

                Please let me know if I may be of additional assistance.


                JesseNinjaTrader Customer Service

                Comment


                  #9
                  Hmmm... well, I can show the results from my strategy, but you know the usual response... "show it in Sample MA Crossover" I was hoping you could provide something that would help me do just that...

                  OK... I will post a follow-up (maybe tomorrow... it's now 3:30 AM where I am ;-) with pics (maybe a video) showing what I see on my complex strategy (at least in terms of parameters)... and will TRY to replicate in Sample MA X. But I fear I might be chasing the wind, if things do indeed work the way I suspect.

                  Cheers,

                  Comment


                    #10
                    Hello tgn55,

                    Hmmm... well, I can show the results from my strategy, but you know the usual response... "show it in Sample MA Crossover" I was hoping you could provide something that would help me do just that...

                    OK... I will post a follow-up (maybe tomorrow... it's now 3:30 AM where I am ;-) with pics (maybe a video) showing what I see on my complex strategy (at least in terms of parameters)... and will TRY to replicate in Sample MA X. But I fear I might be chasing the wind, if things do indeed work the way I suspect.
                    Right, we don't have the capacity to debug custom code so using the sample or a very simple strategy is the recommendation if you are trying to explain an issue. If your custom code is having a problem that will help to set a baseline so you can better understand the problem. If the sample can show the problem then we have a bigger issue that involves the platform and we can most likely just report it. If it requires some specific set of syntax and steps then making a simple example is the quickest way forward. We would need to know what that syntax that was and how it was being used before we could provide any insight or fixes.





                    Please let me know if I may be of additional assistance.
                    JesseNinjaTrader Customer Service

                    Comment

                    Latest Posts

                    Collapse

                    Topics Statistics Last Post
                    Started by frslvr, Today, 04:29 PM
                    0 responses
                    8 views
                    0 likes
                    Last Post frslvr
                    by frslvr
                     
                    Started by wjadevries, Today, 04:18 PM
                    0 responses
                    2 views
                    0 likes
                    Last Post wjadevries  
                    Started by shahabjet1, Today, 03:47 PM
                    2 responses
                    10 views
                    0 likes
                    Last Post shahabjet1  
                    Started by JTizz, Today, 03:21 PM
                    1 response
                    6 views
                    0 likes
                    Last Post NinjaTrader_ChelseaB  
                    Started by EB Worx, Today, 02:49 PM
                    1 response
                    10 views
                    0 likes
                    Last Post NinjaTrader_ChelseaB  
                    Working...
                    X