Strategy Analyzer Summary Report... more errors in results (Time in market)

[tgn55]

See the pic for the story... ran a walk-forward run on a 60 minute chart, the Summary tells me Average Bars was about 61 bars... but then tells me that Average Time in market was... 0.05 minutes !!

Please note that my strategy is ALWAYS in the market, either long or short, but never flat (after initial entry)... so this is clearly bogus. My average time in market should therefor be about 60x61 minutes, right? Apart from which 3 seconds on a 60 minute chart is in any case nonsense...

Not the most serious issue... but still very wrong reporting.



I have not yet been able to replicate on Sample SMA Crossover... but I have over the last couple months seen this on a semi-regular basis. Ignored it until now, as it is not critical, but still would be reassuring to have it fixed.

Cheers,
T.

[tgn55]

So... figured I'd do some deeper digging on this one, not because time in market is critical, but I've suspected for a while that something is amiss when pulling data from previous optimisation/walk-forward runs. I think I can now prove that is exactly the case here...

I located the XML file for the SA optimisation in question... and look what I discovered...

You can see I'm looking at the correct record by comparing some of the other values on screen. All good - EXCEPT for time in market.

I think this clearly demonstrates that the values reported in the SA Summary, at least when pulling data from prior runs, is getting it wrong.
I also checked that if I look at results immediately after running the walk-forward - they are fine. Run another optimisation, then pull up the now-historical run from 5 minutes ago... and it is now wrong.

My conclusion: there is a bug in the code that pulls data from the strategyanalyzerlogs XML files (assuming that is where it pulls it from ?). Now, I wonder if this explains some of the other bizarre results I've reported recently...??

Enjoy,
T.

[tgn55]

Last update for today/tonight... turns out there IS a connection between the wrong time in market data and the crazy Profit per Month I reported in this post

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1069244-strategy-analyser-profit-per-month-inconsistency

Seems that very wrong (short) time in market numbers often (maybe always ??) coincide with inflated Profit per Month.

It's real late 3:30 AM here, more research needed... but perhaps tomorrow!

[NinjaTrader_ChelseaB]

Hello tgn55,

May we see the execution times to show that the positions are being taken for longer than this time?

[tgn55]

I can do better than that... recorded a video showing crazy behaviour on Sample SMA crossover, standard backtest, on Daily bars can't get much simpler than that. And yes I show trade times.
I intend to record more, on my strategy, showing weird stuff happening, but this vid shows there is something rather broken in NT's calculations &/or data extraction from saved XML files.

https://screencast-o-matic.com/watch/cqQnfottVR

Whilst this example may seem rather innocent, in my other post I show how it can create absolutely misleading (actually, crazy nonsense) results, in statistics that really matter.
I'm going to do some more work on this today, but this is surely a major concern... I hope it can be resolved in the next release.

PS Let me know if you have issues getting the video... I have not used this service before... hope it works for you.

Trevor.

[tgn55]

Hey Chelsea... just figured out (in part... but a significant part) the root cause of the weird time in market values... see my other post.
That said, I still think there is a problem with pulling data from historical XMLs... it does seem a bit unreliable.

Cheers,
T.

[NinjaTrader_ChelseaB]

Hello tgn55,

The executions show the time of the executions which would be is necessary to report an issue. Let me know if you would like to continue troubleshooting this.

To confirm, this behavior cannot be reproduced when first running a backtest and can only be reproduced when loading a Strategy Analyzer Log?

[tgn55]

Chelsea... please read this thread... https://ninjatrader.com/support/foru...-inconsistency

I've analysed it extensively, and found the root cause, and made a suggestion that would effectively fix the crazy reporting... and as a side benefit, make the PPM metric useful on walk-forwards again!

Yes, the issue is triggered by trade/execution right near (or at) the end of walk-forward test periods. Everything flows from there, given the time in trade is then calculated as maybe a few seconds... resulting in absurd Profit per Month. So it is an artifact of walk-forward tests... not as I suspected, XML errors.

I think the issue is fixable... but something needs to change in the way NT calculates/reports walk-forward results. On the upside for NinjaTrader, it would eliminate the nonsensical/farcical examples I show in the other post.

I'm kind of in the middle of strategy refinement/backtesting just now... I'll pull out some trade/execution dta files and send through before my night/your day is over.

Regards,

[tgn55]

OK, here's the execution data. Please note it says "Standard" backtest type... as I used the workaround you mentioned in the other thread... the only way I can actually get to the walk-forward historical trades was to open in a new Tab... Woohoo! I have a workaround... and I think this little snippet of info might be valuable for the developers to trace where/when things go wrong. A win-win ;-)

Cheers,
T.

[NinjaTrader_ChelseaB]

Hello tgn55,

The entry time is 6:30:00 AM and the exit time is 6:30:00 AM on the same day and the avg time in market is 0.00.

What are you finding is incorrect about this?

[tgn55]

Chelsea, you're missing the point... I know this results in (in this case) zero time in market. Some other instances it reports like 0.05 minutes... it's all in the images I've attached.
But the significant issue is that this then translates to crazy Profit per Month values... in this case a huge positive result. In other cases, it creates large negative values... sufficient to produce the bizarre outcome that a Walk-forward SUMMARY line can show overall $18,000 profit... and like $2,000 per month LOSS. And that is just weird.

I now know what causes it... but I believe you guys would do well to fix it... it's not really a good look for the platform !!

[NinjaTrader_ChelseaB]

Hello tgn55,

Unfortunately, I don't see any executions where I can calculate the time and compare with the summary.

I can take your word that you have reviewed these and that they are incorrect, but I don't have any evidence to provide in a report.

I did not see a confirmation that this only occurs with Strategy Analyzer logs (which we already know has an issue) so I am assuming this is occurring the moment you click Run without viewing a log.
I am not able to reproduce this behavior. On my end with a simple test of the SampleMACrossOver over very few days with a walk forward, the time in market matches the calculated time from the entry and exit executions times.

[tgn55]

Chelsea I certainly have reviewed these, it is not only in Strategy Analyzer logs, all that stuff is I think unrelated to this. I'll try to generate some evidence for you, then I think we can close out this thread re time in market... I understand what causes what I initially thought were impossible values.

What remains an open issue is the subsequent reporting of bogus Profit per Month... but probably best we continue that discussion under the other topic.

It sure would help if we could just use TeamViewer... would probably save a bunch of time and avoid misunderstanding between us.

Cheers,

[NinjaTrader_ChelseaB]

Hello tgn55,

To request a scheduled call you would need to write to platforumsupport [at] ninjatrader [dot] com.

I would basically get the same information I'm requesting, then get off the call and try and reproduce and calculate the values, then reduce to as little trades, data, and code possible while still being able to reproduce.
I really just need a series of steps to reproduce, and the numbers that are wrong.