I was comparing NT8 Playback-MR with same strategy/data on 2 different system - and was getting different results from 2016/2017. Substantial enough such that I found a weeks worth of data missing and some other data files missing. So I sync'd them up.
I ran a full replay from 1/20/2014 to 4/21/2017 on both setups. There was a difference of $4000 in my results.
I compared both results - Laptop #2 - took 2 "bad trades" on 3/7/2014 and 3/27/2014. They were bad because they didn't follow the entry condition at all. Both values were not > the requirement. The days before replayed normal and no data was missing.
This strategy resets everyday around 4pm and 9:30 am, just in case there is missing data. I didn't save off the bad results - I figured this would be easy to reproduce - I'd be able to put in some debug statements and figure out what was going wrong.
on laptop #2:
I replayed the 2 days in question. It didn't take the trade! argh!
I replayed the whole week. It didn't take the trade! argh!
I restarted the machine and followed same steps I always do and I replayed from the start (1/20/2014), and it didn't take the trade! So much for being able to reproduce the bad trades.
I'm glad it didn't take the trade because it isn't supposed to. But HOW it took the trade the other night is a mystery. I was running no other processes such as TV that night on that laptop.
I'm currently re-running the entire replay and will compare again to the other laptop and see if the results match or differ. Maybe some other days will be "bad".
There might be some timing issue in replay that causes things to get out of sync. (I noticed the values on the day before 3/6 finished with a condition that would cause a good condition if it were used the next day, but the chart with plots loaded in fine)...
This maybe next to impossible to reproduce, especially if I switch to some test code setup.
So: trust but verify on 2 systems.
Comment