Significant results difference between replay and Sim

[Learning1]

Hi There,

I am working on testing a new strategy by using a number of replays from prior days. Thus far, that process seems to be going fairly well, however, today I had an experiece that has me wanting to probe a bit deeper into how replay and Sim work.

Here is what happenned. I ran this new strategy today in Sim. The settings included:
Enforce Immediate Fills
Delay Comms (30ms)
Delay Exchange (150ms)
Apply Commissions to P&L

After the day was over, I just re-ran the same day (today) again in replay.
I started the replay at the same time with the same settings and same contracts, etc. The end results were 1000 / contract higher than the SIM results.

What could account for this difference?
If replay is running from the same data that was recorded when runnning Sim and Enforce Immediate Fills was checked in both instances, I don't see why there would be a significant difference between the two results.

Thanks for your help in clarifying my understanding of how these two testing features run.

Gratefully,

Learning1

[NinjaTrader_Dierk]

>> Delay Comms (30ms)
>> Delay Exchange (150ms)
That is the problem. Replay has no delays.

[Learning1]

Thanks lots for your replay - That makes more sense then. So all the settings in the simulation account options apply to live SIM only and when things run in replay, they are running wiith no delays (In effect Delay Comms (0) and Delay Exchange (0). So testing in replay is fine to debug strategy rules, etc, only. To get a more realistic sense of strategy performance, live Sim is the only way. Is that the case?

[NinjaTrader_Dierk]

Right, replay always with delay=0 to have reproducible results on any speed.

[Learning1]

Thanks. I think this makes sense now, but let me confirm.

What you are saying is that if I run a day in Live SIM with Comms Delay = 0 and Exchange Delay = 0, I will get a set of results. Then if the same day is run in replay (At any replay speed) the results should closely resemble the original Live Sim results. However, if the original Live SIM is run with something other than 0 in the Delay settings, the later replay of the same period will not closely resemble the orginal Live Sim because the replay feature is programmed to run without considering the Delay settings.

Therefore, to get consistant results between Live SIM and replay, the Live SIM needs to have delays set to 0 and it doesn't matter what the delay settings are in replay because they aren't used by NinjaTrader when running a replay. Do I have this right?

[NinjaTrader_JoshP]

I believe your understanding is correct.