backtest replication

[nikkk]

I back tested strategy with Matlab. The result is happy one. I did implement it in (free) NT. The result is very close to one I had in MatLab which makes me confident of analysis.

However, the test with real-time feed shows Veeeery different result... I could agree about different execution price, but not with large time delay. NT calculates triggers at different bars.

I saw disclaimer of NT about possible backtest-execution difference, which makes me uneasy since I am used to such difference of <1-3%, which is ONLY due to price, but almost never due to timing. With NT this difference is huge (can be few times).

Could you share your experience and tricks?

I suspect something has to do with updateOnBar option - end of bar or continuous. What is the difference? How update is done? I tried both with almost equaly bad result.

Thanks
Nik

[NinjaTrader_Bertrand]

Hi Nik, welcome to our support forums! Which market are you using? There are two settings, one updates the calculations on every tick (CalculateOnBarClose = false), the other at the end of the bar (set the previous to true) - http://www.ninjatrader-support.com/H...BarClose1.html

For finer backtesting you might want to check into this concept - http://www.ninjatrader-support2.com/...ead.php?t=6652

[nikkk]

I use Eironext (cash market) + IB as data-feed and execution platform.

I think Euronext (or anything else) is irrelevant as the test is done with free version where only "fake" trades are booked within NT. Also I said initially that with my other strategy which is "tick-by-tick" I managed to replicate my end-of-day real PnL with backtest to satisfactory level.

That system sends Market orders immediately after the trigger occurs. In case of NT I expect that orders will be issued immediately after trigger at the end-of-bar. The problem is that these triggers do not coincide in case of real-time and backtest.

Another, related question - how do you store indicator bar series when indicator values are updated tick-by-tick?

I am considering two possibilities for CalculateOnBarClose = false:
- IndicatorSeries[0] is updated within the current bar, while [1],[2]... are fixed because past bars do not change anymore. Once current bar closes IndicatorSeries shifts;
- when IndicatorSeries[0] is updated whole series shifts, therefore [1] value relates to the previous tick, not bar.

which one is used?

when CalculateOnBarClose = true, IndicatorSeries[0][1][2]... must contain info related to bars and must be identical to my backtest results (I leave out the execution process).

[NinjaTrader_Bertrand]

With CalculateOnBarClose set to true the orders will be send after the bar is closed, that means on the open tick of the next bar.

This option will only affect accessing the last bar on the chart in realtime, with this set to false this bar and the calculations in the OnBarUpdate() will update on every incoming tick. If the bar closes, the values become 'permanent' and the cycle starts all over for the next bar to be completed.

[nikkk]

Seems I have to debug more. Therefore two more questions:
1. can I output Print() into Log file?
2. do you have flag which tells me that the Bar starts (or ends) during real-time?

Regards,
Nik

[NinjaTrader_Bertrand]

Hi, no but you can copy and paste with the normal windows commands from the output window. For realtime order debugging setting TraceOrders to true in the Initialize() will be helpful, for the flag you can work with FirstTickOfBar - http://www.ninjatrader-support.com/H...TickOfBar.html

[nikkk]

is there any way to see the Caller? I am interested in the flag which tells me if my strategy is called either from backtest (optimizer) or from real-time stream.

thanks
Nik

[NinjaTrader_Bertrand]

Hi Nik, you could work with Historical for this - http://www.ninjatrader-support.com/H...istorical.html

[nikkk]

I stored today quotes and replay those through "Market Replay connection".

Strange thing I observe is that if I start replay at let say 8.50, indicators values will get different from if I would start replay at 9.03... How to get my calculations replicated...

What is the size of buffer in outout window?

I still believe that having file-output is crucial for automated post-trade analysis and cost control. Manual copying is rudimentary...

Also I noticed that if I Print detailed output it loads memory and slows computer. This problem would be solved with log-file. Dont understand why users cannot have it. It's trivial.

[NinjaTrader_JoshP]

If you want to output to file you need to do it in your code. http://www.ninjatrader-support2.com/...ead.php?t=3475

I do not know the exact size of the Output Window buffer.

Please try reloading your NinjaScript as you load it up at different times in your replay. They should generate the same values.

[nikkk]

Thanks a lot for all your useful replies. Still a question I have.

Now, for two conseq. days my strategy is run during a day with life-feed (A). I get result based on CalculateOnBarClose=false.

When I replay the same day, I get different result (B). I do not apply "bad tick" filter, which means I take-in everything (right?).

Next to this I replay same strategy with CalculateOnBarClose=true (C). In addition, I run backtest (D).

All four numbers (A,B,C,D) are different and for two days I cannot say they even close:

(each row reps 2 days)
A = 1,360 1640
B = 170 640

C = (87) 320
D = (140) (220)


I would accept the difference of <100.

The algorithm is briefly the following:

if (prefilter
&& sessionTime()
&& BarsSinceExit() !=0) {

if (buy-condition && Position!=Long) {

EnterLong()

}
if (sell-condition && Position!=Short) {

EnterShort()

}

}

if(BarsSinceEntry()>=1) {

if(stop-signal && position==Long) ExitLong()
if(stop-signal && position==Short) ExitShort()

}

if (closeTime()) {

if(position==Long) ExitLong()
if(position==Short) ExitShort()

}

[NinjaTrader_JoshP]

What kind of chart are you running? Tick? Minute?

[nikkk]

the chart is 1 minute. the strategy is designed to run on bar data.
i presume, in the replay mode the result should be equivalent to my real-life run.

[NinjaTrader_JoshP]

Not necessarily. Please go to Tools -> Options -> Simulator tab and ensure you don't have any delays set. These delays do influence your fills drastically from accelerated replay attempts.

[ctrlbrk]

Not necessarily. Please go to Tools -> Options -> Simulator tab and ensure you don't have any delays set. These delays do influence your fills drastically from accelerated replay attempts.

Not according to 6.5.1009 help file.



Mike