Benchmark Testing

[GrumpyTrader]

Hi,
I am trying to create a simple template strategy (attached below) where I can benchmark my code. I thought the simplest way would be to track time between state changes, which raised a few questions:

a) the only call to OnStateChange after a backtest is the Terminated state where it has already cleaned up (including the StopWatch object). Shouldn't there be some call to OnStateChange when all the historical data has been processed (like the Transition state)?

b) To test the backtest time I created a check in the OnBarUpdate method. Is there a way to do this outside of the OnBarUpdate?

c) The documentation says the Terminated state gets called once but I see it getting called multiple times after a backtest. Can you explain what is happening?

d) How do I benchmark backtests? This approach doesn't work as the StopWatch gets reset after each iteration.

[koganam]

Hi,
I am trying to create a simple template strategy (attached below) where I can benchmark my code. I thought the simplest way would be to track time between state changes, which raised a few questions:

a) the only call to OnStateChange after a backtest is the Terminated state where it has already cleaned up (including the StopWatch object). Shouldn't there be some call to OnStateChange when all the historical data has been processed (like the Transition state)?

b) To test the backtest time I created a check in the OnBarUpdate method. Is there a way to do this outside of the OnBarUpdate?

c) The documentation says the Terminated state gets called once but I see it getting called multiple times after a backtest. Can you explain what is happening?

d) How do I benchmark backtests? This approach doesn't work as the StopWatch gets reset after each iteration.

There are multiple finely granular state changes. They are almost all documented.

ref: http://ninjatrader.com/support/helpG...n-us/state.htm

[GrumpyTrader]

hi koganam,
I've read the documentation. Most of my questions are things that are not clear or seem inconsistent in the documentation.

[NinjaTrader_ChelseaB]

Hello GrumpyTrader,

1. I disagree that the only time OnStateChange is called is when the script terminates. OnStateChange will be called several times. As linked by kognam it will change for each state listed on the help guide.
http://ninjatrader.com/support/helpG...n-us/state.htm

You can find this by Printing the State.ToString() on the first line after OnStateChange().

When the script transitions from Historical to RealTime, OnStateChange will be called and the State will be Realtime. This happens when the script is running with real time data and the historical data has finished processing. When backtesting using the Strategy Analyzer the script will not enter real time.

2. No, I am not aware of a way to detect that OnBarUpdate has been triggered outside of OnBarUpdate.

3. I'll need to inquire about why OnTermination appears more than once when backtesting. I am showing it appears only once if the script is added to a chart or to the control center and then is disabled.

4. When backtesting a basket of instruments, each backtest is completely separate. Are you speaking of Optimization? If so, each backtest in an optimization is completely separate.
If you are using a stopwatch, write the time of the test to a file or output window. Then add these up after the backtests have completed.

[GrumpyTrader]

Hi Chelsea,
Thanks for your answers. I think you have misunderstood what I wrote with regard to question a).

I'm referring to "after a backtest is completed" (i.e. all the historical data has been processed through the OnBarUpdate method), the only call, after all the historical data has been processed, to the OnStateChange is the Terminated State. In fact it calls the Terminated State multiple times (question c ). I was wondering when the Transition State gets called as I would assume that would get called at the end of a backtest.

Thanks for your help.

[koganam]

Hi Chelsea,
Thanks for your answers. I think you have misunderstood what I wrote with regard to question a).

I'm referring to "after a backtest is completed" (i.e. all the historical data has been processed through the OnBarUpdate method), the only call, after all the historical data has been processed, to the OnStateChange is the Terminated State. In fact it calls the Terminated State multiple times (question c ). I was wondering when the Transition State gets called as I would assume that would get called at the end of a backtest.

Thanks for your help.

The Transition state happens in moving from Historical to Realtime. There is never a Realtime state in Backtest, so there is nothing to which to Transition.

[NinjaTrader_ChelseaB]

Hello GrumpyTrader,

I inquired with development and was directed to an existing forum thread where the topic of the OnStateChange() with State.Terminated is appearing multiple times is discussed.

http://ninjatrader.com/support/forum...550#post411550

This is due to how properties are added to the properties grid and the strategy is prepared for backtest. Also, Matthew is able to provide a work around to trigger this once.


Also, after a backtest has processed historical data, the backtest is over and the State goes from State.Historical to State.Terminated.

[GrumpyTrader]

Thanks Chelsea