Are the back tested results accurate for all Chart Styles and Data Series Types?

[demarcog]

I am new to NT and just getting familiar with the product and spend a great deal of time back testing strategies and have a simple and for me critically important for NT support staff.

Are the the Strategy Analyzer back tested results, except for slippage, accurate for all Chart Styles and Data Series types if Order Fill Resolution = High and Calculate on Tick is specified?

If the answer is no, then for what Chart Styles and Data Series types are the back tested Strategy Analyzer test results not accurate?

The same question for candle/bar chart based strategies back testing using multiple time interval data series?

Before I invest a lot of time and potentially money relying on the Strategy Analyze and the "Optimized" results I would like to know if these results are accurate and is there any product documentation discussing this topic?

I am so hoping that unlike eSignal where the back testing results for other then bar/candlestick charts and for all chart types using multi time frame results are highly inaccurate that NT would not have the same inaccuracies in the back tester.

Thank you very much

Glen

[bltdavid]

Define 'accurate'.

Do you mean 'exact'?

Except maybe for Renko bars, where back-testing problems are well-documented,
you can't expect your order fills in back-testing to exactly match live order fills on a
live exchange for every order. (Even if they do match on a particular day, there is no
guarantee they will continue to match the next day. In fact, there is absolutely no
guarantee they will ever match, period. End of story. No guarantee, none, zilch.)

What market are you trading?
Are you trading 1 contract or 10?
Are you using Market orders for entry, or Stop, Limit, or StopLimit?
Will back-testing accurately mimic slippage on every order?
Will back-testing accurately account for other market participants on every order?

For example,
I don't think back-testing has any knowledge of queue depth. Long entry Limit orders
resting at the exchange may not fill even though it appears Market price has 'touched'
the Limit price. I mean, it's quite possible for some Limit orders to be so deep in the
queue that Buy Market orders arrive and the Market price moves higher, away from
the Limit price, leaving some orders at that Limit price unfilled. If price never falls back
to that Limit price and never fills those remaining orders, well ... such is life, right?

But, later, when you back-test using Strategy Analyzer, it may report a filled entry order
at that Limit price, because price touched the Limit price, but never traded through it,
but it still touched it, so it records a fill at that price.

Or, consider the exact opposite scenario, where your order in the order queue is filled
in the live market, but in Strategy Analyzer because the Limit price was only touched
and not traded through, the back-tester fails to record a filled order.

[I'm not sure which algorithm the back-tester uses for fills. I'm just saying there are
lots of edge cases and that it can be very hard for back-testing results to be exactly
equivalent to Real-time data results on a live account or Sim101 account. Besides,
there is never a guarantee of exactness -- even if they do happen to be exact.]

If you want to understand the 'accuracy' and/or 'exactness' of the Strategy Analyzer,
I recommend you compare live Sim101 results to the corresponding back-testing
results for the same period, which means you first need to have the live Sim101
results as a baseline. Given enough days to compare, you can decide for yourself
just how well your chosen strategy and chosen bar type(s) perform in back-testing.
(Market Replay can also help give you an initial baseline of results.)

IMHO, perhaps accounting for few percentage points difference due to slippage and
unforeseen edge cases, it's going to depend heavily upon your specific strategy.

Therefore, the short answer to the question in your title is: No.
But, perhaps, if you redefine 'accurate' to be 'x% less than 100% exact', you may
find back-test results satisfactory for certain input bar series -- but this is still highly
dependent on your strategy. You just need to find what your 'x' is.

Just my 2˘.

Good reading here and here.

[Teebone21]

no the backest results for anything except minute bars is wrong. You will need to forward test your strategy to see if its viable when using certain

[NinjaTrader_ChelseaB]

Hello Glen,

There are differences on how historical data is processed.

For more accurate results, I recommend 1 tick granularity and TickReplay. Below is a link to a forum post that details.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

That said, some bar types change their values in real-time which prevents an accurate backtest as bltdavid has described for renko bars.

[demarcog]

no the backest results for anything except minute bars is wrong. You will need to forward test your strategy to see if its viable when using certain

Thank you very much Teebone21 I was hoping this was not the case but appreciate you confirming what I suspected.

Can you suggest the best was to forward test the strategy and excuse the dumb question but I'm new to NT. I

I must be missing something on how to forward test in real time. The last product I used I setup a Demo account and let it run for whatever period of time. Their equivalent of the Chart Trader would track the PnL in the Chart Trader's PnL box for however many trades, days, or indefinite period the strategy was running.

I would look at the back test report and compare that to the executions in real time for comparasion purposes. I'm obviously doing something wrong as I load and run the automated strategy in SIM101 account and the PnL get's reset when a trade is closed out, and it looks like at EOD the balances get reset to zero for realized and unrealized.

I obviously don't know what I'm doing and appreciate the help, from everyone who has been kind enough to assist this basket case.

Thanks again,

glen

[demarcog]

Hi Bltdavid,

Thank you very much for that insightful response. You obviously have tremendous expertise on order processing among may other areas I'm certain and I learned quite a bit reading through the reply. If that is your 2 cents answer I can only imagine the 10 cent response.

Let me try and clarify some of the issues I was trying to raise and answer a few of the question you posed.

By "accurate" and I should have made that clearer, apologies. I meant excluding anything that could even remotely be accounted for by "slippage". You raise a good point as it very much depends on what market and what size. I was referring to the ES front month contract and trading a 1 lot. The difference I was asking about goes well beyond any of the comprehensive order queue conditions you raised. I was referring to discrepancies that go well beyond 1% or 2%, winning strategies 75% Win Pct that drops to 35%.

I'm a bit of a fanatic regarding back testing, been doing it for decades I'm embarrassed to admit. I have tested every conceivable idea published in any trading book or magazine where I could get the code or wrote it myself and shocked how difficult it is to find anything worth trading and the simpler the better.

You mentioned that the issue of accuracy was well documented for Renko, can you please point out some of that documentation. I did come across some documentation on Point and Figure and back testing inaccuracies and curious about the other Chart types, Unirenko, Point Break, Range bars.

I used eSignal for years and recently opened an account with NT. I was blown away with how fantastic the product is and very optimistic that with the capabilities of NT unlike other products I've used in the past that perhaps not a Holy Grail system can be found but one where a reasonable living can be made in the markets.

I know traders aren't "supposed" to help each other as there is the element of a zero sum game obviously. I was hoping however that the NT community could be a place to collaborate on developing a profitable strategy/methodology without giving away the "secret sauce". It would be interesting to share a strategy with a certain degree of profitability (or not) and as a group see if we could improve upon the profitability.

I've only had NT for a week or so but did download every strategy available and going through the back testing now. Minute charts were not necessarily my first choice for a Data Series which is why before I spend too much time back testing 100 strategies wanted to know if there were back testing accuracy issues on other chart type and very much appreciate you help.

Thanks Again,

[email protected]

[NinjaTrader_ChelseaB]

Hello Glen,

Adding 1 tick intra-granularity can improve backtest order fill accuracy for bar types that do not use IsRemoveLastBar support.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

With ChartTrader do you have 'Show gross realized PnL when flat' checked and enabled?

https://ninjatrader.com/support/helpGuides/nt8/index.html?properties3.htm#AvailablePropertiesAndDefinitions

[demarcog]

HI Chelsea,

I am using the 1 tick granularity and Order Fill Resolution = High.

I will run with Show Gross realized PnL checked but not clear how to enable it beyond checking the box?

Thank you for your help, it is very much appreciated.

glen

[NinjaTrader_ChelseaB]

Hello Glen,

When 'Show gross realized PnL when flat'' the pnl will show for the selected account, as shown on the Accounts tab of the control center when the position is flat.

[demarcog]

Hi Chelsea,

I misunderstood when I Check the box Show Gross realized PnL I thought that would be for that particular symbol. I can see the PnL for the account but interested in the PnL for that symbol?

thank you.

glen

[NinjaTrader_ChelseaB]

Hello Glen,

Unfortunately, this shows the account pnl and not specific to the symbol, as this is the value being supplied from the connected broker or from the Sim101 account.

You might consider creating a custom coded indicator that loops through the executions (<execution>.Order.Instrument) on an account for a specific instrument and adds up the pnl.
https://ninjatrader.com/support/help...executions.htm

Below are a few scripts that may give you some ideas.
https://ninjatrader.com/support/foru...ples#post93881
https://ninjatraderecosystem.com/use...lay-indicator/

[demarcog]

Hi Chelsea,

Thank you for the suggestion and example links for a coding solution.

Aside from writing code once I have a strategy that back test's well and please excuse my NT ignorance as it is a very rudimentary general question.

What is the next step in monitoring the strategy real time, it i suggested to only run one strategy on one symbol at at a time and dedicate one Sim account to that configuration?

Normally I would load a strategy into a chart and let it run real time.

I would see the strategies executing trades real time where the chart/strategy would have a one to one relationship and at anytime I could go into the Account Manager which would log the trades executed on the screen in the Account Manager which would match exactly with the buys/sells on the screen.

In addition I could reload the strategy loaded on the chart and open the back test report at any time and compare the back test reported trades with the real time. The other difference was the Chart Trader as we mentioned would track the PnL for the specific symbol. That is not a big issue as the Account Manager would also keep track on a symbol/strategy basis.

This is the functionality I am looking to replicate on NT and sure the answer is right in front of my face but I am looking at the forest and not the trees?

thank you

glen

[NinjaTrader_ChelseaB]

Hello Glen,

Below is a link to the Strategy Builder 301 video which demonstrates how to run a NinjaScript Strategy in real-time.

https://www.youtube.com/watch?v=HCyt90GAs9k&feature=youtu.be&t=1h25s

As well as videos on how to backtest and optimize.

https://youtu.be/RaAWxf2jxp8

https://youtu.be/kjnyYlzzog8

https://youtu.be/SxbGhhkKtXk

As far as monitoring in real-time, what do you want to monitor?

The Strategies tab of the Control Center will show the position and pnl of the strategy.

https://ninjatrader.com/support/helpGuides/nt8/index.html?strategies_tab.htm#UnderstandingTheStrategiesTab

Orders and trades can be viewed in the Strategy Performance window.

https://ninjatrader.com/support/helpGuides/nt8/index.html?strategies_tab.htm#HowToViewStrategyPerformance

https://ninjatrader.com/support/helpGuides/nt8/trade_performance.htm

I recommend only running one strategy per instrument and account combination, but this is up to you.

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/99748-two-independent-strategies-on-the-same-futures-contract?p=792660#post792660

Last, below is a link to a forum post with helpful information about getting started with NinjaScript and C#.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/indicator-development/97575-sample-scripts?p=786040#post786040

[demarcog]

Chelsea,

Thank you for the response that is exactly what I was looking for.

I appreciate your patience and assistance in getting me started working with strategies.

Glen.