IsFirstTickOfBar vs OnBarClose for backtest & live

[Leafcutter]

Hello Leafcutter,

*mic drop*

Thank you so much for the kind words.

my pleasure...

[NinjaTrader_ChelseaB]

Hello Leafcutter,

*mic drop*

Thank you so much for the kind words.

[Leafcutter]

Hello fxRichard,

Live orders are filled on an exchange with a trading partner on an agreed upon price based on market dynamics. Backtest orders are not using these market dynamics. Instead these are filled based on logical rules from processing historical data.

Please review the help guide document on the differences on real-time vs backtest (historical).
https://ninjatrader.com/support/help...ime_vs_bac.htm

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Below is a link to the help guide on Calculate.

https://ninjatrader.com/support/helpGuides/nt8/calculate.htm

Adding intra-bar granularity can help with this.

Intra-bar granularity adds a second data series such as a 1 tick series using AddDataSeries() so that the strategy or indicator has the individual ticks in the historical data in between the High and Low of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with. The orders placed will need to use the BarsInProgress (BarsArray) index of that finer resolution series.
-Adding a 1 tick series with AddDataSeries() does not enable IsFirstTickOfBar to work correctly in historical data. (IsFirstTickOfBar will always be true)

Below is a link to an official reference sample that demonstrates how to add intra-bar granularity.
'Backtesting NinjaScript Strategies with an intrabar granularity' - https://ninjatrader.com/support/help...ipt_strate.htm

And a link to a video demonstrating implementing 1-tick intra-bar granularity.
https://drive.google.com/file/d/12F6...usp=share_link

Also, here is a link to the differences on real-time vs backtest (historical).

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

As well as a link to the help guide on the AddDataSeries() method.

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

A link to the help guide on BarsInProgress.

https://ninjatrader.com/support/helpGuides/nt8/barsinprogress.htm

And a link to the help guide on Multi-Time Frame & Instruments. Please see the section 'How Bar Data is Referenced', and 'Accessing the Price Data in a Multi-Bars NinjaScript'.

https://ninjatrader.com/support/helpGuides/nt8/multi-time_frame__instruments.htm

The Fill limit orders on touch option will also change whether a limit order requires a tick through to fill or if it fills on touch, which can change the price at which a limit order fills and changes the end performance.

https://ninjatrader.com/support/helpGuides/nt8/backtest_a_strategy.htm

https://ninjatrader.com/support/helpGuides/nt8/isfilllimitontouch.htm

Also, below I'm including a link to the help guide on Historical Order Backfill Logic.

https://ninjatrader.com/support/helpGuides/nt8/historical_order_backfill_logic.htm

https://ninjatrader.com/support/helpGuides/nt8/index.html?understanding_historical_fill_.htm#UnderstandingTheHistoricalFillAlgorithm

In NinjaTrader 8, there have been two new enhancements so that programmers may not have to manually add this secondary series and code the script for high accuracy fills (Order Fill Resolution) or for intra-bar actions (TickReplay) depending on the needs of the script.

The first is TickReplay.
TickReplay is used for the intra-bar actions. This adds a 1 tick series to the script to be used for triggering OnBarUpdate when Calculate is set to .OnEachTick or .OnPriceChange. This also allows OnMarketData to trigger historically for last market events only (the bid and ask will not trigger).
-TickReplay will allow for indicators to update for each tick or price change and for scripts to trigger actions intra-bar.
-TickReplay must be enabled for the Calculate setting in State.SetDefaults to take effect in the Strategy Analyzer.
-TickReplay allows for IsFirstTickOfBar to work correctly in historical data and will be true for the first tick of the bar only.
-TickReplay does not provide fill prices for accurate order fills.

Below are links to the help guide on TickReplay.
TickReplay - https://ninjatrader.com/support/help...ick_replay.htm
Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm
OnMarketData - https://ninjatrader.com/support/help...marketdata.htm
IsFirstTickOfBar - https://ninjatrader.com/support/help...ttickofbar.htm
Calculate - https://ninjatrader.com/support/help.../calculate.htm
OnBarUpdate - https://ninjatrader.com/support/help...nbarupdate.htm


The second is Order Fill Resolution.
This has two settings, Standard and High. When set to standard, the behavior of order fills will be accurate to the close of the bar without any intra-bar information. When set to High, this allow you to select a secondary data series to be used to provide intra-bar pricing information solely for the purpose of order fill accuracy. The highest you could set this to is 1 tick.
-Order fill resolution does not provide intra-bar granularity to OnBarUpdate for actions to be triggered intra-bar.
-Order fill resolution does not enable IsFirstTickOfBar to work correctly in historical data.
-Order fill resolution does not work with multi-series scripts using AddDataSeries(). If using AddDataSeries(), manually implement 1 tick granularity.

Below is a link to the help guide on Order Fill Resolution. (Above this you can also read about the Historical Fill Algorithm)

https://ninjatrader.com/support/helpGuides/nt8/index.html?understanding_historical_fill_.htm#UnderstandingOrderFillResolution

Currently, at this time, there is a known issue (tracked with ID# NTEIGHT-10782) where the High Fill Order Resolution and TickReplay cannot be used together. These work great when used on their own, however, when used together a complication arises that prevents the script from processing. Our developers are working hard to have this corrected in an upcoming build of NinjaTrader 8.

In the meantime, if it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself with AddDataSeries() for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically and/or have IsFirstTickOfBar work correctly in historical data.


If you would like to further understand what specifically is different in the data that is causing differences, see the post below.
https://ninjatrader.com/support/foru...nce#post100192


Note, bar types that are IsRemoveLastBarSupported cannot be used with TickReplay and generally cause inaccurate results when backtesting in historical data.

From the help guide:
"Note: The system bar types "Line Break" and "Renko" cannot be used with Tick Replay and as a result, the Tick Replay option will be disabled when configured with those bar types. There may be other 3rd party bar types which may also disable Tick Replay by design. If you are a developer, please see the property IsRemoveLastBarSupported for more information"

https://ninjatrader.com/support/helpGuides/nt8/tick_replay.htm

Discussion threads.
https://ninjatrader.com/support/foru...52#post1100852
https://ninjatrader.com/support/foru...636#post839636


It is not advised or necessary to change the Calculate mode during run-time.

Hello Mr. Chelsea B.,

THIS VIDEO about how to implement the 1-tick intra-bar granularity into a strategy has made my day! As I have never learned coding I have to teach myself the hard way and I´ve been searching for this possibility since a long time. I knew it should be simple, but I just couldn't find anything understandable on the topic. Thank you sooooo much!!! By the way, I've read your name - which sounds more like the name of a rapper than a programmer :-) - here 1000 times and your posts have often helped me. You're doing a great job!

[NinjaTrader_Jesse]

Hello rocker84,

If you were using the logic in the intrabar backtest script that would not account for intrabar conditions so you would end up with a lot of fills in a single bar. If your overall plan is to use OnEachTick in realtime or test with TickReplay and OnEachTick you would likely want to add additional conditions to make sure some amount of bars since the entry happen before allowing a reversal. You can use BarsSinceEntryExecution to mitigate additional entries.

https://ninjatrader.com/support/helpGuides/nt8/barssinceentryexecution.htm?zoom_highlightsub=BarsSinceEntryExecution

[rocker84]

Hello rocker84,

In the linked sample the purpose of the BarsInProgress == 0 is to show two concepts. The first concept would be that in a backtest you select the primary series that you want your strategy to operate, the BarsInProgress == 0 makes sure that the cross conditions happen for the primary series. The second concept is how the order methods specify BarsInProgress 1 so the fill happens on the more granular secondary series.

In your script you would only need BarsInProgress == 0 if you have other series being added. If you have a single series script you don't need the BarsInProgress == 0 condition. If you do have multiple series you would want to make a BarsInProgress condition for the index of the series that you want the logic executed.

Perfect, thank you very much. In the case of the video, where you expect an intrabar mean crossover to enter with series "1" (the smallest, using EnterShort(1, ). What would happen if the averages inside the bar cross over, cross below, several times in the same candle? Multiple orders or does the method ignore the next ones?

[NinjaTrader_Jesse]

Hello rocker84,

In the linked sample the purpose of the BarsInProgress == 0 is to show two concepts. The first concept would be that in a backtest you select the primary series that you want your strategy to operate, the BarsInProgress == 0 makes sure that the cross conditions happen for the primary series. The second concept is how the order methods specify BarsInProgress 1 so the fill happens on the more granular secondary series.

In your script you would only need BarsInProgress == 0 if you have other series being added. If you have a single series script you don't need the BarsInProgress == 0 condition. If you do have multiple series you would want to make a BarsInProgress condition for the index of the series that you want the logic executed.

[rocker84]

Hello fxRichard,

Live orders are filled on an exchange with a trading partner on an agreed upon price based on market dynamics. Backtest orders are not using these market dynamics. Instead these are filled based on logical rules from processing historical data.

Please review the help guide document on the differences on real-time vs backtest (historical).
https://ninjatrader.com/support/help...ime_vs_bac.htm

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Below is a link to the help guide on Calculate.
https://ninjatrader.com/support/help.../calculate.htm


Adding intra-bar granularity can help with this.

Intra-bar granularity adds a second data series such as a 1 tick series using AddDataSeries() so that the strategy or indicator has the individual ticks in the historical data in between the High and Low of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with. The orders placed will need to use the BarsInProgress (BarsArray) index of that finer resolution series.
-Adding a 1 tick series with AddDataSeries() does not enable IsFirstTickOfBar to work correctly in historical data. (IsFirstTickOfBar will always be true)

Below is a link to an official reference sample that demonstrates how to add intra-bar granularity.
'Backtesting NinjaScript Strategies with an intrabar granularity' - https://ninjatrader.com/support/help...ipt_strate.htm

And a link to a video demonstrating implementing 1-tick intra-bar granularity.
https://drive.google.com/file/d/12F6...usp=share_link

Also, here is a link to the differences on real-time vs backtest (historical).
https://ninjatrader.com/support/help...ime_vs_bac.htm

As well as a link to the help guide on the AddDataSeries() method.
https://ninjatrader.com/support/help...dataseries.htm

A link to the help guide on BarsInProgress.
https://ninjatrader.com/support/help...inprogress.htm

And a link to the help guide on Multi-Time Frame & Instruments. Please see the section 'How Bar Data is Referenced', and 'Accessing the Price Data in a Multi-Bars NinjaScript'.
https://ninjatrader.com/support/help...nstruments.htm

The Fill limit orders on touch option will also change whether a limit order requires a tick through to fill or if it fills on touch, which can change the price at which a limit order fills and changes the end performance.
https://ninjatrader.com/support/help...a_strategy.htm
https://ninjatrader.com/support/help...mitontouch.htm

Also, below I'm including a link to the help guide on Historical Order Backfill Logic.
https://ninjatrader.com/support/help...fill_logic.htm
https://ninjatrader.com/support/help...lFillAlgorithm


In NinjaTrader 8, there have been two new enhancements so that programmers may not have to manually add this secondary series and code the script for high accuracy fills (Order Fill Resolution) or for intra-bar actions (TickReplay) depending on the needs of the script.

The first is TickReplay.
TickReplay is used for the intra-bar actions. This adds a 1 tick series to the script to be used for triggering OnBarUpdate when Calculate is set to .OnEachTick or .OnPriceChange. This also allows OnMarketData to trigger historically for last market events only (the bid and ask will not trigger).
-TickReplay will allow for indicators to update for each tick or price change and for scripts to trigger actions intra-bar.
-TickReplay must be enabled for the Calculate setting in State.SetDefaults to take effect in the Strategy Analyzer.
-TickReplay allows for IsFirstTickOfBar to work correctly in historical data and will be true for the first tick of the bar only.
-TickReplay does not provide fill prices for accurate order fills.

Below are links to the help guide on TickReplay.
TickReplay - https://ninjatrader.com/support/help...ick_replay.htm
Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm
OnMarketData - https://ninjatrader.com/support/help...marketdata.htm
IsFirstTickOfBar - https://ninjatrader.com/support/help...ttickofbar.htm
Calculate - https://ninjatrader.com/support/help.../calculate.htm
OnBarUpdate - https://ninjatrader.com/support/help...nbarupdate.htm


The second is Order Fill Resolution.
This has two settings, Standard and High. When set to standard, the behavior of order fills will be accurate to the close of the bar without any intra-bar information. When set to High, this allow you to select a secondary data series to be used to provide intra-bar pricing information solely for the purpose of order fill accuracy. The highest you could set this to is 1 tick.
-Order fill resolution does not provide intra-bar granularity to OnBarUpdate for actions to be triggered intra-bar.
-Order fill resolution does not enable IsFirstTickOfBar to work correctly in historical data.
-Order fill resolution does not work with multi-series scripts using AddDataSeries(). If using AddDataSeries(), manually implement 1 tick granularity.

Below is a link to the help guide on Order Fill Resolution. (Above this you can also read about the Historical Fill Algorithm)
https://ninjatrader.com/support/help...FillResolution


Currently, at this time, there is a known issue (tracked with ID# NTEIGHT-10782) where the High Fill Order Resolution and TickReplay cannot be used together. These work great when used on their own, however, when used together a complication arises that prevents the script from processing. Our developers are working hard to have this corrected in an upcoming build of NinjaTrader 8.

In the meantime, if it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself with AddDataSeries() for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically and/or have IsFirstTickOfBar work correctly in historical data.


If you would like to further understand what specifically is different in the data that is causing differences, see the post below.
https://ninjatrader.com/support/foru...nce#post100192


Note, bar types that are IsRemoveLastBarSupported cannot be used with TickReplay and generally cause inaccurate results when backtesting in historical data.

From the help guide:
"Note: The system bar types "Line Break" and "Renko" cannot be used with Tick Replay and as a result, the Tick Replay option will be disabled when configured with those bar types. There may be other 3rd party bar types which may also disable Tick Replay by design. If you are a developer, please see the property IsRemoveLastBarSupported for more information"
https://ninjatrader.com/support/help...ick_replay.htm

Discussion threads.
https://ninjatrader.com/support/foru...52#post1100852
https://ninjatrader.com/support/foru...636#post839636


It is not advised or necessary to change the Calculate mode during run-time.

Hello Chelsea,

I would like to ask you a small detail, I am trying to obtain in the backtest the most similar results possible to reality, using an indicator with TickReplay (footprint type).

In this post in the strategy you don't use "if (BarsInProgress == 0)". However in this other post yes.

https://forum.ninjatrader.com/forum/ninjascript-educational-resources/reference-samples/6852-strategy-backtesting-ninjascript-strategies-with-an-intrabar-granularity

In the logic of my strategy I am waiting for it to calculate at the close of the bar, but with TickReplay update. In this post you are waiting for an intrabar average cross. So, in my case I should use "if (BarsInProgress == 0)", right?

[NinjaTrader_ChelseaB]

Hello garth,

Below is a link to the help guide on the historical data available for each connection type.

https://ninjatrader.com/support/helpGuides/nt8/data_by_provider.htm

If you are connected to NinjaTrader Brokerage, with the NinjaTrader connection type, the historical tick data is available.

https://support.ninjatrader.com/s/article/Connecting-to-Your-Account-NinjaTrader-Desktop

[garth]

I've hit this forum post with the same issue: the sample intrabar backtest runs but doesn't actually trigger any trades. Specifically, it appears that EnterLong(1, ...) just doesn't ever send an order to the second data series (IE: tick data). I see some mention of Continuum for tick level historical data, but it isn't clear if that is a 3rd party subscription, how to sign up for that, etc. Can someone who has gotten this working advise on the specific steps required to add a data feed/subscription that support historical data for tick level backtesting?

[J2512]

Hi Chelsea

Thanks a lot for your response, finally I did the test sucessfully.

Regards.

[NinjaTrader_ChelseaB]

Hello J2512,

I've given this a test and was not able to reproduce.

Below is a link to a video of the test.

https://drive.google.com/file/d/1Dq2GhSzdRZZvkRnqRnHgXimQoSdjFyYg/view?usp=drivesdk

Be sure to re-download the script (Ctrl + F5 to refresh the cache) to ensure you have the latest version.

To confirm, you are connected to a data feed that provides historical tick data for the instrument you are viewing?
(Can you open a 60 second chart with the same date range and see data on the chart for the full date range?)

If you follow the exact steps shown in the video you are experiencing different behavior?

[J2512]

Hi @NinjaTrader_ChelseaB

Just now I have test the sample from https://ninjatrader.com/support/help...ipt_strate.htm but does not work, could you tell me if there is a new version or what is happening? the message is: Error on calling 'OnBarUpdate' method on bar 49: You are accessing an index with a value that is invalid since it is out-of-range. I.E. accessing a series [barsAgo] with a value of 5 when there are only 4 bars on the chart.

Regards.

[NinjaTrader_ChelseaB]

Hello rafael_delima86,

While there are videos on getting started with programming there are not many videos on specific lines of code.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/indicator-development/97575-sample-scripts?p=786040#post786040

Intra-bar granularity involves adding a 1 tick series with AddDataSeries(), ensuring logic is processed on the primary series with BarsInProgress, and placing orders to that 1 tick series with the BarsInProgress index of that 1 tick series.

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

https://ninjatrader.com/support/helpGuides/nt8/barsinprogress.htm

There is an example script provided for a simplified demonstration.

https://ninjatrader.com/support/helpGuides/nt8/backtesting_ninjascript_strate.htm

Have you started by adding the 1 tick series with AddDataSeries()?

[rafael_delima86]

Does anyone have a video on how to add intra-bar granularity? I’m new to this and it can’t understand coding that good =(

[NinjaTrader_ChelseaB]

Hello connorgrant5,

This particular issue NTEIGHT-10782 is a tough one and is not resolved and is still a limitation.

But using an added 1 tick series for intra-bar granularity works fine. Its only the convenience of Order Fill Resolution to not have to code intra-bar granularity yourself that is not available.