Calculate any indicator for any instrument, timeframe, etc

[jeronymite]

In NinjaScript, I want to be able to use any indicator method to provide the value of the indicator for any instrument on any timeframe with any number of bars look back from any reference bar, etc. Something like the following:

INDICATOR(<instrument>, <timeframe>, <period>, <barsago>, <marketdatatype>)

So, for example, to find the EMA of the Ask of EURUSD on a 7 minute timeframe across a period of 11 bars starting 6 bars ago, I might have something like:

EMA($EURUSD, PeriodType.Minute, 7, 11, 6, MarketDataType.Ask)

I specifically only need it for the actual calculations within a script, but obviously, it may also be desirable to be able to plot this somehow. But the ability to calculate these types of ad hoc values and the high flexibility they afford is, to me, of immense value. It means I can use the indicator values for any instrument at any time on any arbitrary set of parameters, and without the need for a chart that matched the desired characteristics, or indeed any chart at all.

The caveat, presumably, would be that one had a data series that facilitated this. For the flexibility demanded, the only feasible series would be a tick series. Consequently, it may be a requirement of such a feature that a relevant tick series was required in order to be able to invoke the method.

The benefits would be streamlining of code and a major consolidation of data series resulting in huge efficiency increases over having multiple data streams. Essentially, by having such a function and a single tick data series for each instrument and the appropriate market data type one requires, one has access to the equivalent of a massive set of data series and indicator values.

I know this does not exist in NT7, but has anyone written this type of thing?

Can it be added as a desirable feature for NT8, please?

Many thanks for any advice/assistance!

[NinjaTrader_ChelseaB]

Hello jeronymite,

Thanks for your post.

In NinjaScript it is possible to call an indicator using a secondary data series with a desired market data type.

Because the indicator will need a data series to make calculations with you will need to add the data series to the original script.

Attached is a sample strategy that calls the EMA using a 1 hour series with ask as the market data type while the primary data series is processing. If you add this to a 1 minute chart, the strategy will print the value of the EMA of a 1 hour ask series, 6 bars back, once per minute to the output window.

Do keep in mind that if you are expecting historical values, you will need to be connected to a data provider that provides historical minute ask data.

You can check to see if your data provider provides historical minute ask data from the chart in the 'Understanding the data provided by your connectivity provider' section of the help guide linked below.
http://www.ninjatrader.com/support/h...rical_data.htm
.

[jeronymite]

Thanks, Chelsea.

Although interesting, that is really highly constrained compared with what I'm after:

• For any indicator, call a single method once on any instrument for any data series period (1 second to 1 month potentially, in any time increment, e.g. every 37 seconds or 9 days and 27 minutes, etc), any market data type
• Only one single data series required ... at all (perhaps a 1 second or tick series) for a specific instrument and market data type
• Not dependent on having a chart; no chart required!

In essence, it is an all-in-one method for each individual indicator that allows a value to be obtained for any instrument, any market data type, any time span (i.e. "virtual" bars), number of bars to look back, any reference bar to start from.

It is the concept of "virtual" bars of any duration that is key to this and makes it possible.

Obviously, a data series that can be used to construct any virtual bar would be needed, as mentioned already.

Hope that clarifies the functionality I'm looking for.

Thanks!

[NinjaTrader_ChelseaB]

Hello,

Thanks for your reply.

The only difference I am really able to understand is that instead of supplying the indicator with data, you would rather just pass a data type and have the indicator automatically add this.

While this would be possible to code into an indicator once, we cannot support dynamically adding a data series using variables and it would not work if you were trying to dynamically change the data series being used.

This is because the data series must be added in Initialize(). Initialize is called for all indicators, when the Indicators window is opened. If you were to have a variable that changed, this would be reset when the Indicators window is opened.
Further, if you were to change the number of data series and then Initialize() is called, this would cause NinjaTrader to behave unexpectedly.
This also means that a separate instance of the callee indicator is initialized for each data series of the caller script. The initialize of each instance of the callee indicator is run at the same time as the initialize of the caller script.
This means that if you change the data series you are calling, this most likely will not be changed in the callee indicator because Initialize() would not be run again.

This would require allowing a data series to be added outside of Initialize(). It is not possible to add a data series outside of Initialize() because this would cause NinjaTrader to no longer run the indicator for each bar and collect values to data series within it.

In the meantime, as I've mentioned before its already possible to call an indicator using any desired data series as long as you add it to the caller script first and call the indicator using the BarsArray with the desired index.
.

[jeronymite]

Thanks again, Chelsea.

Let me try again to explain how this would work.

1. For a specific instrument and a specific market data type, add just one data series in Initialize of, say, 1 second or 1 tick. This becomes the "base" data series from which all indicator calls for that specific instrument with that specific market data type are derived.
2. From that one data series alone, call the indicator for any and all desired bar lengths; for example, 1 minute, 5 minutes, 1 hour, ... ... 37 seconds, 2 hours 43 minutes and 17 seconds, 4 days, etc etc -- this is the "virtual bar" concept.
3. One would add any number of similar "base" data series in Initialize, but only ever need one for any instrument of a particular market data type.
4. This is entirely independent of indicator windows. This is NinjaScript code. A chart is not required.

The critical difference between this and what currently exists is profound. There would only ever need to be one data series for a specific instrument with a specific market data type. All bar durations of any length could be derived from that one single data series. Currently, every different bar duration requires its own data series; for example, to get the values of an indicator for a 1 minute bar requires a 1 minute data series; a second data series is required to get 15 minute values of that indicator, etc. One must anticipate every possible bar duration and add series to provide that data.

In developing flexible applications, the ability to allow users to choose any values for specific parameters is extremely important. Within an application, I want to allow the user to select any arbitrary bar duration on any supported instrument with any supported market data type. At runtime, that can only be achieved in practical terms with a method such as I am proposing, using virtual bars. It is highly undesirable to "quantize" the bar durations to a limited subset, as is currently the case, where that pre-defined subset is required to have multiple corresponding data series.

To do this in the most effective, efficient and functionally flexible and comprehensive manner, the implications for NinjaTrader are significant:

• There will be implications for the client application architecture
• Server-side changes would be required to support the generation of an efficient virtual bar data stream of any bar duration within a single data series ... which would, in fact, become a single data connection, capable of being used for any bar duration
• To be truly effective, it would be necessary to allow new data series to be added dynamically, not just in Initialize

As I indicated in my original post, this is non-trivial functionality that would be an NT8 candidate, not any permutation of what is currently available within NT7. Nevertheless, one could implement code to do this in NT7, albeit less efficient than a native server-supported architectural implementation.

I'm keen to see that the concept is clearly understood so that consideration is given to it within NT8, but also to see if anyone has implemented anything of this nature in NT7 already.

I hope that helps to clarify further.

[sledge]

Is your CPU maxxed out?


Are you using a 486DX or something?

Thanks again, Chelsea.

Let me try again to explain how this would work.

1. For a specific instrument and a specific market data type, add just one data series in Initialize of, say, 1 second or 1 tick. This becomes the "base" data series from which all indicator calls for that specific instrument with that specific market data type are derived.
2. From that one data series alone, call the indicator for any and all desired bar lengths; for example, 1 minute, 5 minutes, 1 hour, ... ... 37 seconds, 2 hours 43 minutes and 17 seconds, 4 days, etc etc -- this is the "virtual bar" concept.
3. One would add any number of similar "base" data series in Initialize, but only ever need one for any instrument of a particular market data type.
4. This is entirely independent of indicator windows. This is NinjaScript code. A chart is not required.

The critical difference between this and what currently exists is profound. There would only ever need to be one data series for a specific instrument with a specific market data type. All bar durations of any length could be derived from that one single data series. Currently, every different bar duration requires its own data series; for example, to get the values of an indicator for a 1 minute bar requires a 1 minute data series; a second data series is required to get 15 minute values of that indicator, etc. One must anticipate every possible bar duration and add series to provide that data.

In developing flexible applications, the ability to allow users to choose any values for specific parameters is extremely important. Within an application, I want to allow the user to select any arbitrary bar duration on any supported instrument with any supported market data type. At runtime, that can only be achieved in practical terms with a method such as I am proposing, using virtual bars. It is highly undesirable to "quantize" the bar durations to a limited subset, as is currently the case, where that pre-defined subset is required to have multiple corresponding data series.

To do this in the most effective, efficient and functionally flexible and comprehensive manner, the implications for NinjaTrader are significant:

• There will be implications for the client application architecture
• Server-side changes would be required to support the generation of an efficient virtual bar data stream of any bar duration within a single data series ... which would, in fact, become a single data connection, capable of being used for any bar duration
• To be truly effective, it would be necessary to allow new data series to be added dynamically, not just in Initialize

As I indicated in my original post, this is non-trivial functionality that would be an NT8 candidate, not any permutation of what is currently available within NT7. Nevertheless, one could implement code to do this in NT7, albeit less efficient than a native server-supported architectural implementation.

I'm keen to see that the concept is clearly understood so that consideration is given to it within NT8, but also to see if anyone has implemented anything of this nature in NT7 already.

I hope that helps to clarify further.

[sledge]

How would your code handle any indicator?!

And imagine them on the fly? Indicators require many inputs... Are you defaulting them and not allowing overrides??

Everyone loves the macd(12,26,9) since it is default... but I have better.

If Koganam doesn't have a solution - one does not exist.




Code:
INDICATOR(<instrument>, <timeframe>, <period>, <barsago>, <marketdatatype>)

In NinjaScript, I want to be able to use any indicator method to provide the value of the indicator for any instrument on any timeframe with any number of bars look back from any reference bar, etc. Something like the following:

[jeronymite]

Thanks for your responses, sledge.

Each indicator still exists in its own right: EMA is still invoked by EMA, ATR by ATR, CCI by CCI ... MACD by MACD, etc. No change there.

It's the invocation of any particular indicator that is different. And in reality it is only one extra parameter: timeframe. The invocation would allow any desired timeframe to be specified so that the value of the indicator for that particular timeframe would be calculated/derived and supplied to the caller. The methodology of supply has many potential variations, some efficient and some less so, some implemented in the client only, others in the client/server architecture. The concept of "virtual bars", as I have called them, is one approach that has the flexibility and scope required.

The crucial benefit is to allow the indicator values of any number of arbitrary timeframes to be obtained quite independent of timeframe-specific data series. All that is needed is one single "base" or reference data series at a low enough granularity to allow the building of the virtual bars. One tick or one second should do the job.

NOTE: This is aimed (initially at least) purely at NinjaScipt code, not indicators on charts, although the potential is there to extend it to the charts, but with significantly more complexity required.

So, I simply want to call any given indicator with an extra "timeframe" parameter and to do so without having to Add a multitude of data series.

I have a rudimentary implementation in NinjaScript, and as a proof of concept it seems to work well enough. I'm looking to see if anyone else has considered this and worked on it. I would also recommend the concept as an NT8 feature.

Hope that helps in the understanding.

[koganam]

How would your code handle any indicator?!

And imagine them on the fly? Indicators require many inputs... Are you defaulting them and not allowing overrides??

Everyone loves the macd(12,26,9) since it is default... but I have better.

If Koganam doesn't have a solution - one does not exist.




Code:
INDICATOR(<instrument>, <timeframe>, <period>, <barsago>, <marketdatatype>)

I use composite bars to create rolling bars that show a trend. Actually, I think Heiken-Ashi bars are a massaged composite of 2 bars.

[Harry]

3 ways to code MTF indicators

Let me just repost something that I have written for a different forum .....

Three Ways To Code MultiTimeFrame Indicators

There are three options

(1) loading secondary bar series and calculating indicator values from secondary bars
(2) calculating composite bars from primary bars and calculating indicator values from composite bars
(3) tweaking the indicator

Let us have a look at the three options in detail:

(1) Calculating from secondary bar series

-> it is possible to reproduce the genuine values which would be obtained from the original bar series
-> for real-time bars, NinjaTrader processes the primary bars first and the secondary bars afterwards, which introduces a lag of 1 tick (1-tick problem of NinjaTrader)
-> for historical bars the value of the secondary bars can be accessed prior to these bars being processed by OnBarUpdate(), if timestamps of primary and secondary bar series mathches
-> the transition from historical to real-time bars is difficult to code
-> a special effort has to be made for session breaks, as time-built bars and tick-built bars with the time stamp of the session break belong to different sessions
-> the drunken sailor path of snapshots can only be avoided by smoothing or interpolating the distance covering the unstable period of secondary bars
-> when the secondary bars are range or Renko bars, the MTF indicator cannot be called from a strategy

(2) Calculating from composite bars

-> it is possible to reproduce the genuine values which would be obtained from the original bar series
-> composite bars are calculated and stored in arrays, which are similar to DataSeries objects
-> the array are accessed in a similar way as the DataSeries in standard indicators
-> when building composite bars, session breaks have to be taken into account
-> to avoid the drunken sailor path produced by snapshot data during the unstable period of the composite bars, either interpolation (repainting) or smoothing is required
-> the composite bars do not suffer from the 1-tick lag as above and can easily be called by a strategy
-> composite bars are limited to the same bar type as the primary bars
-> the period can only be an integer multiple of the bar period of the primary bars
-> genuine composite bars can only be built for time-based (day, minute, second), tick or volume charts, but not for range or Renko charts
-> for range or Renko charts an approxmination is possible via the square root relation ship between range and time
-> the composite bars approach is relatively CPU intensive

(3) Tweaked indicators

-> by definition the tweaked indicator is slightly different from the genuine higher timeframe indicator, but can be used as a substitute, or might even be superior
-> the tweaked indicator will not cut off higher timeframe bars at the session break or day break
-> tweaked indicators are limited to the same bar type as the primary bars
-> the period can only be an integer multiple of the bar period of the primary bars
-> the indicators do not suffer from lag and can be easily called from a strategy
-> the CPU load is lower compared to the other solutions


Conclusion

(1) When to use a genuine MTF indicator ....

Avoid MTF indicators, if you can. Don't use range or Renko bars as secondary bars, because you will run into trouble sooner or later.

The only reason to use a genuine MTF indicator, is when your indicator values need to be calculated from a different bar type than the primary bars. In fact there are quite a few applications, where this is necessary. Here are some examples.

(a) calculating an exact opening range on a tick, volume or range chart requires minute data
(b) calculating the genuine market profile on a tick, volume or range chart requires minute data
(c) efficient calculation of VWAP can be done by loading minute data


(2) When to use an indicator that uses composite bars ...

Easy answer: When ever primary and secondary bars are identical and you do not find a formula to tweak it.


(3) When to use a tweaked indicator...

Easy answer: When ever primary and secondary bars are identical and you find a formula to tweak it.

[Harry]

Example: Tweaked indicator

This is my favourite example for a tweaked indicator. The indicator is a normalized MACD. Instead of calculating composite bars, the higher timeframe MACDs are simply computed by recalculating the input parameters for the indicator.

The chart below shows 5 MACDs. The fastest MACD (blue) is directly calculated from the bars shown on the chart. The four other MACDs represent higher timeframes with timeframe multipliers of 2, 4, 8 and 12.

[Harry]

Example: Indicator calculated from composite bars

Composite bars are similar to "virtual bars", as they were called in a post below. However, composite bars can only be calculated as integer multiples of the primary bars. If the primary bar is a 15-minute bars, then composite bars can have a duration of 30 min, 35 min, 60 min, 75 min, etc. Any duration which is not an integer multiple is not possible.

The reason is that the data of the primary bars is used, and that data simply does not allow to calculate the open, high, low and close of a 35 min bar. Also the concept of composite bars cannot easily be applied to equal range and renko bars, because a 4-range bar cannot be built from two 2 range bars. There is a possible approximation, which uses the quasi linear relationship between range and the square root of time, but this approximation has its limits.

The chart below shows the same 5 multi-timeframe MACDs, but this time they are calculated in a completely different fashion.

-> composite bars are calculated from the primary bars by using the timeframe multiplier
-> the indicator values are calculated from the composite bars

The composite bars are cut off at the session end in the same way as NinjaTrader cuts off the primary bars. Therefore the indicators obtained are similar to the original indicator calculated from a real secondary bar series.

[jeronymite]

Many thanks indeed, Harry! I truly appreciate the time you took to provide such detailed responses and your positive approach to analysing my concept. Your own analysis is most insightful and helpful! I congratulate you on a deep understanding of the concepts and a sophisticated implementation of them, including charting.

As you say, the composite bars you discuss are very much in the nature of the virtual bars I have described. The ideas align extremely closely.

My approach has been:

• Use a base data series that is time-based and has a granularity that allows derived composite/virtual bars to be of any desired duration
• Each instrument and an associated market data type has its own base data series, so that all required instruments and associated market data types are catered for by adding appropriate data series
• I have chosen to use a base data series of one second bars, although I could choose any time period that allows the desired durations for the derived bars
• One second bars allow virtually any duration of derived bars by using an appropriate multiplier
• Importantly, one second bars also provide an optimal timeframe to ensure expedient trade execution on that data series
• The virtual bars derived from the data series are stored in pseudo-DataSeries arrays, just as you have done, and used as input to custom versions of particular indicators that allow the extra parameter of "timeframe", in addition to all standard parameters they may have, so that functionality and capability of the indicator is not in any way diminished, but rather enhanced
• I have limited the scope of usage to NinjaScript code initially, although, as you have most beautifully demonstrated, the functionality can be extended to charting as well
• I calculate composite/virtual bars, but I would be most interested in your logic for recalculating indicator input parameters to improve performance -- I will look into this myself too

Based on your comments and insights, I will continue to develop the concept. If you were willing to share any code, I would be greatly indebted to you; but I fully realise that you may not wish to do so and that is absolutely fine, of course.

I do hope, however, that we may see these ideas integrated into the set of indicators that will be made available in NT8.

Many thanks again, Harry!

[SharkCub]

Composite vs Tweaking

The problem that I found "tweaking" an indicator is that it was challenging to get a tweaked indicator to match the indicator on the high time frame chart. As such, I took the "composite" approach. This was far more complex than I had originally envisioned. For anyone who does decide to construct composite bars, some of the challenges:

1) Make sure you properly deal with session breaks (as Harry pointed out)
2) Ninja Trader drops zero volume bars from the data series
3) Real time update of "forming" bar needs to be handled carefully. This is not as simple as it may appear at first.
4) It is necessary to interpolate during the formation of and at completion of each synthetic bar. This can also cause issues with strategy studies. Make sure that the study uses the end result ONLY when the composite bar has completed or account for it as a "forming" bar.

As Harry points out, there are limitations to this approach as well - composite MUST be an even multiple of the base time frame.

Though painful, the end results were very satisfying.

[jeronymite]

NT8 ... maybe?

Any chance we may see this concept built into NT8? NT8.1?

In other words, the ability to call any standard NT indicator method (SMA, RSI, CCI, MACD, ATR, etc) as per my original post below:

INDICATOR(<instrument>, <timeframe>, <period>, <barsago>, <marketdatatype>)

So, for example, to find the EMA of the Ask price of EURUSD on a 7 minute timeframe across a period of 11 bars starting 6 bars ago, one might have something like:

EMA($EURUSD, PeriodType.Minute, 7, 11, 6, MarketDataType.Ask)

This would be the equivalent of adding a 7 minute timeframe indicator using Ask data for the required instrument and then invoking the standard NT7 indicator for the given period and specific bars ago. Obviously, one would need a base indicator data series on which to extrapolate to any other timeframe of that data, and so, provided it is clear that any parameter-ized timeframe can only be a multiple of the base data series timeframe (e.g. one second, for adequate granularity), such an approach would provide extremely desirable flexibility for time-based data series. The capability for other types of non-time-based data series would need consideration in this respect too.

Hoping something like this might be become the standard invocation method of NT (8?) built-in indicators.