MTF in an Indicator

[Harry]

Another observation

If you have a limited amount of tick data available for the primary data series (let us say 30 bars), then NinjaTrader will actually start with the secondary data series after switching from historical to real-time data.

Below is an example. The first real-time tick, which by conincidence was after a minute candle was completed, triggered BarSeries1 prior to the BarSeries 0.

True 11.03.2011 13:11:05 0 True
True 11.03.2011 13:11:34 0 True
True 11.03.2011 13:12:00 1 True
True 11.03.2011 13:12:03 0 True
True 11.03.2011 13:12:32 0 True
True 11.03.2011 13:12:53 0 True
True 11.03.2011 13:13:00 1 True
[COLOR="Red"][B]False 11.03.2011 13:14:00 1 True[/B][/COLOR]
False 11.03.2011 13:13:11 0 False
False 11.03.2011 13:14:00 1 False
False 11.03.2011 13:13:14 0 False

Once enough data is there, NinjaTrader will always start the real.time data with DataSeries 0. In this case it should first have triggered an intrabar tick for the primary DataSeries 0, before triggering the correct FirstTickOfBar for Bar Series 1.

This is not the problem we were talking about in the previous post, it is just an inverted order for BarsInProgress==0 and BarsInProgress = 1 for real-time bars.

[Harry]

Not easily reproducible

I had a chart running this morning, and I could not yet reproduce the event with FDAX and EURUSD, which I had occasionally observed yesterday evening for 6E and ES.

I have not changed the indicator, which basically indicates whether a Hybrid bar ("hybrid") or a new bar ("new") was plotted as first real-time bar. Further "true" indicates expected behavior, while "false" indicates false behavior, i.e. inserting the first tick of the Bar Series 1 within the real-time instead of the historical section of the chart.

I think that there is a synchronization issue behind, which delays the insertion, depending on the amount of real-time tick data temporarily stored by NinjaTrader. This would mean that the problem only occurs after NinjaTrader has been running for several hours building up the data base.

In its way this problem is very similar to the price and volume spikes that I have observed for the first real-time bars, although this was on a single BarSeries chart. In that case false tick data from the temporary data base was used to build the first real-time bar. The 50 or 100 price spikes that I have observed always pointed in the direction of the prior data, none of them was out-of-range.

This means that there are now three different bugs that relate to the way NinjaTrader builds the first real-time bar.

-> volume spikes
-> price spikes
-> false FirstTickOfBar for secondary DataSeries

I agree that this only one bar - the "connection bar", but wouldn't it be nice if this is correct as well? I acknowledge that the thing is difficult to reproduce and, even worse, time-consuming. You need to wait for an hour to be able to produce the occurence with the spikes once and only once with a probability of maybe 5-10%.

[Harry]

Here comes an example

So finally had an event that I was looking for and captured both the chart and the contents from the output window. Had to let NinjaTrader run for about 15 minutes, then it worked.

This is the output file for the period between 13:58:59 to 14:01:01, showing the transition from historical to real-time data. It shows the erroneous FirstTick which was inserted in the middle of the 14:01 bar for BarSeries 1. I believe that it is the late coming last loop from the historical section, which inserts the preceding ticks into the current bar of BarSeries1.

True 11.03.2011 13:58:59 0 True
True 11.03.2011 13:59:00 1 True
True 11.03.2011 13:59:10 0 True
True 11.03.2011 13:59:16 0 True
True 11.03.2011 13:59:22 0 True
True 11.03.2011 13:59:31 0 True
True 11.03.2011 13:59:39 0 True
True 11.03.2011 13:59:47 0 True
True 11.03.2011 13:59:58 0 True
True 11.03.2011 14:00:00 1 True
True 11.03.2011 14:00:13 0 True // this tick should be used for building the hybrid bar for BarSeries 1
True 11.03.2011 14:00:27 0 True // idem
True 11.03.2011 14:00:36 0 True // idem
True 11.03.2011 14:00:43 0 True // idem
True 11.03.2011 14:00:48 0 True // idem last loop for BarSeries 1 that builds historical part of hybrid bar is missing
False 11.03.2011 14:00:48 0 True 
False 11.03.2011 14:01:00 1 True // this is not the first tick for the 14:01:00 minute bar, we are approaching the end of that bar
False 11.03.2011 14:00:49 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False // why is this tick not triggering an update of BarSeries 1 as well?
False 11.03.2011 14:00:51 0 True
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:51 0 False // why is this tick not triggering an update of BarSeries 1 as well?
False 11.03.2011 14:00:59 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:59 0 False
False 11.03.2011 14:00:59 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:59 0 False
False 11.03.2011 14:01:02 0 False
False 11.03.2011 14:02:00 1 True  // this is the correct first tick for the next bar of BarSeries 1
False 11.03.2011 14:01:02 0 False
False 11.03.2011 14:01:03 0 False
False 11.03.2011 14:02:00 1 False
False 11.03.2011 14:01:03 0 False
False 11.03.2011 14:01:05 0 True

The chart below and the transcript from the output window shows that the FirstTickOfBar for BarSeries 1 (insertion of prior ticks from historical data into that bar) takes place after the first real-time bar of BarSeries 0 has started.

So you get a false FirstTickOfBar for BarSeries 1, which is the second real-time loop of OnBarUpdate.

Also I do not understand why a new event always triggers BarSeries 0, but only sometimes BarSeries 1. The transcript shows that the loops for the first and second BarSeries are not alternating in a proper way. That is you often get two loops of BarSeries 0 for one of BarSeries 1.

[NinjaTrader_JoshP]

Harry,

The issue is that as you roll historical to live there is a lot of work being done and the thread is actually blocked. As such we do not recommend relying on any FirstTickOfBar logic for that first bar and to code workarounds like you currently have already done.

[Harry]

Thanks for the frank answer.

Unfortunately there is no perfect workaround for this issue, as the logic of NT is event driven, which means that the first tick of the new bar specifies when do make the final call in affecting a value. I have coded around but opened up a new problem, which occurs less frequent, so my code will still occasionally produce errors for that first real-time bar. But I guess I'll have to live with that, just wanted to point to the problem, as it is supposedly related to those price spikes I am occasionally getting for those same bars.

Harry,

The issue is that as you roll historical to live there is a lot of work being done and the thread is actually blocked. As such we do not recommend relying on any FirstTickOfBar logic for that first bar and to code workarounds like you currently have already done.

[Harry]

There remains one question concerning the behavior of NinjaTrader on real-time data.
Why does the test show two consecutive runs for BarsInProgress==0 and then a single run for BarsInProgress==1?

My understanding was that a new real-time tick will generate a new loop for OnBarUpdate for both the primary BarSeries and the BarSeries added by the indicator. Seems like there are fewer ticks generated for the added series, or am I missing something?

True 11.03.2011 14:00:43 0 True // idem
True 11.03.2011 14:00:48 0 True // idem last loop for BarSeries 1 that builds historical part of hybrid bar is missing
False 11.03.2011 14:00:48 0 True 
False 11.03.2011 14:01:00 1 True // this is not the first tick for the 14:01:00 minute bar, we are approaching the end of that bar
False 11.03.2011 14:00:49 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:50 0 False 
[COLOR="Red"]False 11.03.2011 14:00:51 0 True // two consecutive runs for BarsInProgress==0[/COLOR]
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:51 0 False 
[COLOR="#ff0000"]False 11.03.2011 14:00:59 0 False // two consecutive runs for BarsInProgress==0[/COLOR]
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:59 0 False
[COLOR="#ff0000"]False 11.03.2011 14:00:59 0 False // two consecutive runs for BarsInProgress==0[/COLOR]
False 11.03.2011 14:01:00 1 False
False 11.03.2011 14:00:59 0 False
[COLOR="#ff0000"]False 11.03.2011 14:01:02 0 False // two consecutive runs for BarsInProgress==0[/COLOR]
False 11.03.2011 14:02:00 1 True  
False 11.03.2011 14:01:02 0 False
F[COLOR="#ff0000"]alse 11.03.2011 14:01:03 0 False // two consecutive runs for BarsInProgress==0[/COLOR]
False 11.03.2011 14:02:00 1 False
False 11.03.2011 14:01:03 0 False
[COLOR="#ff0000"]False 11.03.2011 14:01:05 0 True // two consecutive runs for BarsInProgress==0[/COLOR]

[NinjaTrader_Brett]

Hello,

Since Josh has been working with you I will have him follow up on Monday when he returns in the office.

Let me know if I can be of further assistance.

[Harry]

Thanks Brett, no urgence with this question, I will wait.

Hello,

Since Josh has been working with you I will have him follow up on Monday when he returns in the office.

Let me know if I can be of further assistance.

[NinjaTrader_JoshP]

Harry,

I have never seen that before. How were you able to produce that outcome?

[Harry]

Harry,

I have never seen that before. How were you able to produce that outcome?

Just by adding this line

Print(Historical + " " + Time[0] + " " + BarsInProgress + " " + FirstTickOfBar);

directly after the opening bracket of OnBarUpdate().

I you have never seen it, what about your own example in post # 14. There is also one case of two consecutive runs of BarSeries 0 in real-time mode.

[NinjaTrader_JoshP]

What I posted earlier was sliced and diced many ways with cut and paste before and I would not know if it is reflective of anything.

Here is the modified code which would actually specifically track such a scenario. I have ran it for the past 20 minutes with multiple resets to no avail. I even tested several scenarios where the first real-time tick of BIP0 has FirstTickOfBar = true.

if (!Historical && BarsInProgress == prevBIP)
      Print(Historical + " " + Time[0] + " " + BarsInProgress + " " + FirstTickOfBar + " DOUBLE BIP");
else
      Print(Historical + " " + Time[0] + " " + BarsInProgress + " " + FirstTickOfBar);
prevBIP = BarsInProgress;

Are you experiencing this as a prerequisite?

True 11.03.2011 14:00:48 0 True // idem last loop for BarSeries 1 that builds historical part of hybrid bar is missing
False 11.03.2011 14:00:48 0 True 
[COLOR=Red]False 11.03.2011 14:01:00 1 True // this is not the first tick for the 14:01:00 minute bar, we are approaching the end of that bar[/COLOR]

[Harry]

Sorry Josh, only saw your answer now. I am not able to test it immediately, but will come back in a few hours.

[Harry]

Special case: Only few tick bars in primary series

First problem.

If I test your indicator on a starting tick bar series (Interactive Brokers has no historical tick data), you will find that the tick data is ignored altogether, and only the minute data is called.

NinjaTrader requires a minimum number of tick bars to be willing to call the primary bar series at all. With my data series settings of 3 days, NinjaTrader calls the added minute data for three days, but is unwilling to insert the real-time tick data to this series, although it has inserted a few historical bars, as you can see from the output files below.

Problem: This regularly crashes my indicators, if there are too few tick bars, so I have to wait for a few minutes to avoid these crashes.

Have saved the output files and attached below.

[Harry]

Too little market activity now to do further testing. So will do this tomorrow in the early morning.

[NinjaTrader_JoshP]

Harry,

You can probably try adding this to Initialize() to help that issue:

BarsRequired = 0;