Strategy ignoring bars before 3/23/2015

[molecool]

I have problems having any of my strategies execute historically beyond 3/23/2015 on both of my production systems. First I thought it was a bug but I remember experiencing a similar problem in the past and having written a test strategy for that (enclosed). My strategies all use a second minute series - so this has something to do with that.

Anyway, here's the output of my test strategy:

**NT** Enabling NinjaScript strategy 'HistoricalTest/308f01d2f985474da77e70b1a4bc5a7f' : On starting a real-time strategy - StrategySync=SubmitImmediately SyncAccountPosition=False EntryHandling=AllEntries EntriesPerDirection=1 StopTargetHandling=PerEntryExecution ErrorHandling=StopStrategyCancelOrdersClosePositions ExitOnClose=False Set order quantity by=Strategy ConnectionLossHandling=KeepRunning DisconnectDelaySeconds=120 CancelEntryOrdersOnDisable=False CancelExitOrdersOnDisable=True CalculateOnBarClose=False MaxRestarts=10 in 30 minutes
**** 0 FTOB ****
Zero series time: 3/23/2015 5:00:00 PM
One series time: 3/23/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/24/2015 5:00:00 PM
One series time: 3/24/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/25/2015 5:00:00 PM
One series time: 3/25/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/26/2015 5:00:00 PM
One series time: 3/26/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/27/2015 5:00:00 PM
One series time: 3/27/2015 4:59:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = False
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/30/2015 5:00:00 PM
One series time: 3/30/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 3/31/2015 5:00:00 PM
One series time: 3/31/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/1/2015 5:00:00 PM
One series time: 4/1/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/2/2015 5:00:00 PM
One series time: 4/2/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/3/2015 5:00:00 PM
One series time: 4/3/2015 4:58:23 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = False
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/6/2015 5:00:00 PM
One series time: 4/6/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/7/2015 5:00:00 PM
One series time: 4/7/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/8/2015 5:00:00 PM
One series time: 4/8/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/9/2015 5:00:00 PM
One series time: 4/9/2015 5:00:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
Last 0 bar: False
Last 1 bar: False
***************
**** 0 FTOB ****
Zero series time: 4/10/2015 5:00:00 PM
One series time: 4/10/2015 4:58:59 PM
isHistoricalBOT = False
Historical = True
Times[0][0] <= Times[1][0] = False
Last 0 bar: True
Last 1 bar: True
***************

I have several years of 1440 minute bars open on this chart. Curiously it's not plotting before 3/23/2015. I just can't figure out the issue. It's not the TwoHundredFifty six bar setting either - that one only affects internal indicators, correct? Enclosed also a screen grab of the strategy settings.

Just to be clear - I only have that problem on my production systems but not on my dev systems. There it goes all the way back to the very first candle on my chart. So clearly it's some setting somewhere.

[NinjaTrader_ChelseaB]

Hello molecool,

Your strategy requires tick data.
(Line 45 Add(PeriodType.Tick, 1)

May I confirm that you have tick data on your system from 3/23 and previous?

(The backtest will start after all data series have data for the test.)

[molecool]

Hello molecool,

Your strategy requires tick data.
(Line 45 Add(PeriodType.Tick, 1)

May I confirm that you have tick data on your system from 3/23 and previous?

(The backtest will start after all data series have data for the test.)

Actually my other strategies are using minute data as the second series. However even with tick data it should have loaded more. Also, I have several Kinetick subscriptions which I'm using on those systems - not a question of access to data.

[NinjaTrader_ChelseaB]

Hello molecool,

I am referring to a particular line of code in your strategy. Line 45. This line has Add(PeriodType.Tick);. If you do not have enough tickdata, this line will prevent your script from processing before there is tick data available.

By letting me know that you use Kinetick, you have let me know that there is a question as to access of data.

When requesting tick data with Kinetick between the hours of 9:30am ET to 4:30pm ET you will have access to 8 days of historical tick data. If you request the data outside those hours the full 180 days is available. After you request the data, it will be downloaded to your historical database and cache, and you will be able to access this data at any time.

Requesting tick data consumes an extremely large amount of bandwidth and heightens data server loads. When bandwidth or server load is elevated to high levels, the result is that all data, including live data can be slowed down. We decided to make 180 days of tick data available, far more than other providers, and will not compromise the data by filtering it. We simply reduce any possibilities of overloading servers and bandwidth by limiting the amount of data available during peak trading time; the times that traders need data to be as fast as possible.

If the test is going back 180 days, this most likely the issue. You do not have enough tick data to run the test.

So I ask again. May I confirm that you have tick data on your system from 3/23 and previous?

[molecool]

Hello molecool,

I am referring to a particular line of code in your strategy. Line 45. This line has Add(PeriodType.Tick);. If you do not have enough tickdata, this line will prevent your script from processing before there is tick data available.

By letting me know that you use Kinetick, you have let me know that there is a question as to access of data.

When requesting tick data with Kinetick between the hours of 9:30am ET to 4:30pm ET you will have access to 8 days of historical tick data. If you request the data outside those hours the full 180 days is available. After you request the data, it will be downloaded to your historical database and cache, and you will be able to access this data at any time.

Requesting tick data consumes an extremely large amount of bandwidth and heightens data server loads. When bandwidth or server load is elevated to high levels, the result is that all data, including live data can be slowed down. We decided to make 180 days of tick data available, far more than other providers, and will not compromise the data by filtering it. We simply reduce any possibilities of overloading servers and bandwidth by limiting the amount of data available during peak trading time; the times that traders need data to be as fast as possible.

If the test is going back 180 days, this most likely the issue. You do not have enough tick data to run the test.

So I ask again. May I confirm that you have tick data on your system from 3/23 and previous?

Okay, not to get bogged down by that one line of code - just change it to 1-minute and then tell me what you get. 180 days is not my problem - if you read my post I'm not getting anything before 3/26/2015 - via minute data. THAT is my problem. And it's not happening on my dev system for some reason - no matter when - only on my two production systems.

[NinjaTrader_ChelseaB]

Hi molecool,

How are you determining that the script is not processing before 3/26/15?

As far as I can tell this script does not place orders.

To test this, I have removed the Add(PeriodType.Tick, 1); line. Then I removed any reference to the second data series.

Last, I added a simple DrawDot() to the first line of OnBarUpdate().
(I added a dot drawing object because the output window is being cutoff due to too many lines and reaching the maximum threshold.)

The dots appear on each bar correctly all the way to the beginning of the chart.

[molecool]

Hi molecool,

How are you determining that the script is not processing before 3/26/15?

As far as I can tell this script does not place orders.

To test this, I have removed the Add(PeriodType.Tick, 1); line. Then I removed any reference to the second data series.

Last, I added a simple DrawDot() to the first line of OnBarUpdate().
(I added a dot drawing object because the output window is being cutoff due to too many lines and reaching the maximum threshold.)

The dots appear on each bar correctly all the way to the beginning of the chart.

It's not printing anything before 3/26 - look in your output window. At least it doesn't do it for me on my production systems.

[molecool]

Hi molecool,

How are you determining that the script is not processing before 3/26/15?

As far as I can tell this script does not place orders.

To test this, I have removed the Add(PeriodType.Tick, 1); line. Then I removed any reference to the second data series.

Last, I added a simple DrawDot() to the first line of OnBarUpdate().
(I added a dot drawing object because the output window is being cutoff due to too many lines and reaching the maximum threshold.)

The dots appear on each bar correctly all the way to the beginning of the chart.

Can you please attach your modified strategy so I can run it on my end? Thanks.

[NinjaTrader_ChelseaB]

Hello,

Attached is the modified script.

How are you determining that the script is not processing before 3/26/15?

[molecool]

Hello,

Attached is the modified script.

How are you determining that the script is not processing before 3/26/15?

Yeah, I don't care about the 1st series at all - the problem is with the 2nd series. It seems you're not getting the problem I'm dealing with. Use these methods instead:

  protected override void Initialize()
        {
            this.CalculateOnBarClose = false;
			TraceOrders = true; 
			ExitOnClose = false;
			this.BarsRequired = 0;
			
			ConnectionLossHandling = ConnectionLossHandling.KeepRunning; // Keeps the strategy running as if no disconnect occurred
			DisconnectDelaySeconds = 120; // Disconnect has to be at least 120 seconds
			MaxRestartAttempts = 10;
			MaxRestartMinutes = 30; // restart strategy only if restart attempts < MaxRestartAttempts within the last 30 minutes
			
			Add(PeriodType.Minute, 1);
        }

        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {
			//DrawDot("myDot"+CurrentBar, true, 0, High[0]+3*TickSize, Color.Blue);
			
			bool[] lastBars = new bool[BarsArray.Length];
			for (int x = 0 ; x < BarsArray.Length; x++) {
				lastBars[x] = CalculateOnBarClose ? CurrentBar == Count - 2 : CurrentBar == Count - 1;
				//Print("Last bar " + x + ": " + lastBars[x]);
			}
			
			bool isHistorical = Historical;
			//bool isHistoricalBOT = lastBars[1] ? Times[0][0] <= Times[1][0] : true;
			bool isHistoricalBOT = true;
		
			int startIndexSeriesZero = isHistorical ? 0 : 1;
			int startIndexSeriesOne = isHistorical ? 1 : 0;
			
			int startIndexSeriesZeroBOT = isHistoricalBOT ? 0 : 1;
			int startIndexSeriesOneBOT = isHistoricalBOT ? 1 : 0;
			
//			if (BarsInProgress == 0) {
//				
//				Print("**** 0 Outside ****");
//				Print("Zero series time: " + Times[0][0]);
//				//Print("One series time: " + Times[1][0]);
//				Print("isHistoricalBOT = " + isHistoricalBOT);
//				Print("Historical = " + Historical);
//				//Print("Times[0][0] <= Times[1][0] = " + (Times[0][0] <= Times[1][0]));
//				Print("***************");
//				
//				if(FirstTickOfBar) {
//					Print("**** 0 FTOB ****");
//					Print("Zero series time: " + Times[0][0]);
//					//Print("One series time: " + Times[1][0]);
//					Print("isHistoricalBOT = " + isHistoricalBOT);
//					Print("Historical = " + Historical);
//					//Print("Times[0][0] <= Times[1][0] = " + (Times[0][0] <= Times[1][0]));
//					Print("Last 0 bar: " + lastBars[0]);
//					//Print("Last 1 bar: " + lastBars[1]);
//					Print("***************");
//				}
//			}
			
			if (BarsInProgress == 1) {
				if(FirstTickOfBar) {
					//Print("**** 111111 ****");
					//DrawDot("myDot"+CurrentBars[1], true, 0, Highs[1][0]+3*TickSize, Color.Blue);
					Print("Zero series time: " + Times[0][0]);
					Print("One series time: " + Times[1][0]);
					Print("isHistoricalBOT = " + isHistoricalBOT);
					Print("Historical = " + Historical);
					Print("Times[0][0] <= Times[1][0] = " + (Times[0][0] <= Times[1][0]));
					Print("***************");
				}
			}

This is what I'm getting on my end:

**NT** Enabling NinjaScript strategy 'HistoricalTestModified/19955d86646649cdb63708ac840930bc' : On starting a real-time strategy - StrategySync=SubmitImmediately SyncAccountPosition=False EntryHandling=AllEntries EntriesPerDirection=1 StopTargetHandling=PerEntryExecution ErrorHandling=StopStrategyCancelOrdersClosePositio ns ExitOnClose=False Set order quantity by=Strategy ConnectionLossHandling=KeepRunning DisconnectDelaySeconds=120 CancelEntryOrdersOnDisable=False CancelExitOrdersOnDisable=True CalculateOnBarClose=False MaxRestarts=10 in 30 minutes
Zero series time: 3/23/2015 5:00:00 PM
One series time: 3/23/2015 5:01:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
***************
Zero series time: 3/23/2015 5:00:00 PM
One series time: 3/23/2015 5:02:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True
***************
Zero series time: 3/23/2015 5:00:00 PM
One series time: 3/23/2015 5:03:00 PM
isHistoricalBOT = True
Historical = True
Times[0][0] <= Times[1][0] = True

See how it starts on 3/23/2015? WHY? It's connected to a live feed of Kinetick - it should START at the first candle on that chart. It does not. THIS is the problem.

[NinjaTrader_ChelseaB]

Hello molecool,

I've copied your code into the script using a secondary series of 1 minute.

I un-commented the DrawDot line you have commented out as this is the only way I can tell where the strategy is actually starting (the output window cannot be used because it is reaching the max threshold).

I am still finding that dots are being drawn at the start of the script correctly and continue to draw until the last bar. I am unable to find anything wrong on my end.


Attached is the script that I have tested and a screenshot showing that the dots are drawing from the first bar correctly. I am not finding any issues with this script.

However, its sounds like you do not want me to work with you on this and that I am not being helpful to you.
I will try and find another NinjaScript support agent to continue assisting you.

[molecool]

Hello molecool,

I've copied your code into the script using a secondary series of 1 minute.

I un-commented the DrawDot line you have commented out as this is the only way I can tell where the strategy is actually starting (the output window cannot be used because it is reaching the max threshold).

I am still finding that dots are being drawn at the start of the script correctly and continue to draw until the last bar. I am unable to find anything wrong on my end.


Attached is the script that I have tested and a screenshot showing that the dots are drawing from the first bar correctly. I am not finding any issues with this script.

However, its sounds like you do not want me to work with you on this and that I am not being helpful to you.
I will try and find another NinjaScript support agent to continue assisting you.

Forgive me but I just don't think you understand what the problem is, Chelsea - my strategies execute on the 2nd series and you keep drawing dots on top of the 1st series which isn't the issue at all. I may not make myself clear - if so then perhaps a fresh set of eyes may be of assistance, yes

[molecool]

Is anyone taking over this thread? I still am waiting for someone to help me fix this problem.

[NinjaTrader_PatrickH]

Hello molecool,

Thank you for your response.

I understand you are referring to the secondary series. Chelsea was providing an example using Minute data as the secondary series. So let us go back to using tick data.

I would like you to confirm you have the needed tick data by going to Tools > Historical Data Manager > $EURUSD > Last > Tick. Check that the data goes back as far before the 23rd of March 2015. If it does, then disconnect from Kinetick and run the backtest againand then please confirm that you are still not seeing your internal logic processed before the 23rd.

[molecool]

Hello molecool,

Thank you for your response.

I understand you are referring to the secondary series. Chelsea was providing an example using Minute data as the secondary series. So let us go back to using tick data.

I would like you to confirm you have the needed tick data by going to Tools > Historical Data Manager > $EURUSD > Last > Tick. Check that the data goes back as far before the 23rd of March 2015. If it does, then disconnect from Kinetick and run the backtest againand then please confirm that you are still not seeing your internal logic processed before the 23rd.

No, this is all going upside down now.

1) Sending you a test with tick data was a mistake. All my strategies are using 1-minute bars as the secondary series. I am testing this with a 1440 minute chart so the 1-minute series is the second one. Hope that is clear.

2) There is no problem with my particular system as I am experiencing this on two separate production systems.

3) And yes, I have relaunched Kinetick several times and it doesn't make any difference.

I don't know what's causing this on my two prod systems - I only know that it's not happening on my dev system.