SP500 Daily 200 MA and a Futures Tick-based Strategy

[dbw451]

I would like to add an SP500 cash index 200 day MA to an e-mini futures based strategy. My strategy is Tick based. I have 50+ years of historical SP500 cash index daily data. The tick data is in 3 month contracts (i.e. I am not using a continuous contract).

I am able to display a 200 day MA on a daily SP500 chart without any problems. In my strategy however, the values calculated for the 200 day MA using the SMA indicator are not correct. They are often 10+ points different than the 200 day SMA values on the daily chart.

I'm suspecting that since 200 days is a much longer lookback period than the days available in each individual historical futures contract, the SMA indicator might not be able to calculate the 200 day moving average correctly. If I use a smaller daily period MA (e.g. 10 days), the SMA indicator calculates the daily MA values properly.

There's nothing special about my strategy's implementation of the 200 day MA. I add the daily SP500 cash index in the Initialize() procedure:

protected override void Initialize()
{
Add("^SP500", PeriodType.Day, 1);
CalculateOnBarClose = true;

.....
}

And I calculate the 200 day MA when BarsInProgress == 1 in the OnBarUpdate() procedure:

protected override void OnBarUpdate()
{
if (BarsInProgress == 1)
{
daySMA200 = SMA(Close,200)[0];
....
}
}

Is there a way I can calculate a 200 day MA correctly in a multi-instrument and multi-timeframe strategy that uses 1 futures contract (i.e. 3 months) or less of tick data at a time?

Thanks,

David
dbw451

[NinjaTrader_JoshP]

Unfortunately not. You would need more data.

[dbw451]

I'd love to load more tick data, I have many years worth. However, for some contracts I cannot even load a full contract at once in a strategy because they are very large and once NinjaTrader reaches about 1.3 GB of memory usage (as per the Windows Task Manager), I get 'out of memory' errors which requires shutting down NinjaTrader. I deal with that issue by back-testing one month at a time.

I would think there should be a way for secondary data used in strategies for other instruments or timeframes to be specified independently of the back-tester input parameters. There are times when the lookback periods required by an indicator is a significant magnitude in scale different than the time period desired for a back-test. It would be great if when adding a secondary data series if either the Start Date or the number of bars could be specified for the new data series. For example:

Add("^SP500", PeriodType.Day, 1, 300);

would create a secondary data series that had 300 bars of daily data available to it for calculation of indicators. Each data series is already managed separately via BarsInProgress and the BarsArray[], so I would imagine this might not be a difficult enhancement. But then again, I can only guess at the underlying complexity of managing multiple instruments and timeframes.

Regards,

David
dbw451

[NinjaTrader_JoshP]

You need 20 bars of data from all input series before your strategy will start running. Any indicator you run will run when the strategy starts. This is easy for you to confirm simply by running a strategy that just prints from OnBarUpdate() notifying you as to which BarsInProgress is triggering. You will find the daily will just keep triggering when higher granularity time frames are missing.

[dbw451]

In my case, the highest granularity is the daily and my primary input series is a much smaller tick based series. The daily series only fires off once each time the day changes. The tick based series fires off 50+ times each day.

While this information is interesting, it's not really this thread's issue. The issue is the requirement that the time length of all input series (i.e. the Start Date and the End Date) in a strategy be the same. I am unable to calculate a long term daily indicator on a secondary input series without expanding the time length of the primary data series loaded in a strategy. The two problems with expanding the primary data series are:

1. Tick data is very large and NinjaTrader runs out of memory when large tick data sets are loaded.
2. Most historical data sources for futures tick data provide the data in 3 month contracts.

In my previous post, I made an enhancement suggestion that could enable NinjaTrader users to calculate long term indicators from a strategy that is based on a much smaller timeframe. A 200 day moving average is a very common indicator and is referenced by day-traders that use all sorts of intraday timeframes. There should be a way to calculate a 200 day moving average from historical daily data in a strategy without loading 200 days of tick data of which 170 days will not be used for anything. My enhancement suggestion could be nonsense because of ignorance, but hopefully the requirement is understood.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

The issue is it does not make sense to process your daily data when your tick data is not present. It may not be an issue for your scenario, but in general most strategies calculate based off of all time frames used and as such the logic only makes sense when it is calculated from all the time frames. When you start the strategy when only one time frame is present it completely throws off the logic.

Also, I guess I do not understand why you cannot access your SMA 200 when you need it on the tick data. Just do SMA(BarsArray[1], 200)[0] and it will give you the SMA of the daily data as determined when your tick data starts "flowing".

[dbw451]

This thread started with almost exactly what's being suggested. In the first post of this thread, I posted the code:

protected override void OnBarUpdate()
{
if (BarsInProgress == 1)
{
daySMA200 = SMA(Close,200)[0];
....
}

which is essentially the exact same thing as using:

SMA(BarsArray[1], 200)[0]

I cannot access the SMA 200 data because the code SMA(BarsArray[1], 200)[0] does not calculate the correct 200 day SMA values. Here is a specific example:

ES 500 tick data series
From Date: 01/29/08
To Date: 02/29/08

For February 29, 2008:
SMA(200) on the ^SP500 daily chart = 1469.80 (correct value)
SMA(BarsArray[1], 200)[0] in strategy = 1356 (incorrect value)

Attached is a snapshot the daily ^SP500 chart.

From this thread, I thought I understood that a 200 day SMA would not calculate correctly unless all the input series had 200 days of data loaded in the strategy.

I guess my mindset is different than 'most'. I don't think of fundemental market information like an SP500 200 day SMA, weekly T-Bill rates, T-Bond yields, CPI numbers, etc. as indicators calculated from my trading charts. They are long term market data used for biasing. I think of my trading chart data (i.e. tick data) as being totally different.

I simply want to reference long term bias numbers in my small timeframe strategies.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

You need to compare the correct values. Consider this. The daily bars only update once per day. Chronologically this update happens at the end of the day.

For simplicity sake let us consider a 1 min and a 1 day series. Tick series would make no difference.

12/29/2008 4:00 PM 1min
12/29/2008 4:00 PM 1day
12/30/2008 9:30 AM 1min
12/30/2008 9:31 AM 1min
12/30/2008 9:32 AM 1min
....
12/30/2008 3:59 PM 1min
12/30/2008 4:00 PM 1min
12/30/2008 4:00 PM 1day

That is the chronological order in which bars update. Now consider the value when you try to do SMA(BarsArray[1], 200)[0] on 12/30. The daily chart has not updated because it is not the end of the day. You do not know the 12/30 SMA value. Instead when you access it you know the most recent daily update value and that is the 12/29's SMA. This is how it works and that is what happens from a chronological standpoint during backtesting. Since you cannot update the 1 day time frame intrabar there is no way to get 12/30 from a 1min time frame.

[dbw451]

Thanks Josh. I understand how NinjaTrader references the data values between different timeframes in a strategy. The help file on this subject is well written and very clear. Thus, I understand that intraday data on 12/30 would be referencing the daily 200 SMA calculated at the end of 12/29.

A 200 day SMA changes very slowly. Referencing the chart I attached in my previous post, you can eaily see that the 200 day SMA is not changing more than +- 2 ES points a day. However the calculated SMA(BarsIndex[1],200)[0] in my strategy example is over 113 ES points different than the correct 200 day SMA value. That difference is much more than (using your example) referencing the 12/29 200 SMA value intraday on 12/30.

I think that the SMA(BarsIndex[1],200)[0] function in my strategy is not looking at 200 days of data. I think it is looking at the number of days since the tick data starts flowing. If you look at the SMA indicator code, it does not wait for the input number of 'periods' to elapse before it starts calculating values. The SMA indicator starts calculaing the simple moving average at CurrentBar == 1. Thus at 30 days, the SMA value is a 30 day moving average and at 60 days, it is a 60 day moving average.

I added a 30 day SMA indicator on the daily SP500 chart to approximate the one month time period I use in my strategy. The 30 day SMA value on Feb 29 (the date referenced in my previous posted chart) is 1351. While this is not exactly the 1356 calculated by SMA(BarsIndex[1],200)[0] in my example, it's close enough to add some validity to my hypothesis that the SMA(BarsIndex[1],200)[0] function is not looking at 200 daily bars in my strategy. If it were, the calculated SMA would be closer to 1470 which is almost 9% larger than 1351.

If I'm understanding you correctly, you're telling me that SMA(BarsIndex[1],200)[0] should give me the correct 200 day SMA value, just one day offsetted. As detailed above, that is not the behavior that that I see.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

I cannot comment on your exact code, but on my end it works perfectly. Please look at this screenshot. It works as expected.



On 12/30 I get the SMA(200) value of 12/29. The value is accurate. I checked several different days and they are all correct.

In the screenshot you can clearly see the switch from BarsInProgress from 0 to 1. The update of the daily bar series happens on 1 and then you can see all subsequent calls to SMA(BarsArray[1], 200)[0] gets ammended as well.

Find the strategy attached. I suspect you are still comparing apples to oranges. The reference I have put together is as simple as it gets and it works as expected, suggesting that there is no problem with NT's handling of this scenario.

[dbw451]

Now I know why I've been having problems communicating. The strategy should be multi-instrument and multi-timeframe. The daily SP500 data is a different and separate instrument than the ES contract tick data. Each ES contract instrument has 3 months of tick data. The daily SP500 data has 50+ years of history.

I changed the Add line in your strategy include the SP500 instrument:

Add("^SP500", PeriodType.Day, 1);

The results I get are similar to my original strategy (see attachment).

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

I reran with multi-instrument, multi-time frame. No problem.

Series 0 = 1min AAPL
Series 1 = 1day ^SP500


The chart on the left is daily chart of ^SP500. The print out is from the strategy as ran from the Strategy Analyzer on 1min AAPL. You can see it prints the value of SMA(BarsArray[1], 200)[0] perfectly. On 12/18 you get the proper 12/17 SMA 200 of ^SP500.

On your chart you have a bunch of extra things. I do not know what those are and for simplicity sake please just try a very basic ^SP500 daily chart. Throw SMA 200 on it and then compare. Everything matches up on my end.

[dbw451]

Josh,

I get similar results to my previous post (see attachment). I now suspect that the issue could be either:

1. the amount of tick data (i.e. 3 months) in the ES futures contacts.
2. using a tick timeframe instead of a minute based timeframe.

Does your AAPL have more than 200 days of data? Maybe you could replicate the issue if you create an ES 12-08 instrument and download tick data from the Ninja historical data servers. Then back test using a 500 tick timeframe of ES 12-08 contract.

Regards,

David
dbw451

[NinjaTrader_JoshP]

Backtested on ES 12-08 tick 500. My tick chart only goes back a month.

0 12/18/2008 7:53:30 PM 1203.56765 891.5
0 12/18/2008 11:59:17 PM 1203.56765 889.25
1 12/18/2008 12:00:00 AM 1201.47235 885.28
0 12/19/2008 1:06:44 AM 1201.47235 891.5
0 12/19/2008 2:19:42 AM 1201.47235 889
0 12/19/2008 3:33:34 AM 1201.47235 887.5
0 12/19/2008 4:11:13 AM 1201.47235 886.75
0 12/19/2008 4:42:19 AM 1201.47235 887.75
0 12/19/2008 4:56:00 AM 1201.47235 889

Line in red is the SP500 updating on daily chart. You can see the SMA updated to 1201. All subsequent calls from 500tick ES acquires the correct value of 1201.47 as determined from 12/18. A check on the daily chart of ^SP500 and a SMA(200) confirms the value at 12/18 was 1201.47.

Also, in your screenshot you are comparing the wrong thing. At 12/05 your prints should return you the 12/04 SMA(200) value, not the 12/05 value. The 12/05 value is not known till after 12/05 is over.

As far as why yours is off and mine is not I am out of ideas. Please ensure you are on 6.5.1000.8. If you are please try a fresh reinstall. Do not forget to clear your browser cache when you redownload NinjaTrader.

[dbw451]

Josh,

I'm at a loss of ideas also. I have Ninja v6.5.10000.8. To recap:

1. We have the same Ninja version.
2. We have the same strategy.
3. We have ES tick data from the same source.
4. The daily ^SP500 data might be sourced differently (mine's from OpenTick and Yahoo), but I cannot see how the daily data would make a difference.

I did notice something interesting with my strategy Initialize() procedures, but I'll start a separate thread because I'm not sure if it relates to this daily SMA calculation issue or not.

Regards,

David
dbw451