Data minutes shifted 1 day and weekend issue

[WhoKnows]

I am trying out NT to see if I can port a TDA StratetgyDesk strategy to NT for use with IB.

Have spend a week on the manual and read part of forum. I hope I did not miss my problem when searching.

As I have a multi-time frame strategy (day and 15 min) I started optimistically with the examples and tried to code my strategy. The code run, but results were awful. So I went back to square one building the strategy from scratch part for part.

After analyzing some simplified strategy back tests I however found strange things in the results and charts. Apparently NT sees my TDA day data with US time stamp and the minute data with my local time. (I am based in Brisbane Australia , therefore US market time Friday is for me 12.30AM-07.00AM Saturday local time.

Before going further:
-I did synchronize my PC to windows time server
- I deleted the historic data before every run

TDA feed
AUY type:day - chart looks correct, the market data of Friday US are shown on 28th. .... so the daily data are shown on US time frame, not my local time. I would like this, but I think its not supposed to work like that in NT.
see AUY 12_11_2008 - 28_11_2008 (Daily).jpg
Running again with exclude weekend is true: the chart looks the same.

AUY type:390 minutes 12.30AM-07.00AM exclude weekend false -> the 390 minute bar is shown at 11/29.
How can you run a multi-time frame comparison when minute and day bars are shifted 1 day??
see AUY 29_11_2008 (390 Min) no weekend false.jpg

AUY type: 390 minutes 12.30AM-07.00AM exclude weekend true -> it gets even worse, apparently all US Friday data are ignored and last bar is 11/27.
see AUY 27_11_2008 (390 Min) exclude weekend true.jpg


OK, now the same on IB data feed
IB AUY type:day - chart looks correct, but the market data of friday US are shown now on the 29th Saturday .... so the daily data are shown on my local time. So TDA Daily is different from IB daily.
see AUY 13_11_2008 - 29_11_2008 (Daily) IB
Exclude weekend sure enough removes all Saturday bars (which were the Friday US market data).
see AUY 7_11_2008 - 27_11_2008 (Daily) IB exclude weekend true

IB AUY type:390 minutes 12.30AM-07.00AM exclude weekend false - the 390 minute bar is shown at 11/29.
see AUY 29_11_2008 (390 Min) IB
Exclude weekend sure enough removes all Saturday bars from the chart (which are the Friday session US market data).

So:
1. IB is at least consistent with the daily and 390min bars on the local time.
But the use of 'exclude weekend data' removes the LOCAL time Saturday data => the US Friday data are gone.

2. TDA daily data show up on the US market date and 390min data on the local date.
The 'exclude weekend' option leaves the daily data intact (US timed) but kills the local time Saturday data (so the US Friday minute data gone).

3. If I don't delete the daily historical data, disconnect from IB .. connect to TDA and reload data or run backtest I get some protesting about data errors. But under the backtest chart tab I do get a chart.... with two almost indentical bars (one on the 28th and one on 29th).
see AUY 13_11_2008 - 29_11_2008 (Daily) 2 same bars


As I was using the TDA feed in the weekend (when IB is down) and during the week both IB and TDA (do circumvent data access limits with IB), I guess I will have quite some data errors in my instruments. Maybe this is also a reason why my multi-time frame (day and minutes on TDA feed) strategy was not working.

I am quite confused, obviously I am rather not forced to needing a third data vendor. I would assume NT would work of the shelf with TDA and IB. Any suggestion ?????
Is this mess because I am not in the US time/date band??

BTW:
I must say that I really don't understand why you don't leave things in US time. With the daylight savings shifts the offset needed to define US markets is very complicated. I hope that in backtesting NT knows how to handle these changing offsets (or are data lost when backtesting over daylight savings shifts?).
Also I suggest:
- pls. drop the use of AM and PM.... a 24.00 notation is much clearer.
- pls. get the mm/dd/yy (chart crosshairs) to dd/mm/yy

RGDS Peter
got to 5 attachments ... if you want the other 2 too, pls let me know

[NinjaTrader_Jason]

Hello Peter,

NinjaTrader uses your PC clock to display the times in the charts and time stamp price bars. More information can be found at the link below.

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

I suggest to set your PC clock and time zone setting to the local exchange time. Check if the issue persist.

As per the suggestions, I will forward the suggestion to change the time notation to our development team and ask them if they can add this to the list of future considerations for the software.

You can set the date to be displayed as dd/MM/yyyy in the chart properties menu (double click chart).

[WhoKnows]

Thanks,
ok, found the settings for the crosshairs ... that was the easy part (still don't understand why the default on that one is different).

I'll change the date to US Eastern for testing, but it is not really something I like to do.
1. The time stamps of other programs and files I use get confused
2. I have programs that will just stop if time/date is changed (e.g. statistical) after being installed (even if I change the date back). Think its to prevent them working outside license. One will need a complete reinstall.
For this 'the set to US solution' I'll need a permanent separate PC to run NT.

OK, after time/date reset:

TDA feed: 390 minute bars ok and on US time.
TDA feed: daily .... the bars are at correct US date, butt the 26th is last bar, don't know why Friday 28th does not show.

IB feed:
Both 390 and daily chart ok.

Also another question:
- in the daily analysis and chart you can not define the market hours ... so the daily always includes the 8.30 -9.30 and after market period. Is this true or is there a setting I missed??
If so, the 9.30 - 16.00 setting in the 390 min chart would be by definition different when compared to daily?

So what would be the best simulation of the day via minutes?



BTW It would be nice to mention this time shift problem for Non US users in the manual. It would have saved me a days.
And also a warning is warranted that that non US users should NOT use the 'exclude weekend is true' option as they will exclude the US Friday data in their testing.

I really hope NT7 will address this.

[NinjaTrader_Jason]

NinjaTrader uses the US date format (mm/dd/yyyy) by default.

Please right click in the TDA daily chart and select "Reload Historical Data" and check if the Friday bar is displayed. If not, please right click in the chart and select Format Data Series window. Make sure the current date is entered next to "Last date".

Correct, daily bars include 24 hours of data.

I suggest to use an interval of 1440 minutes.

The link below in our Help Guide lists our charts use local PC time.

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

[WhoKnows]

I did read that part in the Help Guide you refer too, and understood that incoming ticks are stamped with the local computer time.

However I failed to understand from the manual text that this means that if you backtest and data are downloaded from the broker, (non US users) can get a time shift problem,

A more explicit warning might be more user friendly. Something like:
------------------------------------------------
- when connecting to different brokers, you may get conflicts in the historical data due to different method of date stamping. Especially when not in the relevant market time zoine, we urge you to evaluate the minute and daily charts to assess how NT interprets the broker data.
- also pls. realize that when the local PC time is outside the US date line to NOT use the 'exclude weekend is true' option. Depending on the time offset part or all of the US Friday data will be excluded as NT will see these data with a Saturday datestamp (and hence exclude these as they fall within the weekend).

Also consider that the daily bars will include the pre-and after-market data. If you want to emulate the daily bars with minute data, is is advised to use 1440 minutes.
Using 390 minutes on the minute setting combined with 09.30-16.00 US Eastern time will give you the emulated daily bar of the regular stock market data.
--------------------------------------------------

Such information added to the manual would IMHO greatly help new users.

[NinjaTrader_Jason]

Hello Peter,

Thank you for your feedback. I will forward it to Management.

[WhoKnows]

I thought the PC clock to US time had solved the problems, but unfortunately not completely.

I had the 1440 minutes in a strategy and got very strange results over the weekend. After looking in detail at the charts, I found that setting the period to 1440 minutes would shift the day numbering.
All data were shifted one day, e.g. Friday data to the Saturdays.
This happened only with the 1440 minute bars, not at settings with 390 or 1439 minutes.

Am I correct that IB and TDA don't close the Friday bar on Friday ningt??? Or is there another issue I don't understand.

See both attached charts of TDA and IB feeds. The PC Clock said Sunday 14 December, about 10:00 PM your time.


PS the Daily charts looked ok with respect to the dates, but the indicators for the last Friday bar are not yet calculated. Does this mean that NT thinks that this last bar is not yet closed, although it is already Sunday?
Maybe to rephrase it: how is it exectly determined that a bar is finished? Does NT look at the PC clock, or do the bars remain open until a tick comes in with a new date stamp?

I am really trying to understand the detail here, as this would influence the testing of mixed timeframe strategies in the weekend.

If NT does not yet calculate the indicators of the last bar, is this then what we define in the strategy as the current bar [0]. And is the last bar for which NT can calculat the indicator values the strategy bar[1]?

[NinjaTrader_Jason]

I am not sure why you use an interval of 1440 minutes, since both IB and TD AMERITRADE support historical daily data. Customers usually use an 1440 minute interval when they do not have access to historical daily data.

NinjaTrader time stamps a price bar at the close. More information can be found at the link below.

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

As per the indicators, please set 'Calculate to bar close' to False. Indicator values will be calculated on each incoming tick instead of the close of a bar.

[WhoKnows]

I used the 1440 minutes to prevent the daily data to be offset compared to the minute data (issue due to timedifference, see start of thread). You suggested in mail 4 below to use the 1440 as a good daily approximation. I have seen the 1440 minutes suggestion also in other posts.

I did read the page you referred to before, but this page is (for me) not unambiguous about the historic data. I guess IB and TDA send unstamped minute and daily historic data. I might have been unclear, but your answer did not address my main questions. I'll try again.

Ok, NT timestamps the bars with the local clock time. How then is the Friday data bar still not closed at Sunday. If NT works as the webpage states, the bar of Friday would close at 12.00 PM and indicator values for the 1440 minute bar would be calculated.
Also the indicator values of the daily charts are not yet calculated on Sunday. If NT stamps the data and uses the PC clock, it should know that Friday has passed. Hence I would expect that on Saturday the daily indicators (and value in the strategies) could be used.
Apparently this is not the case... why does NT not close the bar?

Next to the point of not closing the bar and calculating the indicators after the bar period is closed, I still don't understand why with both IB and TDA all 1440 minute simulated days are shifted 1 day in the future (see attachment previous post). Is this a bug of NT?



The 'Calculate to bar close' to False. will work in live situations, but not while backtesting strategies
I understood calculations at backtesting are always on the close of a bar with NT.
The issue of closing the bar and the possible effect on strategy backtesting is is part of the point I try to understand.
Suppose it is Saturday and I use the statement that daily EMA[0] > EMA[1]. In the charts the Friday bars are not closed and the indicator values for Friday are not yet calculated. I assume then that the most recent daily EMA[0] the strategy uses is then NOT Friday, but Thursday and most recent EMA[1] is Wednesday.


BTW I still don't understand NT doesn't allow for 'on the fly' (calculate on close=false) calculations in backtesting in those cases where bars are built by known data. E.g. hourly bars are built with progressive minute data, so an indicator could be calculated during the hour in a simulated realtime fashion. But for my the previous points are more important to understand.

[NinjaTrader_JoshP]

IB and TD are timestamped locally. NinjaTrader is tick driven and not time driven. As such you will not close a bar till you receive a new tick with a time that is outside the range of the prior bar. This is why your Friday bar is still "open".

[WhoKnows]

Would it be possible for a user to feed NT a tick e.g. an extra tick that duplicates the last official tick but just a second after the range of the prior bar.

I would see this as very useful for Friday night. Then in the weekend backtesting can be done including the last day and week.

I can't imaging such a duplicate tick will influence results, and if the user want he can always delete the last day data and reload.
Or is it even possible to send a timestamp only without price?

[NinjaTrader_Jason]

You can import historical data using the instructions at the link below.

http://www.ninjatrader-support.com/HelpGuideV6/Import.html