Missing historical data on multi timeframe chart

[pmn100]

Further to my thread here https://ninjatrader.com/support/foru...d.php?t=105120, particularly the second part concerning missing historical price data on session open, it happened again this morning for @GER30 cfd with FXCM.

Last night I loaded up NT8 and was sure to select 'reload historical data' for the @GER30 chart and then apply my strategies and enable them. On waking this morning I noticed the lack of orders, the same as last week. On closer inspection I noticed the D1 data bar for the day ending 19th October was missing. I use multi-timeframe strategies so have a M1 and D1 data series loaded to the chart and the D1 section is made as small as possible so didn't see that the bar was missing last night.

On performing another 'reload historical data' from the chart options this morning, the D1 candle appeared and the expected orders were submitted.

This only seems to be an issue on @GER30, I have other charts open for forex and gold cfd with FXCM and the D1 bars loaded fine last night. I could be wrong but it seems @GER30 won't load historical data correctly and fully unless the refresh is performed within trading hours. Thats the only difference I see between dax cfd and gold cfd and forex, the latter markets were open last night at the time of refreshing the historical data.

Should there be a problem loading historical data this way when out of market hours? If the data doesn't load correctly I'm going to have to try and program a data refresh from within the strategy the first minute of market open on a Monday, which I believe NT doesn't really support.

[NinjaTrader_PatrickG]

Please clarify - you saw this bar last night when you reloaded all historical data and then it was gone this morning? Or did you not check specifically?

What trading hours template is your @GER30 chart using? To check, right-click on the chart > Data Series > Trading Hours.

What time zone is NinjaTrader configured in? To check, go to Tools > Options > General > Time zone.

Is this easy to reproduce? In other words, what would I need to do to cause the same as you're experiencing on my end?

[pmn100]

Please clarify - you saw this bar last night when you reloaded all historical data and then it was gone this morning? Or did you not check specifically?

I didn't specificaly look for the bar last night, only this morning when my orders didn't submit did I take a closer look and see the bar was missing.

What trading hours template is your @GER30 chart using? To check, right-click on the chart > Data Series > Trading Hours.

"Use Instrument Settings"

The chart loads 07:00- 21:00.

What time zone is NinjaTrader configured in? To check, go to Tools > Options > General > Time zone.

UTC+00:00 London

Is this easy to reproduce? In other words, what would I need to do to cause the same as you're experiencing on my end?

If its something to do with opening the chart and reloading historical data outside of trading hours, you could wait for dax to close then open a chart with M1 and D1 data series on. Or it may be a weekend only thing where opening the chart Sunday evening fails to load the data properly. I will do some tests myself and see if its easily reproduced. I was just wondering in the mean time if this is a common issue with an easy fix but it appears not.

[NinjaTrader_PatrickG]

We are not received other reports of this specific issue, and I could not reproduce this in my testing so far.

I suspect you're correct that it requires the market to be closed if it is reproducible. I will be performing this test on my end, and I look forward to your reply once you've done some testing to see if you can reproduce the issue.

[pmn100]

Just logged onto another computer with NT8 to check a DAX chart out of hours. As you can see in the first image, on creating a new chart with M1 and D1 data series for past 100 days I found a daily bar missing for day ending 17th of October. M1 data appears to be intact.




After selecting 'reload historical data' the bar appears.




This isn't as much as an issue as my previous example as the bar appeared on reloading historical data, but it may be a clue that something is amiss. Its my understanding that all bars should be backfilled on opening a new chart.

[NinjaTrader_PatrickG]

I followed these steps and could not reproduce the issue:

• Set my PC clock to UTC+0 (London)
• Start NinjaTrader
• Create a 60m and 1d @GER30 chart
• Connect to FXCM
• look for missing daily data

This may indicate a corrupted cache or historical data on your end. As a test, I'd like you to delete this local information and see if the issue still occurs:

To delete the cache and locally stored historical data, follow the steps below:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache', 'day', 'minute', and 'tick'.
• DO NOT delete any other folders or files
• Restart NinjaTrader and test.

[pmn100]

I will try this although I have done this on past occasions anyway. If it IS a corrupt cache though its on two different machines and installations! But now I know my problems with @GER30 seem to be missing historical data I know what to look for of a Sunday evening when I start my strats. I will post if I run into more of the same problems.

[pmn100]

Another Sunday night and another Monday morning with problems on @GER30 with FXCM.

Last night (out of market hours for @GER30) I load up the platform having deleted the cache data suggested last week. My charts loaded up and even after reloading historical data I got the following again with a missing daily candle for Thursday 26th:



So I started up another computer with the same version of NT8 loaded to investigate this further. I deleted all historical data from this machine for @GER30, then I started NT and loaded up a new @GER30 daily chart connected to FXCM. I got the following:



Everything loaded correctly. I then clicked 'Reload Historical Data' and this happened:



See the difference? The Thursday 26th bar disappears!

So with this in mind I go back to my other computer where I run the live strategies, I delete all the @ger30 data again from My Documents, load up my stratagy chart for @ger30 and get all the data, and there I leave it making sure I don't select 'Reload Historical Data' because that would make the Thursday daily bar disappear. My stratagies are green and I leave it running overnight. On checking the chart this morning however, even though I'm getting realtime data loading for @ger30, the strategies are not recognising the bar updates. Orders that should have been placed haven't been placed, and printed text hasn't printed which should have been.

So I 'reload historical data' and of course the orders get placed and everything is running as it should.

I have been going through this same thing every Monday morning since starting testing @ger30 strategies about a month ago, opening my charts and finding the strategies for @ger30 are just not working. And its not because the strategies themselves are faulty, something is up with NT not recognising the live @ger30 feed from FXCM when it opens at 7am (London time) but when the chart and strategies have been loaded on Sunday night out of trading hours. I will add that I run almost identical strategies on @XAUUSD and several forex pairs without issue. Perhaps because these markets are open Sunday night when I start up NT and load up the charts/strategies?

But NT clearly has issues loading historical data correctly when out of market hours for @ger30 with fxcm. For example, why would a daily bar disappear when selecting 'reload historical data'? Something very wrong here.

[NinjaTrader_PatrickG]

Are you using NinjaTrader or FXCM historical data servers? To check, go to Connections > Configure > Choose your FXCM connection > Is 'NinjaTrader historical data server' enabled or disabled? Check both of your PCs.

Does the issue occur if you reverse the current setting here, disconnect, then reconnect?

[pmn100]

Are you using NinjaTrader or FXCM historical data servers? To check, go to Connections > Configure > Choose your FXCM connection > Is 'NinjaTrader historical data server' enabled or disabled? Check both of your PCs.

Does the issue occur if you reverse the current setting here, disconnect, then reconnect?

Both have the option disabled (unchecked). I will have to wait until out of market hours to check with NT historical data because as I've stated, everything refreshes fine when in market hours. It may even be a Sunday night only thing and work fine in the weekdays when out of hours. I will check.

[pmn100]

Just tried loading the @ger30 daily chart out of hours still with the FXCM feed with NT historical unchecked and it is loading correctly and staying correct on reloading historical data, so this issue only occurs prior to Monday's open. I will therefore have to wait another week before trying NTs historical servers to test if the Thursday daily bar disappears again.

[NinjaTrader_PatrickG]

It is a known on our end that the FXCM historical servers have issues with loading daily bars over the weekend in NinjaTrader 7, however, this would be the first report of this occurring in NinjaTrader 8.

Please clarify - do you remain connected 24 hours a day? Or are you shutting down/disconnecting NinjaTrader each day?

I'll be testing on my end this weekend as well. Here are the steps I'll be taking:

• On Sunday afternoon/evening, connect to FXCM (use FXCM historical servers)
• Load a daily chart of @GER30
• Observe the chart for missing bars
• Reload all historical data
• Observe the chart for missing bars

Would you agree that this is a valid test? If not, what would you suggest to do to replicate the behavior?

[pmn100]

It is a known on our end that the FXCM historical servers have issues with loading daily bars over the weekend in NinjaTrader 7, however, this would be the first report of this occurring in NinjaTrader 8.

Please clarify - do you remain connected 24 hours a day? Or are you shutting down/disconnecting NinjaTrader each day?

I'll be testing on my end this weekend as well. Here are the steps I'll be taking:

• On Sunday afternoon/evening, connect to FXCM (use FXCM historical servers)
• Load a daily chart of @GER30
• Observe the chart for missing bars
• Reload all historical data
• Observe the chart for missing bars

Would you agree that this is a valid test? If not, what would you suggest to do to replicate the behavior?

Thanks for getting back to me. I used to stay connected 24 hours a day Monday to Friday but I started getting lock ups so more recently I've been restarting NT at the end of each day when my strategies are flat.

The test you've specified is exactly what I did Sunday night. The only difference would be that I deleted all db data for @ger30 too. This allowed the daily bar to initially appear when loading up a chart for the first time. Then on reload historical data the bar disappeared and remained absent.

I also observed the following after deleting all db data in My Documents for @ger30:

• load a daily chart for the last 10 days (not 365 default)
• notice the data is complete
• select reload historical data, the Thursday daily bar disappears
• select reload historical data again, the Thursday bar remains absent
• select data series on the chart and adjust to 100 days instead of 10
• the missing Thursday daily bar re-appears
• select reload historical data, the Thursday daily bar disappears

You can then go back into data series and adjust to 365 day load and the missing Thursday bar will re-appear again, only to disappear when selecting reload historical data. So a fresh look up of historical data reloads the database fine, its on the reload that something goes awry.

But its also worth noting that even when I had the daily bar present (by deleting the db data and loading up a chart just once and avoiding reloading) and my strategies green, NT still didn't register the bar updates when @ger30 opened at 7am London time. I have a feeling using NT historical servers on my live client will probably solve the issue.

[pmn100]

This is happening right now if you want to check. And its not only @GER30, its also happening on @XAUUSD and GBPUSD and EURUSD and probably more. All are missing daily data when loading charts out of trading hours. But I will just show screen shots for @GER30.

First load of the daily chart and I get this:



Looks okay on first glance but its actually missing the 2nd November bar, which as in prior examples is the most recent Thursday bar. If I delete all db data for @ger30 and load the chart again I get the full chart:



Then on selecting 'reload historical data' the Thursday bar disappears again and I get the following:



Using NT historical servers for FXCM connection does solve the problem but I would prefer not to do this as I'd have to make adjustments in my strategy codes for identifying previous days highs/lows due to the fact FXCM fills gaps between bars in historical data where as NT doesn't.

[pmn100]

This is still occuring. Was you able to verify?

FXCM cfds missing D1 data Sunday night/Monday morning. The D1 bar that is missing is always the previous Thursday bar. Intraday data for Thursday is present. The D1 bar for Thursday only appears if I reload historical data within market hours for that instrument. For @XAUUSD this means after 11pm Sunday night (London time). For @GER30 this means 7am Monday morning (London time).

Also, and maybe because of the missing D1 data, strategies on these instruments do not activate until I reload historical data within market hours.