Large gaps in data - NinjaTrader FXCM NT8

[Jellybean]

Hi

I'm trying to get Forex charts to work with a demo NinjaTrader FXCM account. However, I am struggling.

Configuring the connection as stated in the Connection Guide for FXCM gives very large gaps in daily and longer time frame charts.

Configuring the connection with “NinjaTrader Historical data” from unchecked works for a while with one chart, but as soon as I have two or more charts I get errors when loading. However, the charts seem to load OK (although slowly), which is unexpected.

I've attached the steps I took to produce this, some screenshots and the most recent log & trace files.

I'm using NT 8.0.2.0.

I look forward to your advice.

Regards
Tony

[FMtrader]

Hi

I'm trying to get Forex charts to work with a demo NinjaTrader FXCM account. However, I am struggling.

Configuring the connection as stated in the Connection Guide for FXCM gives very large gaps in daily and longer time frame charts.

Configuring the connection with “NinjaTrader Historical data” from unchecked works for a while with one chart, but as soon as I have two or more charts I get errors when loading. However, the charts seem to load OK (although slowly), which is unexpected.

I've attached the steps I took to produce this, some screenshots and the most recent log & trace files.

I'm using NT 8.0.2.0.

I look forward to your advice.

Regards
Tony

I have seen similar gaps in recent weeks when using historical data from NinjaTraders NT8 FOREX servers. When I disabled the connection to NinjaTrader's historical server and instead pulled history solely from FXCM's servers the gaps disappeared. My conclusion at the time was that Ninja's NT8 FOREX database was unreliable as it had large data gaps.

Using just FXCM data seems to work OK when you don't require too much history but it certainly doesn't seem to like large historical requests or requests for tick based data required to build non time based charts. I have also noticed that FXCM's servers shut down early in a weekend so one can't download data and do any analysis until they come back online.

I have noticed the time delay in getting larger data requests fulfilled by the FXCM servers and I wonder if they have some limit in the data request size. When you called up 5000 bars of a 240 min chart you are effectively asking for asking for 3 to 4 years of minute based history which may be tripping an FXCM data request limit?

One thing you may also see if you look closely is that NinjaTrader's historical data on their NT8 FOREX server is slightly different to FXCM's. The OHLC values can typically vary by up to a pip or so but I have occasionally seen up to 3 pips or more.

[NinjaTrader_JessicaP]

Hello Jellybean,

There was a recent historical data service outage for the following instruments :

• $USOIL
• $AUS200_CFDAUD
• $BUND_CFDEUR
• $JPN225_CFDJPY
• $SPX500
• $US30
• $COPPER
• $NAS100
• $NGAS
• $GER30_CFDEUR

If you trade any of the above instruments, please use FXCM's historical data servers. If this was for another instrument, please let us know when the outage occurred and for which instrument so we may investigate further.

[Jellybean]

Hi Jessica

I don't trade any of the instruments you mentioned - I'm assessing NinjaTrader broker for spot forex trading, which uses a data connection to FXCM, I think. The instruments I tested were all spot forex:
AUDCAD, AUDCHF, AUDJPY, AUDUSD, CADCHF, CADJPY, CHFJPY, CHFJPY, EURAUD, EURCAD, EURCHF, EURGBP, EURJPY, EURNOK, EURNZD, EURSEK, EURUSD, GBPAUD, GBPCAD, GBPCHF, GBPJPY, GBPNZD, GBPUSD, NZDCAD, NZDCHF, NZDJPY, NZDUSD, USDCAD, USDCHF, USDJPY, USDTRY, USDZAR.
All show the same gaps when using NinjaTrader historical server.

At least these gaps appear on all daily & longer time frame charts:
14 Jun 2015 - 4 Jan 2016
2 Jan 2015 - 4 May 2015
23 Dec 2012 - 16 Dec 2013
7 Oct 2012 - 17 Dec 2012
16 Sep 2012 - 1 Oct 2012
20 Feb 2011 - 10 Sep 2012

I forgot to mention earlier that my time zone is NZT, which has been an issue in the past, although I doubt it is relevant for this (but you never know).

Please advise what I should do next.

Regards
Tony

[NinjaTrader_JessicaP]

Thank you Jellybean, all of these outages have been confirmed and reported. For now I would advise connecting to the FXCM HDS (Historical Data Server) , downloading and exporting the missing historical data, reconnecting through Ninja, and importing this data as needed.

[Jellybean]

Hi Jessica

Thanks for your quick response.

I have just tried using Kinetick EOD to load historical daily data, then connecting to Ninja FXCM. However, the same problems persist. I think whenever I change instrument or time frame, historical data is being reloaded either from FXCM or NinjaTrader depending on the connection setting, which recreates the issue.

Loading Kinetick EOD data cleans up all the daily, weekly & monthly charts. I then...
Disconnect Kinetick EOD.
Connect Ninja FXCM (without Ninja historical data).
First load is OK.
Change instrument a few times gives the "Downloading historical data - invalid instrument" error, as before.

Then I tried....
Loading Kinetick EOD data, which cleaned up all the daily, weekly & monthly charts.
Disconnect Kinetick EOD.
Connect Ninja FXCM (with Ninja historical data).
The gaps in daily and longer time frame charts appear, as before.

Unless I'm missing something, it seems to me that the cause is simply data missing from the NinjaTrader historical data server. Wouldn't it make more sense for NinjaTrader to do what you suggested I do, i.e. load the missing historical data?

Please advise if I can do anything more.

Regards
Tony

[NinjaTrader_JessicaP]

To clarify, we have a reduced staff today due to the New Year's holiday. The staff responsible for maintaining the HDS have been notified of these gaps. The steps I have given you are only a temporary work-around so that you can make the historical data you require from the slower to load FXCM HDS available to you while loading as much data as possible from the faster to load NinjaTrader HDS. This work-around is accomplished by visiting Tools -> Export -> Historical Data and Tools -> Import -> Historical Data. Storing this data in text files on your end will make it available regardless which HDS you are using.

When our staff returns from the holiday they will be able to respond at the server level.

[Jellybean]

Hi Jessica.

Thank you for the explanation. I understand the point of your work-around process now. It's not practical to scan charts importing data for each instrument as you go, though.

Also, please note that using FXCM for historical data is not just slower, it causes errors that make is unusable for me.

I await the return of your HDS staff.

Regards
Tony

[The_Wiz]

I have the same problem, with the currencies but also for GC (gold futures). I have a gap in september 2016. There is data the 29-08-2016 and then there is no data until 30-09-2016.

Is there an update on the issue?

[FMtrader]

I have the same problem, with the currencies but also for GC (gold futures). I have a gap in september 2016. There is data the 29-08-2016 and then there is no data until 30-09-2016.

Is there an update on the issue?

Agree with comment on GC data. In fact it appears as if the data supplied for roughly the whole month of August 2016 is extremely low volume and is not coming from the correct contract for that month. The data then disappears completely for most of September.

The NinjaTrader 7 historical data server is showing exactly the same problem!

[NinjaTrader_JessicaP]

Thank you all for your continued interest. I have reached out to the teams overseeing the HDS, and these reports are still being investigated. I would like to encourage you for the time being to rely on the FXCM HDS or saved historical data for these time periods. I will update this thread once more when we have more information.

[Jellybean]

Hi

This is just an update for others struggling with this because I realised I was doing something that made the situation worse.

I trade mid- to long-term so I have use monthly & weekly charts for my higher time frame analysis. I had set numbers of bars in the charts that resulted in daily data required earlier than what I had stored locally. This situation results in NT attempting to download all data required for the chart from the historical data server. (See the conditions for downloading in the "Help Guide | Operations | Historical Data | Loading Historical Data".)

As a result, NT kept on overwriting my good data with the 'gappy' data. I improved the situation dramatically, by setting a data series start after my locally stored data starts.

Jessica, I cannot use the FXCM historical data - it fails too often. (See FMTrader's reply to my first post.)

Cheers
Tony

[NinjaTrader_JessicaP]

Thank you for this additional information Jellybean. I have followed up with the teams responsible for maintaining the servers. The first NT8 server is under maintenance. If you have the latest copy of NinjaTrader, you should see this URL in your logs, emphasis mine

hds-us-002.ninjatrader.com

If you see 001 in your logs, please let us know.

I will be following up on this thread after we have done some more extensive testing over the weekend. When our maintenance is done, if you are still monitoring this thread, the gaps covering the following periods will then be closed :

o 13 JUN 2015 - 03 JAN 2016
o 02 JAN 2015 - 03 MAY 2015
o 22 DEC 2012 - 15 DEC 2013
o 06 OCT 2012 - 16 DEC 2012
o 15 SEP 2012 - 30 SEP 2012
o 12 FEB 2012 - 09 SEP 2012

I appreciate your bringing this to our attention, and I look forward to letting you know when our server maintenance is complete.