VERY slow loading of minute data for FXCM's CFD products

[Trader_55]

Try the following:

1. Start up an instance of NinjaTrader 7.
2. Connect to FXCM using Ninja's historical data.
3. Open up a chart with 1000 bars of 60 second bar data for $GER30_CFDEUR.
4. Open up a chart with 1000 bars of 1 minute bar data for $GER30_CFDEUR.

You should notice that the 1 minute bars take a LOT longer to load than the 60 second bars (which are tick-based). There is something wrong with this, it should not take that long to load historical data, and since this is your (Ninja's) historical server working with your software in theory you should be able to fix it.

NT support, please confirm that you are seeing this same condition, and please open a support issue for the dev team to deal with it if that is the case.

[NinjaTrader_PatrickG]

I would like to confirm which historical data servers you are connected to.

Can you take a screenshot of your account connection settings? Here is an example of my connection:

http://screencast.com/t/XJ4yjdV7

[Trader_55]

I would like to confirm which historical data servers you are connected to.

Can you take a screenshot of your account connection settings? Here is an example of my connection:

http://screencast.com/t/XJ4yjdV7

I have the same connection settings.

Interestingly, I am only having this issue for $US30 and $GER30_CFD, which are the only instruments I have actually traded. Can you try these instruments specifically and let me know how fast it seems?

If you are getting the same slow problems with minute data on these two, then it is something specific about those instruments. On the other hand, if you are getting a fast response then it must be something to do with the fact that I have many trades logged in my database, and for some reason the code is reacting to that.

If that is the case, how can I clear the database of all my executions without resetting literally everything in the DB? Please do the test and let me know what your results are.

Just an update: a 5 minute chart of GER30 takes at least a **minute** to load, and incidentally takes the same amount of time to load from the hard drive, even when there is no connection. A 300 second chart (5 minutes of time) loads within a couple of seconds.

Whatever the issue is, there is something VERY wrong with the minute-based code connecting to FXCM, at least for these instruments, possibly when there are many executions in the database. I would pass this on to your developers and see if they can track it down and debug it.

[NinjaTrader_PatrickG]

I do not experience the same delay when I test the same instruments with my FXCM demo account. The 60 second chart takes a little longer to load but that is expected as tick data is larger in file size.

Deleting your cache and your current historical data will cause the connection to re-download the historical data.

To delete your cache:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 7 > DB folder. Delete the cache, tick, minute, and day folders

Please restart NinjaTrader and test.

[Trader_55]

I do not experience the same delay when I test the same instruments with my FXCM demo account. The 60 second chart takes a little longer to load but that is expected as tick data is larger in file size.

Deleting your cache and your current historical data will cause the connection to re-download the historical data.

To delete your cache:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 7 > DB folder. Delete the cache, tick, minute, and day folders

Please restart NinjaTrader and test.

Interestingly, this worked. Now when I connect and try to open a chart connecting to FXCM even for GER30 and US30, it's fast in loading up.

However, this brings me to my next point.

When loading this data off of the disk, with no connections, it still takes many minutes to load that saved data from FXCM. Saved futures data loads about 10 times faster for exactly the same chart, period, and template.

I know from talking to support in the past that NT is doing calculations for those charts when loading data off of the disk (which is actually very dumb and unnecessary), but for some reason the calculations for the futures data are much, much faster than the calculations for the FXCM data.

Interestingly, when getting the data straight from the server now (not off the disk), the same data loads very quickly.

So, there are still 2 issues for the devs to consider:

1. Why does it matter how much data is in the cache or on disk when getting data from a connection to FXCM in memory? This is bad design, and should be looked into.

2. More importantly, why are the calculations sooooooo slow when loading FXCM data off of the disk?

As an experiment, try setting up a series of linked charts. 5 days worth of data of 1 minute, 5 minute, 15 minute, 60 minute, and 30 second bars. (Put a few moving averages on there just to simulate a few simple indicators.) Link them together with a color. Set the instrument to be FDAX 03-16 from whatever futures broker you want, and load it up.

Then close down NT, and restart it.Watch how fast it loads those charts off of the disk.

Next, connect to FXCM and change the instrument to $GER30_CFDEUR. Shut down NT again, and bring it back up. This time it will try to load exactly the same charts for the new instrument of $GER30_CFDEUR. See how long that takes. For me, it is the difference between waiting 30 seconds and waiting (literally) 5-10 minutes for the workspace to load. See if the same thing happens to you, and if it does, report it to the developers.

[NinjaTrader_PatrickG]

I have run the experiment as you described and my data only takes a few seconds to load even if I clear out all of my cache and historical data.

Switching to the FXCM historical servers (which are noticeably slower) causes my 30 second chart to takes under a minute while all other charts load almost instantly.

Have you tested this independent of any previous workspaces? Delete your cache and historical data folders and then as a test, please take the following steps to force NinjaTrader to start without your saved workspace.

• Shut down NinjaTrader
  
• Navigate to (My) Documents\NinjaTrader 7\workspaces
  
• Delete the file named "_Workspaces.XML". DO NOT delete the 'workspaces' folder - only delete the '_Workspaces.XML' file within the 'workspaces' folder
  
• After deleting this file, restart NinjaTrader which should startup without any workspaces.
  
• Once NinjaTrader has started on just the control center, please open a new chart with no custom bar types/indicators/templates and test for issue.

Let me know the results of your test.