False Price Spikes Occuring with IB FOREX

[NinjaTrader_JoshP]

On our end we will only be able to test from a blank workspace. Can you please confirm that you can actually reproduce from a blank workspace? This would be a critical isolation step. Thank you.

[Harry]

Will try again tomorrow, could not produce from a blank workspace today.

[Harry]

The spike problem only seems to occur once NinjaTrader is running for a while. This morning I could not reproduce, but after several hours I could now reproduce with my default workspace - but not with a single chart.

High of bar shown on chart is 1.3325, while high of historical data is 1.3312. This is a difference of 13 pips and qualifies as a spike.

The problem is that NinjaTrader is building up something if you have it open and connected for several hours:

(1) RAM required starts with about 200k when starting NinjaTrader, then gradually increases to about 300k after two hours. If I close all workspace and just leave an untitled empty workspace, RAM usage stays at 260k. If I open a simple chart and close it again, RAM may go up to 280k and stay there.

(2) Editing historical data is fast after starting NinjaTrader, but if you try to do this after two hours it takes over a minute to get down to edit the minute data.

(3) Volume and price spikes only occur after a longer connection time. The size of volume and price spikes is proportional to the connection time. The longer the connection time, the larger the price and volume spikes.

If I close down NT -> problem solved when restarted
If I close the charts and/or the workspace -> problem solved when chart and/or workspace reopened.

(4) The problem cannot be reproduced with a single chart open

Some Guesswork

I can only make a guess what is happening: When I have my 6 charts open, Ninjatrader collects real-time data, which is tick data. This gradually builds into RAM increasing the RAM usage by 50% over several hours.

With the cumulated real-time data lurking in the background, NinjaTrader creates false hybrid bars for FOREX. Instead of building the hybrid bar from historical data and real-time data, NinjaTrader adds another single data point that it treats as a real tick. This single false tick creates the spike. Sometimes it is created prior to the display of the price bars and sometimes this single false tick is added about 1 second after the price bar has been built. Price will then jump up 10 or 15 ticks and jump down back to the last price within a fraction of a second leaving that spike. This single false tick is not displayed on the T&S.

I cannot do more than describe what NT is doing. Even if you do not like it: The price spikes, which I have only observed for FOREX nad the volume spikes, which I have observed for all instruments build in an identical fashion.

The price and volume spikes will go away when reloading the historical chart data.

[NinjaTrader_JoshP]

Harry,

Would you be willing to let us try with your IB account and your workspace (proprietary indicators stripped out) and just leaving it to run? Then trying to pull up some forex charts after a few hours?

My understanding is you are experiencing this only on the last bar on the chart (a.k.a. the currently building bar). This spike is not represented in Historical Data Manager. This spike is not represented in any T&S windows.

Now for the T&S windows, you are sure you have them open prior to switching the chart to the forex instrument correct? As in, you are capturing every event the whole time as opposed to having it linked to the chart and clean slating itself as you switch the instrument on the chart?

[Harry]

Harry,

Would you be willing to let us try with your IB account and your workspace (proprietary indicators stripped out) and just leaving it to run? Then trying to pull up some forex charts after a few hours?

My understanding is you are experiencing this only on the last bar on the chart (a.k.a. the currently building bar). This spike is not represented in Historical Data Manager. This spike is not represented in any T&S windows.

Now for the T&S windows, you are sure you have them open prior to switching the chart to the forex instrument correct? As in, you are capturing every event the whole time as opposed to having it linked to the chart and clean slating itself as you switch the instrument on the chart?

Will try to reproduce with a different workspace and let you try. V´But again this thing is not easy to produce.

Yes the spike only builds on the last bar when opening a chart, that is the hybrid bar built from historical and real-time data. This bar is always identical with the last minute bar shown in the historical data manager. The spike is not represented in the historical data manager, otherwise it would not go away, when reloading data.

I had the T&S both open prior to opening the chart and sometimes linked to the chart.

[NinjaTrader_JoshP]

Harry,

Let's try without linking the T&S. Just have a bunch of T&S open, one for each of the various forex instruments you are trying so we can capture any real-time events which might be missed because of the link switching instruments.

[Harry]

Harry,

Let's try without linking the T&S. Just have a bunch of T&S open, one for each of the various forex instruments you are trying so we can capture any real-time events which might be missed because of the link switching instruments.

I tried to reproduce today with a bunch of T&S open, but could not produce any spikes. Maybe the fact that a T&S is open stops the spikes. However, this is only a guess, not a confirmed observation.

Will continue testing during the next days and come back.

[NinjaTrader_JoshP]

Ok. Also, do you have "Store real-time data as historical" checked in Tools>Options>Data?

[Harry]

Ok. Also, do you have "Store real-time data as historical" checked in Tools>Options>Data?

I never check this box.

Resaon: I do not want to save that low quality (false volume, false time stamps) data from IB. Also, if I save it, I would have to overwrite/reload it regularly, which I do not want. Don't need the tick data either.

[Harry]

The FOREX spikes reappear:

Conditions:

-> must be connected to NinjaTrader/IB for several hours
-> open a new chart for a FOREX instrument, for which no chart had been open / no historical data loaded during the last hours

Just opened a chart for EURUSD, and you will notice the spike. As exposed before the spike is the last bar on the chart, when opened, which is the hybrid bar.

The FOREX spikes behave exaclty like the volume spikes: Long connection time required, first loading of instrument data, but volume spikes occur with all instruments.

To get rid of the spikes reloading of data is required, The historical bar will then override the false real-time data bar.

[Harry]

If you like I can conduct some tests. I have not used any FOREX instruments today, so there are some of them left for experiments.

[Harry]

Tried to reproduce with an empty workspace. Does not work. It apparently needs some charts running in the background to create this problem. I think it is linked to the synchronizing of real-time and historical data, when opening a chart. Something going wrong there. 1 tick of the first realtime bar is completely off.

[NinjaTrader_JoshP]

When you say charts running in the background, do you mean in another workspace? So you are saying you already have a chart of the instrument that spiked somewhere else already going?

"-> open a new chart for a FOREX instrument, for which no chart had been open / no historical data loaded during the last hours"

At what time was this "last hours" observed? Perhaps this could be time sensitive as to the exact time the request was made.

[Harry]

I only have one workspace open, always.

To produce a spike, no market analyzer, Times & Sales or anything is allowed to be open for this instrument.

The spike only appears, when the these conditions are met

(1) NinjaTrader has been connected for several hours to Interactive Brokers with an active workspace (6 or 7 charts running and collecting real-time data, which is displayed on the charts)

(2) during the hours the FOREX instrument that produces the spike had not been touched (no Times&Sales, no Market Analyzer), all charts on the workspace show other instruments

(3) A new chart for this instrument is opened, which triggers the loading of historical data for several hours (or alternatively an existing chart is toggled to this instrument.)

The spike will occur on the last bar of the new chart, which is a hybrid historical / real-time data bar. Spikes only occur for FOREX instruments (and volume bars), but not for futures contracts.

The spike will go unnoticed on a 15 min chart. But you can see it immediately on a 1 min chart. This is because the spike's size does not depend on the timeframe.

I am doing a test right now. I have now been connected for exactly 1 hour with my usual workspace. I have executed two orders and I am connected. It is 7:20 PM CET, or 1:20 PM EST.

Test: I will now open new 1 min charts with a lookback of 65 days for 9 forex pairs, one by one and check for spikes. Test starts now.

Result of the test:

No spikes for AUDUSD, EURAUD, EURGBP, EURJPY, USDCAD, USDCHF, USDJPY.

Spikes were produced for EURUSD and GBPUSD. Charts with spikes attached.

[NinjaTrader_JoshP]

Harry,

For your test we need to remove variables. Please remove all custom indicators and just use vanilla charts. We want to try and find a scenario that does not contain any extras to be able to eliminate those as potential causes.