How to modify DataProvider Data?

[Kabua]

How do I access the current DataProvider (or inject into the pipeline) so that I can modify the incoming data stream for a specific instrument?

Background: I need to "fix" the Forex volume, in real-time it is quoted in currency, in historical data it is stored in units of 10,000. Thus all my charts and indicators are completely messed up between historical data and anything that was captured during real-time.

thanks.

[NinjaTrader_Kate]

Hello Kabua,

Thank you for your post.

There's no supported way to modify the data "in-flight" so to speak. Would you be able to provide some examples of the discrepancy you're seeing between historical and real time data? Who is your real time data provider for Forex, and how are you getting the historical data? Is it coming from your data provider's historical data servers, or another source?

Thanks in advance; I look forward to assisting you further.

[Kabua]

My data provider is FOREX.com for Real-time and NinjaTrader historical data server for History.

[NinjaTrader_Kate]

Hello Kabua,

Thank you for your reply.

As of Release 8.0.14.0, May 24, 2018, Forex.com/G2 historical volume was set at 10,000 to be consistent with real-time volume.

Unlike CME FX Futures products traded on a central exchange, the Chicago Mercantile Exchange, cash forex is traded across many banks, brokers, and private networks thus true volume is impossible to measure. For this reason, volume is set at 10,000.

In real time, indicators such as the VOL indicator would simply count 10,000 for each tick that comes in. For historical, all bars will return 10,000 when using the Forex.com historical data servers, which would be suggested over the NinjaTrader Historical data servers as NinjaTrader's historical data comes from FXCM and data between the two is different.

I would not expect any Volume indicator to work properly with currency pairs as due to the decentralized nature of the market there is no official Volume.

Please let us know if we may be of further assistance to you.

[Kabua]

I get all that. Most Forex traders understand that Forex is decentralized and therefore there is "No official" volume per se. And with an average volume of 3+ Trillion dollars in EurUSD currency per day then the average volume is roughly 2 million per minute. However, not every minute is equal and most Forex traders don't care about "official" volume but relative volume from the same provider does have value.

What doesn't work is the difference between real-time $10,000 summation and Historical unit counts.

For example 15 ticks; Real-time reports $150,000 whereas the historical database reports 15, which I must multiply by $10,000. Now, naively I copied the VOL indicator and modified it to "If NOT real-time multiple by 10,000 otherwise leave as-is" which worked just fine until I ran the playback connection which all data is flagged as real-time. So, one solution is to check the connection type as well as the real-time status.

Therefore, how do I know if I'm running from the playback connection or not?
Thanks.

[NinjaTrader_Kate]

Hello Kabua,

Thank you for your reply.

You're correct when you say that "most Forex traders don't care about "official" volume but relative volume from the same provider does have value". The important piece of that sentence is "from the same provider". It's also important to note that that 10,000 is not $10,000 but 10,000 units - think of it like reporting a single mini lot every time a tick comes in.

If you're using the NinjaTrader historical data servers with a Forex.com real time connection, that historical data is not sourced from the same real time provider you're using, nor would the data recorded as Market Replay data. If you reload the historical bars, it's not guaranteed that multiplying them by 10,000 would result in the same as what was originally counted in real time from the Forex.com connection.

There's not a supported way to check if you're running on the playback connection, but it does look like checking Instrument.GetMarketDataConnection() == Connection.PlaybackConnection does currently return True if the playback connection is connected and False if it isn't. You could potentially use that to check - however, as it isn't technically supported that is subject to change, so that would be a use-at-your-own-risk item.

Please let us know if we may be of further assistance to you.

[Kabua]

Understood.

It is a better approach than what I currently have, so thanks!

[linuxguru]

Kabua, et al.

I have developed a solution for this problem for my charts and backtesting. I have modified the @DayBarsType.cs file to divide out the 10000 multiply so that the volume data from FOREX.com is consistent while not breaking other non-FOREX charts.

The reason I had to make this change is because while the historical volume data was not multiplied by 10000, the live volume updates were, so the volume for the live bar was growing substantially compared to the previous historical bars.

The code change basically checks the volume on added or updated bars to see if the volume is over 10000 and is evening divisible by 10000, and if the instrument is FOREX, then it divides the volume.

I had to do this in my continuous (so far month-long) effort to successfully create a viable strategy development framework for FOREX on Ninjatrader.

I am pleased that Ninjatrader is designed in such a way that the end user has access to the internal workings of the software and can modify details such as these. Thank you Ninjatrader!

Attached is the modified file. This is from NT8 8.0.21.1 64-bit. I hope it helps.

[Kabua]

linuxguru thank you! I will check it out.