Support for csv data providers

[ivannp]

From watching some videos about NT8, I gather that there are significant improvements in this area under way. As a user of this feature, let me share my experience and ideas.

Currently the process is quite painful. I have two or three historic data providers (Pinnacledata, CSIData, etc) and although I can convert from their format to NT's format easily, that's not sufficient. The problem is that NT mixes up all data from the same instrument in a, frankly, wrong way. My current solution, which is a nightmare, is to re-define the futures I am interested in for each individual provider and to use them as separate instruments. For instance I have: ZW (NT's default for wheat), CSI_ZW and PI_ZW. This way everything is kept separate, but ... I hope you see the problem.

My main desire is for NT to have a clear option to not mix the data for the *same* instrument under no circumstances, i.e. unless otherwise specified. My feeling is that this should be the default.

The second request is to allow in the namespace to easily distinguish between instruments from different data providers. One way to achieve that would be to allow provider specification in the name when referring to instruments and building lists, a.k.a. PI:ZW or CSI:ZW. Or even in the list name.

Finally, on the csv support: it should allow the data to not be cached in the database. Rather to be loaded every single time from the corresponding file. This way the files can transparently be updated by external programs. From my limited experience - multicharts does precisely that.

My understanding is that the extended csv support will come in the form of opening the APIs, which is fine with me, however, you might want to consider supporting more providers out of the box - for instance counting them as a form of a "broker" as well, thus, increasing the value of the multi broker license.

[NinjaTrader_PatrickH]

Hello ivannp,

Thank you for your post.

1. Is the historical data for the same period (date and time)? If so, then it would be expected that the data is "mixed". In addition, if the data was not for the same period, but was for earlier or later periods than the other providers' data it will still be "mixed" together to form one consistent set of historical data. What are you seeing that you would qualify as mixing the data together that would not be expected?
2. On the instrument specific data feed providers, we look to improve upon this within the next major release of NinjaTrader.
3. Please expand further on what you are looking for in regards to historical data loading. Historical data would be stored in the NinjaTrader database when loaded and called to the chart based on the guidelines in the link below, in what manner are you looking for a change here? Are you looking for NinjaTrader to pull the data from a dynamically updating database as it would when requesting data from a data feed provider (as if the external database was treated like a data feed provider)?
   • For information on when NinjaTrader loads historical data please visit the following link: http://www.ninjatrader.com/support/h...rical_data.htm
4. Multi-Broker License Keys apply to brokers and not data feed providers. I cannot confirm who, if any additional data feed providers, will be provided for the next major release of NinjaTrader.

[ivannp]

Usually daily data providers provide a set of tools, which I use to setup generation of csv backward adjusted contracts on daily basis. I run these tools nightly, thus, the files in the corresponding folders are always up to date. I'd like to be able to use them with least effort in NT. Note, the data from these providers is usually not just what they saw during the daily session. It's made consistent, it's adjusted for various events (dividends, splits, contract/option expiry) etc. NT cannot do similar merges without having all the additional information the data provider has. If someone wants to merge - so be it, however, to me that's a no no.

1. The historical data is for the same period and the same time - yes. Currently I deal mostly with futures on daily bars, and the current implementation is simply broken. After loading historical data (already adjusted), and later on connecting to Kinetick and pulling data from there (usually it happens after a few days, but some really ugly charts starts showing up). As I said above, data providers for futures take care of adjusting the data, so it's WRONG for NT to do similar magic on top. It might not be so bad for stocks, but even then, I'd think twice before merging data from different providers.

If you are wondering why do I connect to Kinetick if I have another feed - sometimes I want to see the latest daily chart without going through the pain (due to the current implementation) to reload the daily data from my feed. Sometimes I just want to verify something quickly and don't care much about the quality of the data. Either way, I don't want my high-quality historic data to get messed up with anything else.

3. Yes. Ideally, I'd like to be able to add a feed called MyFeed, to which I can connect, disconnect and control how the data is loaded. I personally don't have much problems coding it myself, as long as the appropriate APIs are provided. For my needs, I'd pull the data (for an instrument) every single time from csv files (let's say the folder is part of the feed configuration, while the file name overlaps with the instrument name), removing any previous history from the database and resisting other efforts to be smart and efficient. More overhead - yes, but as I said in 1. - futures data is already manipulated by the providers on daily basis, thus, any gimmicks are likely breaking its consistency. In a way, pulling data from csv files is already done for Google and Yahoo, so this is not much different, except the requirement of no caching and no mixing up. NT needs to implement the providers or expose the APIs.

Thanks in advance

[NinjaTrader_PatrickH]

Hello ivannp,

Thank you for your response.

I will forward your suggestion on the custom data feed to the development team. For the data being mixed on futures data, what Merge Policy are you using? You can check this under Tools > Options > Data. For information on Merge Policies please visit the following link: http://www.ninjatrader.com/support/h...rical_data.htm

[koganam]

Usually daily data providers provide a set of tools, which I use to setup generation of csv backward adjusted contracts on daily basis. I run these tools nightly, thus, the files in the corresponding folders are always up to date. I'd like to be able to use them with least effort in NT. Note, the data from these providers is usually not just what they saw during the daily session. It's made consistent, it's adjusted for various events (dividends, splits, contract/option expiry) etc. NT cannot do similar merges without having all the additional information the data provider has. If someone wants to merge - so be it, however, to me that's a no no.

1. The historical data is for the same period and the same time - yes. Currently I deal mostly with futures on daily bars, and the current implementation is simply broken. After loading historical data (already adjusted), and later on connecting to Kinetick and pulling data from there (usually it happens after a few days, but some really ugly charts starts showing up). As I said above, data providers for futures take care of adjusting the data, so it's WRONG for NT to do similar magic on top. It might not be so bad for stocks, but even then, I'd think twice before merging data from different providers.

If you are wondering why do I connect to Kinetick if I have another feed - sometimes I want to see the latest daily chart without going through the pain (due to the current implementation) to reload the daily data from my feed. Sometimes I just want to verify something quickly and don't care much about the quality of the data. Either way, I don't want my high-quality historic data to get messed up with anything else.

3. Yes. Ideally, I'd like to be able to add a feed called MyFeed, to which I can connect, disconnect and control how the data is loaded. I personally don't have much problems coding it myself, as long as the appropriate APIs are provided. For my needs, I'd pull the data (for an instrument) every single time from csv files (let's say the folder is part of the feed configuration, while the file name overlaps with the instrument name), removing any previous history from the database and resisting other efforts to be smart and efficient. More overhead - yes, but as I said in 1. - futures data is already manipulated by the providers on daily basis, thus, any gimmicks are likely breaking its consistency. In a way, pulling data from csv files is already done for Google and Yahoo, so this is not much different, except the requirement of no caching and no mixing up. NT needs to implement the providers or expose the APIs.

Thanks in advance

If you do not want the DataFeed to load historical data from the server, then just turn off the option.

ref: https://www.ninjatrader.com/support/...d.php?p=371491

[NinjaTrader_PatrickH]

Hello ivannp,

Thank you for your suggestion on how we can improve our product. It has been inserted into our tracking system with the unique ID # 298.