Using mongodb with NinjaTrader 8 (Part 1)

[mmeninger]

Using mongodb with NinjaTrader 8: Part 1 Setting up your environment, defining your data structure

Part 1: Setting up your environment, defining your data structure
Part 2: Creating JSON and example NinjaTrader/C# indicator code to perform mongodb CRUD functionality (coming soon)

Purpose:

This is an advanced post explaining one way of using the mongoDB C# drivers within a NT 8 indicator to store tick data in mongodb 3.2.7.
If you are going to do this yourself, you have to be skilled enough with software deployment, development and debugging to get through all the steps described.

What is mongodb
- a schema-less database storing data in JSON format
- used widely within the financial industry for storing financial data of all sorts.
- See the links for examples of how mongodb is used by the trading/financial industry
https://www.mongodb.com/presentation...-tick-database

https://www.mongodb.com/post/45116404296/how-banks-use-mongodb-as-a-tick-database

https://www.mongodb.com/blog/post/mongodb-single-platform-all-financial-data-ahl

Pre-requisites
- NT 8 (tested on Beta build 12). I have not tested on latest (but will someday)
- HIGHLY recommend an i7 CPU
- HIGHLY recommend a high-end graphics card as per NT 8 recommendations to distribute processing across the GPU
- HIGHLY HIGHLY recommend adding a Solid State Drive for exclusive mongodb use. Note: I have tested on a 7200 RPM spindle SATA drive and under fast/intensive market conditions, the spindle drive slowed down dramatically. This is especially important when you are doing lots of writes of new data with simultaneous reads into your charting tool for visualization of your data

Pre-requisite 3rd party software:
- mongodb install instructions:

https://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/#requirements

- Following DLLs added here: "...\Documents\NinjaTrader 8\bin\Custom"
- C# mongodb drivers (drivers must match the mongodb version you use):

https://docs.mongodb.org/ecosystem/drivers/csharp/

MongoDB.Bson.dll
MongoDB.Driver.Core.dll
MongoDB.Driver.dll
- JSON dll

http://www.newtonsoft.com/json

Newtonsoft.Json.dll...

- comments on 3rd party software.
- in your indicator code you need to right-click and choose references and then choose 'add' and navigate to the above DLLs you've added to the directory listed above
- install the mongdb on your separate SDD drive. Make the SDD exclusive for mongodb use only
- I created JSON documents of the data I stored and passed the JSON to mongo. There may be other ways of creating JSON. I used newtonsoft.

Creating your JSON structure for mongodb
- this link started me off on thinking how to do this for time-series data (e.g. tick data):

https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb

- different JSON structures have performance impacts on different read/write processing

- I have created 2 structures. The first one, immediately below stores data under the 'values' key from an algorithm that occur within every minute.
A new document is added to the collection every minute. So under fast market conditions, the logic does updates to an existing record; when a new minute occurs a new document is created.

{
        "_id" : ObjectId("583e3e6cbf1caf10cce229bf"),
        "timestamp_hour" : "2016-11-29T21:50:00Z",
        "type" : "rt",
        "epochTimeStamp" : 1480474200,
        "values" : [
                -0.07,
                -0.08,
                -0.09,
                -0.08,
                -0.09,
                -0.08,
                -0.09,
                -0.08,
                -0.09,
                -0.08
        ]
}

- The second way I store is one document for every new data-point to store. This means that if I get 100 updates per minute, I create 100 new documents. That is a waste if you don't need to it that way. But I needed to do it way for faster processing in my front-end charting app - as I need to process the data in values in a specific way that was much easier if I have one data point per document.
- So you need to have enough knowledge on how you're going to use the data to help determine the best JSON design and how documents will get inserted/updated/removed etc
- If you use a high-performance machine and with your fast SDD you have mongo installed on, it should not matter too much.

Part 2 coming soon.

Mark

[mmeninger]

Part 2A: http://ninjatrader.com/support/forum...ad.php?t=94486
Part 2B: http://ninjatrader.com/support/forum...ad.php?t=94487