Using mongodb with NinjaTrader 8 (Part 2A - mongo driver code in Part 2B )

[mmeninger]

Indicator to mongo logic flow

The following outlines the pieces of code I put within the indicator class and the additional classes I added to the Indicators namespace to be able to store data in mongodb.

(Mongo driver code is in Part 2B due to NT forum thread limit of 10,000 characters.)

Summary of Logic & Location


Declarations

 
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using Newtonsoft.Json.Linq;

using MongoDB.Bson;
using MongoDB.Driver;

INSIDE Indicator sub-class (public class MyIndicator : Indicator)

CLASS attributes:

                                       
          private InsertUpdateProcessor insertOrUpdate;
          private List <double> zScoresToPersist; #for updates only

*Initialization*
State.Configure
- variables to store data used by the indicator

State.DataLoaded
- variables that are time specific: what type of bar am I: month, week, day or tick
- this is important because I want to store data differently depending upon the type of bar I am using. Different document creation logic for a weekly bar vs a daily bar vs a small tick bar etc
- state object storage object: I 'global' object that gets passed between the indicator class and the mongo classes to track the state of data

insertOrUpdate = new InsertUpdateProcessor();

OnBarUpdate:
- Core Indicator Logic - where algorithm calculations are made and values for persistence are created
- Data comes from the indicator and gets added to an Accessor object that gets passed to the JSON creator class to create the required JSON document
- The state of the persistence is passed back to the indicator from the mongo support classes so new data created includes correct instructions for persistence (what is the latest bson ID (e.g. _ID) in case the next data value created by the indicator gets added to an existing document (update) or a new document is created (insert)
- Conditions for Persistence: is the bar type and the associated storage logic for that bar type; what is the last record/timestamp recorded; on reload/load-up when does data get persisted; is data historical or is it real-time
- Objects used in this method:
- DataObject (data created in indicator sub-class)

				        ZScoreDataObject dataObj = new ZScoreDataObject();
					dataObj.dateString = Time[0].ToString("dd/MM/yyyy HH:mm");
					dataObj.month = Convert.ToInt32(Time[0].ToString("MM"));
					dataObj.week = week;
					dataObj.day = Convert.ToInt32(Time[0].ToString("dd"));
					dataObj.minute = Convert.ToInt32(Time[0].ToString("mm"));
					dataObj.second = Convert.ToInt32(Time[0].ToString("ss"));
					dataObj.milliSecond = Convert.ToInt32(Time[0].ToString("fff"));
					dataObj.zScore = PLLVolZScoreHolder[0];
					dataObj.dataType = dt;
					dataObj.cutOffDT = this.cutOffDT;
					dataObj.timeNow = Time[0];

- InsertUpdateProcessor (determines whether this new data value should be persisted and whether it should be inserted into a new collection or updated into an existing collection)

					if(!insertOrUpdate.NewZScoreNeeded(dataObj))
						return;

					bool create = insertOrUpdate.DoICreateNewRecord(dataObj);

- MongoInterface (insert this data or update the existing record with this data)

					if(create)
					{

					    mongo.collectionName = dataStoreName;
   					    mongo.InsertNewZScore(dataObj, ref localTrackObj, dataStoreInsertUpdateTracker);	
    					    lastMongoID = mongo.GetLatestMongoObjectId();
					}
					else
					{
 					   mongo.collectionName = dataStoreName;
					   mongo.UpdateAddManyZScores(zScoresToPersist , lastMongoID, ref localTrackObj, dataStoreInsertUpdateTracker);	
					   lastMongoID = mongo.GetLatestMongoObjectId(); //this is the mongo bson id
                                           zScoresToPersist.clear()
					}

SEPARATE CLASSES outside of MyIndicator sub-class

- this is ALL WITHIN the namespace NinjaTrader.NinjaScript.Indicators

public class DataObject
- C# Accessor object (getter/setters of data)
- https://msdn.microsoft.com/en-us/lib...(v=vs.71).aspx
- contain all the data you want to store. Gets created and populated within the MyIndicatorSubclass

	public class ZScoreDataObject
	{
		
		public string dateString { get; set; }
		public int month { get; set; }
		public int week { get; set; }
		public int day { get; set; }
		public int minute { get; set; }
		public int second { get; set; }
		public int milliSecond{ get; set; }
		public double zScore { get; set; }
		public int dataType{ get; set; }
		public DateTime cutOffDT{ get; set; }
		public DateTime timeNow { get; set; }
		
	}

public class InsertUpdateTrackObject
- accessor object for:
- datetime stamp of last mongo command
- latest mongo bson id: e.g. "_id": ObjectId("507f1f77bcf86cd799439011")
- count of the collection being worked on
- cleanup data: offset; DT

	public class ZScoreInsertUpdateTrackObject
	{
		public DateTime cutOffDT {get; set;}
		public string documentMongoID {get; set;}
		public long collectionCount {get; set;}
		public int cleanUpOffSet  {get; set;}
		public DateTime cleanUpDT {get; set;}
	}

protected class TimeManagement
- accessor object for the previous minute, day, week, month, barType
- I want this because I want to compare the new data with the previous data
- Configuration will tell me

	protected class TimeManagement
	{
		public int previousZScoreMin {get; set;}
		public int previousZScoreSec {get; set;}
		public int previousZScoreDay {get; set;}
		public int previousZScoreWeek {get; set;}
		public int previousZScoreMonth {get; set;}
		public NinjaTrader.Data.BarsPeriodType barType {get; set;}
		
	}

protected class CreateJSONObject
- create the JSON object with the data from the accessor DataObject

	protected class CreateJSONObject
	{
	        //use newtonsoft JSON methods
		public JObject CreateZScoresJSON(ZScoreDataObject dataObject)
		{
			
			GetDateFromString theConverter = new GetDateFromString();
			DateTime cutOffDT = theConverter.GetTheDateTimeObject(dataObject.dateString);
			DateTime convertedDT = new DateTime(cutOffDT.Year, cutOffDT.Month, cutOffDT.Day, cutOffDT.Hour, cutOffDT.Minute,0,DateTimeKind.Utc);
			JObject localMainObj = new JObject();
			localMainObj["timestamp_hour"] = convertedDT;

			string strDataType = "";
			if (dataObject.dataType ==0)
				strDataType = "rt";
			else if (dataObject.dataType ==1)
				strDataType = "h";
			
			localMainObj["type"] = strDataType;
			
			long epoch =  ConvertToUnixTimestamp(convertedDT); //helped method to store my DT in epoch time
			localMainObj["epochTimeStamp"] = epoch.ToString();
	
        	        JArray arrOfValues = new JArray();
			arrOfValues.Add(dataObject.zScore);
			localMainObj["values"] = arrOfValues;
			
			return localMainObj;
			
						
		}		
		
		public JObject CreateDateTimeJSON(string dateTime)
		{
			JObject localMainObj = new JObject();
		        localMainObj["timestamp"] = dateTime;
			
			return localMainObj;
			
			
		}

public class InsertUpdateProcessor
- is the date of the current data in the indicator newer than what is in mongo. If if the data is > mongo data then I need to add it to mongo; if not don't
- so if I need the data, do I need to create a new document in mongo (insert) or update an existing one (update)

          public bool NewZScoreNeeded(ZScoreDataObject newDataObject)
                 return true if this new data item is newer than what if I have in mongo

          public bool DoICreateNewRecord(ZScoreDataObject newDataObject)
                return true if the local logic states that a new mongo collection is needed; false 
means to update the existing latest record

[mmeninger]

Using mongodb with NinjaTrader 8 (Part 2B - mongo driver code)

Continuation of Part 2A due to NT forum character length limitation.
------

public class MongoInterface
- this class does too much and I could create a separate class to manage other functions
- wraps the needed C# mongo objects and methods: _client, _database
- insert wrapping method when inserts are needed
- update wrapping method when updates are needed

	public class MongoInterface
	{
		protected static IMongoClient _client = new MongoClient("<uri>");
		protected static IMongoDatabase _database = _client.GetDatabase("<db name>");				
		public string collectionName {get; set;}
		public string mongoObjectId;
		//Local JSON object
		public JObject jObjectInstance {get; set;}
		//creator object
		public CreateJSONObject createObj;
		private BsonDocument bsonOfJObject;
		//Exposed method that gets a new data object from the indicator
		//One instance created in the indicator class

		public void InsertNewZScore(ZScoreDataObject newDataObject, ref ZScoreInsertUpdateTrackObject objTrack, string lastUpdateCollectionName )
		{
			
			createObj = new CreateJSONObject();
			jObjectInstance = createObj.CreateZScoresJSON(newDataObject); //the data object comes from the indicator and the object returned is JSON of the data
			bsonOfJObject = this.ConvertToBSON(jObjectInstance); //convert the JSON to BSON
                        //C# driver
			var collection  = _database.GetCollection<BsonDocument>(this.collectionName); //collection name is "instrument name.bar type.bar size" 
                        //C# driver
			collection.InsertOne(bsonOfJObject);					
			this.mongoObjectId = bsonOfJObject["_id"].ToString();
                        //Here call the LateUpdateTimeStamp method, passing in the objectID to update			
                        this.LateUpdateTimeStamp(newDataObject.dateString, ref objTrack, lastUpdateCollectionName); //local method to save the mongo bson id of the record created
		}
		
		public void UpdateAddManyZScores(List<double> storedZScores, string mongoObjectId, ref ZScoreInsertUpdateTrackObject objTrack, string lastUpdateCollectionName )
		{
			BsonArray localBSONArray = ConvertListToBson(storedZScores); //local method to convert a list of data into a BSON array for insertion into an existing object
                        //C# driver
			var collection  = _database.GetCollection<BsonDocument>(this.collectionName);
                        //C# driver
			var filter_id = Builders<BsonDocument>.Filter.Eq("_id", ObjectId.Parse(mongoObjectId));
                        //C# driver
			var updateBuilder = Builders<BsonDocument>.Update;
                        //C# driver
			var update = updateBuilder.PushEach("values",localBSONArray); //push each value in the BSON array to the "values" key in the mongo document
                        //C# driver
			var result = collection.UpdateOne(filter_id,update);
                        //C# driver
			var document = collection.Find(filter_id).FirstOrDefault();
			string latestDateTime = DateTime.Now.ToString("dd/MM/yyyy HH:mm");
			this.mongoObjectId = mongoObjectId;	
                        //Here call the LateUpdateTimeStamp method, passing in the objectID to update
			this.LateUpdateTimeStamp(latestDateTime, ref objTrack, lastUpdateCollectionName);
			
		}

		//Re-useable method for removing based on a date
		public long RemoveFromMongo(string cleanUpDT, IMongoCollection<BsonDocument> collectName)
		{
			var filter_id = Builders<BsonDocument>.Filter.Lt("timestamp_hour", cleanUpDT);
		        var document = collectName.DeleteMany(filter_id);			
			return document.DeletedCount;
		}
 

         //... other supporting methods for the logic
       }

[NinjaTrader_ChelseaB]

Hello mmeninger,

Interfacing with an external application is far outside of what is supported by NinjaTrader Platform Support, however, this thread will remain open for any community members that would like to assist.