Historical Data Export Formats

[NinjaTrader_Kate]

Hello trader_rick,

Thank you for your reply.

The data being exported is in Tick Replay format with subsecond granularity.

The format is:

yyyyMMdd HHmmss fffffff;last price; bid price; ask price;volume

Please let us know if we may be of further assistance to you.

[trader_rick]

Thank you Kate

[afcmaff]

NinjaTrader_Kate NinjaTrader_ChelseaB -- Is there any way to export tick data as it's displayed in the Historical Data UI database? To clarify, the UI simply shows " Date, Time, Price, Volume " (See pic). If not, why is that?

I would like to be able to export tick data with millisecond resolution based on the last price and tick volume. It seems that no matter what, NT includes bid and ask prices. Another thing that seems to be non-standard is that NT formats the data with spaces and semi-colons with no options to select the export format. If the ability to modify the export format does not exist, I would like to add my vote to a feature request that allows you to do this so that other platforms can easily import the data without having to modify millions of lines in a text file with powershell or a text editor.

Example: Currently, exporting tick data (selecting Interval 'Tick' and Data Type 'Last') produces the below output per line

yyyymmdd hhmmss ffffff;last;bid;ask;volume

Desired output is as follows:

yyyymmdd;hhmmss.fffff;price;volume

Notice the standardized semi-colon delimiter as well as joining the milliseconds with a period (could be anything other than a semi-colon). Please tell me there's a way to do this...

[NinjaTrader_ChelseaB]

Hello afcmaff,

The export historical data file format is the format for importing (only).

But you can export the historical data window grid contents. Right-click on the right where the grid is -> select Export.

https://ninjatrader.com/support/helpGuides/nt8/working_with_data_grids.htm

[afcmaff]

Thanks NinjaTrader_ChelseaB that answers my question… unfortunately. So export format is limited and internally hard-coded to essentially support importing back into NT platform. I did see that I could right click and export the window grid contents… the only issue with that is that it’s completely impractical at scale. For example, you have to right click and export for every hour displayed — so for just one quarter-based contract like ES 03-23, which typically contains roughly 23 hours of daily content, and there are on average 25 days per month, and about 4 months per contract — that’s 23x25x4 = 2,300 times you’d have to right click and wait for the excel spreadsheet to load. That’s just a single contract! Not to mention you can’t just quickly export to a text file.

On that note, I’ll have to craft a custom shell script to reformat the exported text file… which still takes a while since there are 100s of millions of lines to edit.

My ask at this point is to either create a feature request with high priority or add my vote to an existing request that specifically addresses the ability for end users to craft the export format for any type of data export. This would not be a massively complex task… perhaps even 3 to 5 story points at most and would significantly improve customer experience and use-case value for the platform.

[NinjaTrader_ChelseaB]

Hello afcmaff,

I will submit a feature request on your behalf.

Once I have a tracking ID for this request I will post in this thread for future reference.

[afcmaff]

Thanks NinjaTrader_ChelseaB

[QuantKey_Bruce]

You could write an indicator or addon that does this in whatever format you like.

[afcmaff]

Thanks QuantKey_Bruce,

Would you happen to have some sample code or a fully functional solution to this in the form of an Add-On or Indicator? That would be tremendously helpful.

Looking for something within the ecosystem or free of charge since this would be something that would help the community significantly and potentially become a part of the production release.

[QuantKey_Bruce]

What you would be doing, to do it the "easy way", is something like this but modifying the output line to be the format you are looking for.

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Indicators in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Indicators
{
    public class ExportBarData : Indicator
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"Enter the description for your new custom Indicator here.";
                Name                                        = "ExportBarData";
                Calculate                                    = Calculate.OnBarClose;
                IsOverlay                                    = false;
                DisplayInDataBox                            = true;
                DrawOnPricePanel                            = true;
                DrawHorizontalGridLines                        = true;
                DrawVerticalGridLines                        = true;
                PaintPriceMarkers                            = true;
                ScaleJustification                            = NinjaTrader.Gui.Chart.ScaleJustification.Right;
                //Disable this property if your indicator requires custom values that cumulate with each new market data event.
                //See Help Guide for additional information.
                IsSuspendedWhileInactive                    = true;
            }
            else if (State == State.Configure)
            {
                Lines.Clear();
            }
            else if (State == State.Transition)
            {
                System.IO.File.WriteAllLines("C:\\Users\\MyUserName\\Desktop\\NT " + Instrument.MasterInstrument.Name + ".last.txt", Lines); // change this path or make it a parameter
                Lines.Clear();
            }
        }

        private List<string> Lines = new List<string>();

        protected override void OnBarUpdate()
        {
            string Line = Time[0].ToString("yyyyMMdd") + // HH:mm") +
                ";" + Instrument.MasterInstrument.FormatPrice(Open[0]) +
                ";" + Instrument.MasterInstrument.FormatPrice(High[0]) +
                ";" + Instrument.MasterInstrument.FormatPrice(Low[0]) +
                ";" + Instrument.MasterInstrument.FormatPrice(Close[0]) +
                ";" + ((int)Math.Round(Volume[0])).ToString();
            Lines.Add(Line);
        }
    }
}
​

[afcmaff]

Bruce,

Thanks - this is great. Now if I wanted ‘per tick’ trades data, I’m guessing I would need to apply this to a 1-tick chart… Since OnBarUpdate depends on the data series.

[QuantKey_Bruce]

Yes, or you could modify the script to AddDataSeries a 1-tick series and then run it on that by checking BarsInProgress == 1 before doing the output.

[afcmaff]

Thanks Bruce! This works perfectly. Much faster than scripting it externally.

Cheers

[QuantKey_Bruce]

You're welcome.