BarsRequest and tick data

[Zapzap]

I am working on an indicator, that needs 1 tick data on given bars, so I am requesting 1 tick bars knowing the from and to times of those bars.

There are two cases:
-I have a tickchart open: in this case I can request form the repository knowing that it has the data. This works fine.
-However when I don't have a tickchart open (and lets say no tick data downloaded either), that means that the repository is empty, so I need to use Lookuppolicy.Provider.
This returns tick data fine FROM BEFORE the time of connecting to data (so fully historical requests work).

There is a black area though: from the time of connecting to data (using Kinetick) I can't request any type of data if a chart doesn't write it into repository. It is not really historical yet (not written to repository until platform close) but already not realtime (as I request data on past data, sometimes hours before realtime)

What is the solution to this?

My BarsRequest is nothing fancy:
BarsRequest request = new BarsRequest(instrument, start, end);
request.BarsPeriod = new BarsPeriod { BarsPeriodType = BarsPeriodType.Tick, Value = 1 };
request.TradingHours = tradingHours;

tradinghours and instrument are copied from the indicator, no problems with that.

[NinjaTrader_ChelseaB]

Hello Zapzap,

To confirm, you are wanting to save all real-time streaming data as historical data is this correct?

There is an option for this.
Tools -> Options -> Market data -> Real-time -> Record live data as historical.

https://ninjatrader.com/support/helpGuides/nt8/en-us/options_marketdata.htm

I will not be able to assist with Lookuppolicy.Provider, but this thread will remain open for any community members that would like to assist.


Regarding a BarsRequest, this can be used to retrieve a chunk of historical data out of sync with the chart bars and optionally start a real-time stream.
Below are a few links on BarsRequests.

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/87988-barsrequest-question?p=742518#post742518

https://ninjatrader.com/support/forum/forum/ninjatrader-8/add-on-development/97523-receiving-onbarupdate-for-multiple-instruments?p=786208#post786208

[Zapzap]

To confirm, you are wanting to save all real-time streaming data as historical data is this correct?

Regarding a BarsRequest, this can be used to retrieve a chunk of historical data out of sync with the chart bars and optionally start a real-time stream.

Thanks for your answer Chelsea!

What does "historical" data really mean? So lets say:
-I connect at 7 am.
-at 11am I need tick data between 9am and 10:30am

Is this historical? Because its not realtime. So is there a third type of data?

My problem is that I only get data before 7 am. Or maybe after every full hour after that: I only tested within a full hour, but ncd files have hourly timestamps - which suggest they are saved into historical at every hour turn. (The example above won't get me the full data for sure, even though all the examples you provided should retrieve until DateTime.Now)

What is the solution to this? Start a real-time stream on all charts in case I need the data later?

[NinjaTrader_ChelseaB]

Hello Zapzap,

Historical data is downloaded as historical from the data provider when requested.

This will appear in the Tools -> Historical Data window -> on the Edit tab -> in the Historical section

Below is a link to the help guide on when historical data loads.

https://ninjatrader.com/support/helpGuides/nt8/en-us/index.html?loading_historical_data.htm#WhendoesPRODUCTNAMEloadhistoricaldata

Real-time data is data that was streamed in in real-time. This data will show on a chart but will not show in the Historical Data window.

If you need data that wasn't processed by the script, then that would be historical data.

Just requesting it would download the historical data. This could be from opening a chart, doing a backtest, or by doing a barsRequest.

[Zapzap]

Ok, so why am I not getting the full requested bars then, thats the question.
From what you said I am thinking that no matter when and for what data I am requesting bar data, I would need to fully get it, am I right?

[Zapzap]

Chelsea, thank you for the support! I learnt a lot from your answers.
BUT the culprit in this case was the Tradinghours.
I set it to NinjaTrader.Data.TradingHours.Get("Default 24 x 7"); and now it works.
Previously I was using BarsArray[0].Bars.TradingHours.

[Zapzap]

Nope, sorry, I am back to it not working.

SO "Just requesting it would download the historical data. This could be from opening a chart, doing a backtest, or by doing a barsRequest." this means it should download automatically, right?

[NinjaTrader_ChelseaB]

Hello Zapzap,

What do you have missing?

What is the instrument and what is the bar type?

Who are you connected to for data?

BarsRequest should download historical data. I have some examples scripts I haven't had any trouble with.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/add-on-development/97523-receiving-onbarupdate-for-multiple-instruments?p=786208#post786208

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/87988-barsrequest-question?p=742518#post742518

If there's an issue I can do some testing for it. Let me know.

[Zapzap]

What do you have missing?
Data does not download on BarsRequest. If I open a new tick based chart or I manually download under tools > historical data it does download new data.

What is the instrument and what is the bar type?
CL 10-19 1 tick data initiated from a minute chart.

Who are you connected to for data?
Kinetick

I restarted NT and its fine again. I'll experiment with it tomorrow and try to narrow it down to where and when it starts to fail. Thanks for the support.

[NinjaTrader_ChelseaB]

Hello Zapzap,

I'm asking you about the exact data missing.

So I can test on my end. Please provide times and dates.

[Zapzap]

Hello Zapzap,

I'm asking you about the exact data missing.

So I can test on my end. Please provide times and dates.

There is no exact data missing. I can always download data manually.
When I restart the platform barsrequest works fine for a random time, then it stops downloading data FOR barsrequest only (with nothing changed in code) until I restart the platform. Then it works fine again for some time.

I don't know what the trigger is, I'll return to you if I can safely reproduce the issue.

[Zapzap]

It seems to be random. I made a little tool to test it.

here are the outputs:
first I am requesting until 14:41:41. The request goes out at 14:45.

requesting from 14:38:41 to 14:41:41
Request callback. Last time:14:43:41 Total cnt: 562258 Filtered cnt: 997

the next request happens AFTER the previous request's last time: at 14:47, and requesting until 14:45. BUT the returned bars last time didn't change, meaning data is not updated.
requesting from 14:42:54 to 14:45:54
Request callback. Last time:14:43:41 Total cnt: 562258 Filtered cnt: 233

It usually works for some time, and then stops working. I experimented with minimizing the charts, and switching workspaces, but I couldn't find any pattern.

 public class BarsRequestTest : Indicator
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Calculate = Calculate.OnPriceChange;
                IsOverlay = true;
                DisplayInDataBox = false;
                DrawOnPricePanel = true;
                PaintPriceMarkers = false;
                ScaleJustification = NinjaTrader.Gui.Chart.ScaleJustification.Right;
                IsSuspendedWhileInactive = false;
                IsInHitTest = false;
                AllowRemovalOfDrawObjects = false;
                MaximumBarsLookBack = MaximumBarsLookBack.Infinite;
            }
            else if (State == State.DataLoaded)
            {
                if (ChartControl != null)
                    ChartControl.MouseDown += new MouseButtonEventHandler(OnMouseDown);
            }
            else if(State == State.Terminated)
            {
                if (ChartControl != null)
                    ChartControl.MouseDown -= new MouseButtonEventHandler(OnMouseDown);
            }
        }

        protected override void OnBarUpdate()
        {
        }

        private void OnMouseDown(object sender, MouseButtonEventArgs e)
        {
            DateTime start = DateTime.Now.AddMinutes(-5);
            DateTime end = DateTime.Now.AddMinutes(-2);

            Print("requesting from " + start.ToString("HH:mm:ss") + "   to " + end.ToString("HH:mm:ss"));

            BarsRequest request = new BarsRequest(this.Instrument, start, end);
            request.BarsPeriod = new BarsPeriod { BarsPeriodType = BarsPeriodType.Tick, Value = 1 };
            request.TradingHours = NinjaTrader.Data.TradingHours.Get("Default 24 x 7");

            request.Request(new Action<BarsRequest, ErrorCode, string>((barsRequest, errorCode, errorMessage) =>
            {
                if (errorCode != ErrorCode.NoError || barsRequest == null || barsRequest.Bars == null)
                {
                    Print("Error on requesting bars: " + errorCode + ", " + errorMessage);
                    return;
                }
                else
                {
                    int cnt = 0;
                    for (int c = 0; c < barsRequest.Bars.Count; c++)
                    {
                        DateTime t = barsRequest.Bars.GetTime(c);
                        if (t >= start && t <= end)
                            cnt++;
                    }
                    Print("Request callback. Last time:"+barsRequest.Bars.LastBarTime.ToString("HH:mm:ss") + "     Total cnt: " + barsRequest.Bars.Count + "   Filtered cnt: " + cnt);
                }
            }));
        }
    } 

[NinjaTrader_ChelseaB]

Hello Zapzap,

Thank you for your patience.

Our development has let us know that the data requested from BarsRequests are cached when using the DateTime parameter overload.
It is not cached when using the barsBack parameter overload.

The Cache is reset when a Reload All Historical Data is done on a chart for that instrument/data type.

The intention of BarsRequest is to be called once, and then the real-time data subscribed and used from that point forward.