Historical and Live Data differences

[stefanjagger]

Hi,

I'm having an issue with live data showing slightly different prices than backtest data, even though the backtest data is an export of the live data.

I have NT8 connected via Trader Workstation (Interactive Brokers) for the live data connection. Using NCHL as an example today the red bar high is at 56.31.

My plan is to test this entry method again in backtesting so I created an export of the NCLH.Last.txt prices (attached) using tools > export > historical data, for the past few days as my strategry requires 35 historical prices to trade. I removed all data on my test machine for NCLH in tools > historical data. I then imported the NCLH.Last.txt prices back in using tools > import historical data > default settings. The imported data shows the red bar high is at 56.26.

I've attached a couple screenshots showing the prices data box on live and backtest/historical import.

Any ideas how to fix this please?

Many thanks,
Stefan

[stefanjagger]

It looks like the data for today's prices, is not present in the exported last prices. I tried to do a new export with prices up to 21/12/10 hoping it would take todays prices and export them but it didn't.

Is it possible to export todays prices, today, or do I have to wait until tomorrow?

Could I export if I disconnected the live connection temporarily?

[NinjaTrader_PatrickG]

You are able to export any historical data, up to the current second, as long as it resides in NinjaTrader's local database. Let's first talk about the difference between real-time and historical data.

When you connect and load a chart, NinjaTrader will first look in it's local database for historical data for that instrument. If it does not have it, it will request the data from the connected provider. This populates your chart from the moment you connect and historically (more details here). Simultaneously with this process, NinjaTrader begins printing real-time data on your chart.

Let's say you connect at 8AM. This means your chart has historical data loaded prior to 8AM and real-time data populating after 8AM. Assuming default settings, If you do not cause any sort of reload of the chart, real-time data will continue to print without necessarily being 'saved' anywhere.

Now, lets say you choose to export historical data at 3PM. If you did not manually download historical data or cause a chart reload as mentioned before, you do NOT have historical data from 8AM to 3PM locally stored in NinjaTrader's database. Therefore, I would only expect to see up until 8AM exported for today.

So, it sounds like you either need to cause a chart reload (change the interval, right-click on the chart and select 'reload all historical data, etc.) or manually download the historical data.

[stefanjagger]

Thanks Patrick. It's definitely good to understand how data is processed/stored for future, I appreciate the detailed response.

I just tried the switch timeframe trick, and even switching to a new chart, but I suspect because the strategy is running live on a 1 minute timeframe in the background it will not write that real-time data to the database until I disable the strategy / switch timeframes / export. I'll disable the strategy to test this theory later this afternoon and will report the outcome for future searches.

[stefanjagger]

I closed the NCLH chart completely, disconnected the data feed, and tried to export the data, but the last few lines of the export show the same. It doesn't appear to download the current day of data so far.

20191219 235800;55.34;55.34;55.34;55.34;0
20191219 235900;55.34;55.34;55.34;55.34;0
20191220 000000;55.34;55.34;55.34;55.34;0
20191220 000100;55.34;55.34;55.34;55.34;0

[NinjaTrader_PatrickG]

First, keep in mind that historical data is exported in UTC, so we should see something like 1646xx if we were to export data right now.

If you manually download historical data following the steps in my earlier post, can you then export the data? You can double check what historical data you have locally stored in the Control Center > Tools > Historical Data > Edit tab:

https://ninjatrader.com/support/helpGuides/nt8/index.html?editing.htm#HowToEditHistoricalData

[stefanjagger]

I think I may have exported the data before it had time to write to the local database before as I have just tried the export again and can see the data. I can see the UTC timestamps too.

20191220 163600;57.51;57.54;57.51;57.54;1900
20191220 163700;57.53;57.53;57.51;57.51;800

Thank you for all your help.