Live-feed historical data vs. downloaded data

[digibob]

Hi.

I'm not sure I get the hang of testing historical data using live-feed. What happens is this: I first tested the first three months of this year 2019 via the strategy analyzer. The data for the test period was downloaded before the test ran.
Then I closed the strategy analyzer and ran a test via the chart for year-to-date 2019, but there were trades only during the first three months, ie. only for the data that was downloaded in the prior test.
Then I ran a test via the chart on a period that didn't include downloaded data, and there were trades all throughout the period.

Conclusion, as it seems, when a part but not all of the test period has downloaded data, NT performs trades only for that part that has the downloaded data. Is this correct?

I find myself continuously exporting data to see what dates were downloaded so I don't accidentally test on a 'mixed' period with part downloaded and part not. Surely it's not the way. Should I remove all the downloaded data first? Should I alternatively download everything (which I suppose would take time) before testing specific periods? What is the right approach here to not have to be vigilant about the test period?

Thank you

[NinjaTrader_Jesse]

Hello digibob,

Thank you for the post.

Conclusion, as it seems, when a part but not all of the test period has downloaded data, NT performs trades only for that part that has the downloaded data. Is this correct?

Generally that is the case however this depends if the data can be downloaded and if you are connected. You would need to download the data to be tested over and if only part of the timeframe is there, that is all that can be used. If you open a chart to the period you are testing that data should be downloaded. The analyzer can also download data when you request a time period, did you not see this happen or were you not connected at this time?

You also don't need to export data to see what data you have, you can either Open a chart and view the data or use the historical data manager to view what dates are downloaded.

Should I remove all the downloaded data first?

No, you will need this data if you want to test over that period you just need to now download the rest of the data you wanted to use.

Should I alternatively download everything (which I suppose would take time) before testing specific periods?

Yes, if you plan to test 1 year you need to download data for 1 year. The platform would need some data to process the logic with.

If you are not seeing data being downloaded in some situation when you are connected and requesting a period of data, that is a different situation which we could focus on.


I look forward to being of further assistance.

[digibob]

Hi Jesse. Thanks for the reply and the good tips on how to see the data. I should mention that the strategy I'm testing uses four time frames, hence what I see on the chart when running the strategy does not always tell the whole story unless I open more charts separately.The question remains, why the need to see what data is there in the first place. If I run a test for Jan-Feb and have previously downloaded data only for Jan, the system should use the existing data for Jan + download data for Feb and show trades for both months instead of just for Jan.

There is more to it than just the trades. When testing a 'mixed' period the test takes longer, irrespective of the period length. Sometimes it doesn't run at all, or maybe I just wasn't patient enough to wait (I did wait, long). The system clearly doesn't like it when a period is mixed, which in turn requires one to very carefully map (download) the data. For example if I downloaded Jan 1 to 14 and then Jan 16 to 31, I would never know that Jan 15 is missing.

I suppose once all the data is mapped it is no longer an issue and soon I won't care enough to inquire about it. Nonetheless, the behavior of the system during mapping strikes me as being more developer-friendly than user-friendly.

Having said that, there is plenty for me yet to discover and so I take into account that I'm still missing something here.

If I may, a conceptual question to get a better grip on things: what is the difference between importing from file and using a live-feed (as far as historical data is concerned)? Is live-feed just another way to import? Is there no difference to do with the 'live' in live-feed?

Thanks.

Kind regards

[NinjaTrader_Jesse]

Hello

If I run a test for Jan-Feb and have previously downloaded data only for Jan, the system should use the existing data for Jan + download data for Feb and show trades for both months instead of just for Jan.

Correct, if you are not seeing that we could look into that situation. The platform will generally download data so long as you are connected at that time to a provider which offers the data being requested. This should happen when using a chart or the other tools that require historical data.

When testing a 'mixed' period the test takes longer, irrespective of the period length. Sometimes it doesn't run at all, or maybe I just wasn't patient enough to wait (I did wait, long).

If the test does not run at all, that is a good indication that one of the series did not load data or that the logic the script uses was somehow not being met to place trades. You noted you use 4 series, so all 4 series would need to have data downloaded for that time period to run.

For example if I downloaded Jan 1 to 14 and then Jan 16 to 31, I would never know that Jan 15 is missing.

If you are testing specifically those date ranges, no you wouldn't know that the date was missing as it was not included. Generally if you plan to test over multiple areas in a span of time, it is best to just download the larger set of data. For example open a chart from Jan 1 to March 1 and then there is no issue with data not being downloaded. The next time you test it will just use the data you have or request it if there is missing data from the dates you choose. You can use a chart or the historical data manager to download large periods of data at once. You can also request the full length of time in a backtest to include all data to be downloaded.

If I may, a conceptual question to get a better grip on things: what is the difference between importing from file and using a live-feed (as far as historical data is concerned)? Is live-feed just another way to import? Is there no difference to do with the 'live' in live-feed?

When importing from file you only have that data once meaning if anything were to ever be updated on the actual data held by the historical data servers, you would not get that update. The historical data servers can be queried from a live data feed and then download the new data including any updates. Live data streamed in realtime is just saved as historical data and collected.

The only major difference is that the servers can include updates to the data for example if some data was missing for some reason and was later inserted. Aside from that data by file may be available for further back periods, the servers generally contain a running total of data which never exceeds a set number of days/years depending on the type of data. By file you can import saved data as far back as you have data for, this could be much longer than what is available on the server depending on where you downloaded the data.

I look forward to being of further assistance.

[digibob]

Jesse, thank you for the detailed reply. I have a better grasp now for sure. My experience, meanwhile, is different from what you describe. Please see below screenshots -

The first is a test Jan 1 - Jan 31 (daily chart). There are trades all throughout the period.
Then I move the period two weeks forward in time to be Jan 15 - Feb 15. At this point, before the test is ran, there is downloaded data for January for all four series, there is downloaded data for February for the main price series, and there is no downloaded data yet for February for the other three price series.
The result of this test as can be seen in the second screenshot - there are trades in January but not in February. The data for February for the three other series are never downloaded, as confirmed by checking the historical data manager.





When I move the start of the test period to be Jan 31, the result is still the same. When I move it to be Feb 1, only then the data is downloaded and trades show up. This is what I meant by having to be precise when setting the dates.

I think there might be a scenario here the developers have not considered.

As for downloading everything in one go, that would be ideal but I would like to have data for four series starting as far back as provided, and I don't have the option to leave the machine running all day, so I have to do it piece by piece.

As for the connection to the provider, it is not the problem. I have tested the above scenario more than once. The connection is always on and the result is always the same.

Thanks

[NinjaTrader_Jesse]

Hello digibob,

Thank you for the reply.

Can you tell me what data provider is being used? The historical data being downloaded is part of the overall data provider connection so that could potentially be involved, I would need that information to review this further.

As for downloading everything in one go, that would be ideal but I would like to have data for four series starting as far back as provided, and I don't have the option to leave the machine running all day, so I have to do it piece by piece.

I am not certain I understand what you mean here, are you saying that your data provider does not offer historical data for the instruments used and you need to record the data? You should be able to download larger sets of data at once if your provider offers historical data for these instruments.
If you are also using Daily bars, there should be jan- march on the server, daily bars generally go back the farthest of the data types. Are the other 3 series all daily series as well?

Can you also detail the specific steps you used as a list, am I correct in the steps of:

run Jan 1 - Jan 31 (daily chart).
Change to Jan 15 - Feb 15. and re run the test?

I look forward to being of further assistance.

[digibob]

Hi Jesse. Yes, the data provider is an important point that I neglected to mention. It is Interactive Brokers. Of the four series in use, two are daily (one of which is main) and two are minute. In particular, one of the minute series is the S&P 500 index. IB provides two sets of index data, native from the exchange and in-house calculated. The setting for this is missing from my TWS for some reason. I believe the default is in-house calculated, which is probably what I'm downloading.

To explain why downloading everything in one go is not doable at the moment I should mention I'm in a transition stage with currently limited free time and limited access to my machine, and those minute series do take time to download. A one-year test takes between 30 to 45 minutes to retrieve all the data. Is this abnormally long?

I should also note that a main part of what I'm doing right now is simply learning the NT platform and how to work with a live-feed, as this is new to me. Efficiency will hopefully come later as my overall setup is going to change and I'm also in the process of looking for a different data provider. I'm just confused at this point about this behavior of the system, is it by-design or undesired. Perhaps it is only happening on my machine ?

This is the sequence:

1) A strategy with four series: daily as main, another daily, and two minute series, one of which is an index if that makes a difference. The trades take place on the 4th series (minute).

2) Have data for the main daily series for the entire period, in this example Jan-Feb. Do not yet have data for the other three series.

3) Run test Jan 1 - Jan 31 from the chart of the main daily series. The data for the other three series should download and the test perform in full. All is good so far.

4) Change to Jan 15 - Feb 15 and re run. You could also try Jan 31 - Feb 15 for that matter. And then to see if there is a difference, try Feb 1 - Feb 15.

.
The topic is a little over my head, thus I may not immediately know what details to provide. My apologies and thanks for bearing with me.

[NinjaTrader_Jesse]

Hello digibob,

Yes it sounds like the data is downloading slow if the 35 minutes is strictly only downloading data. If you are also including testing time, that may be expected depending on the test. Downloading data generally does not take very long even for larger sets of data, this may take a few minutes unless you are loading extremely large sets of data which may take longer. From the given details it does not sound like this is a large set of data, however if you are requesting farther back such as the whole year for all series that may take slightly longer.

When you load a chart with all 4 series, does it take the same time to load or are you only seeing this in the backtest?

I will see if I can reproduce the steps you provided on my end so that we can look further into the data not being loaded.

As for the time this is taking to download, are you using a stable internet connection that is hard wired to your modem/router? You also noted you have limited access to the machine, is this because it is a VPS/server with limited allocated resources that can be used, or you just in general wont have access to the machine?

Is there any possibility to use a different machine for testing? That may be a better option so that you can explore the platform for longer periods of time, this would also allow you to check if you can download the data faster using a different pc/connection.



I look forward to being of further assistance.

[digibob]

Hi Jesse.

I haven't tried loading all 4 series together on a chart but I've tried them separately, and the minute series for one year does take a while to load, I'd say between 10 to 15 minutes, so downloading two of these plus the other daily, I can see how it adds up to 30+ minutes.

But issues to do with the quality of the internet connection are probably to be expected here - I am abroad. I am not too worried about that as I trust such issues will be resolved later when I'm back home. It is very good to learn that downloading should normally take faster. The said 30+ minutes had almost all to do with the downloading. The test itself once the data is there takes less than a minute.

There is just the "to download or not to download" issue, that doesn't seem to be connection-related. I look forward to your testing it on your end.

[NinjaTrader_Jesse]

Hello digibob,

Yes that will be problematic if you are using a limited connection. Generally it is suggested to use a hard wired connection that is high speed, once you are back home I am sure you will see an improvement in speeds. This could also relate to some of the other items you are seeing in case requests are timing out. I would likely suggest starting your actual analysis of how you like the platform etc once you are back home as you will see some improvements with a better connection.

I tried the steps you provided on my end however I can see all data was downloaded in my tests. I will test this more however I am currently seeing all data download.

I Look forward to being of further assistance.

[digibob]

Hi Jesse. I would have to agree with your good advice. The setup here is not ideal and might be causing all sorts of issues I'm not aware of. My aim was to save time for later and do the setting-up and getting-to-know part now, but I guess there is no going around the hardware requirement. Your seeing other results from what I'm seeing is a further testimony of that. It is encouraging to know that the platform can deal with the said scenario. It probably won't hurt to re install NT at this point, and in the meantime I will focus on offline tests using the data that was downloaded so far. Later on I will start again clean.

Your help is much appreciated. The goal was to learn and I have learned a lot. Many thanks!