• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Live-feed historical data vs. downloaded data

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Live-feed historical data vs. downloaded data

    Hi.

    I'm not sure I get the hang of testing historical data using live-feed. What happens is this: I first tested the first three months of this year 2019 via the strategy analyzer. The data for the test period was downloaded before the test ran.
    Then I closed the strategy analyzer and ran a test via the chart for year-to-date 2019, but there were trades only during the first three months, ie. only for the data that was downloaded in the prior test.
    Then I ran a test via the chart on a period that didn't include downloaded data, and there were trades all throughout the period.

    Conclusion, as it seems, when a part but not all of the test period has downloaded data, NT performs trades only for that part that has the downloaded data. Is this correct?

    I find myself continuously exporting data to see what dates were downloaded so I don't accidentally test on a 'mixed' period with part downloaded and part not. Surely it's not the way. Should I remove all the downloaded data first? Should I alternatively download everything (which I suppose would take time) before testing specific periods? What is the right approach here to not have to be vigilant about the test period?

    Thank you

    #2
    Hello digibob,

    Thank you for the post.

    Conclusion, as it seems, when a part but not all of the test period has downloaded data, NT performs trades only for that part that has the downloaded data. Is this correct?
    Generally that is the case however this depends if the data can be downloaded and if you are connected. You would need to download the data to be tested over and if only part of the timeframe is there, that is all that can be used. If you open a chart to the period you are testing that data should be downloaded. The analyzer can also download data when you request a time period, did you not see this happen or were you not connected at this time?

    You also don't need to export data to see what data you have, you can either Open a chart and view the data or use the historical data manager to view what dates are downloaded.

    Should I remove all the downloaded data first?
    No, you will need this data if you want to test over that period you just need to now download the rest of the data you wanted to use.

    Should I alternatively download everything (which I suppose would take time) before testing specific periods?
    Yes, if you plan to test 1 year you need to download data for 1 year. The platform would need some data to process the logic with.

    If you are not seeing data being downloaded in some situation when you are connected and requesting a period of data, that is a different situation which we could focus on.


    I look forward to being of further assistance.
    JesseNinjaTrader Customer Service

    Comment


      #3
      Hi Jesse. Thanks for the reply and the good tips on how to see the data. I should mention that the strategy I'm testing uses four time frames, hence what I see on the chart when running the strategy does not always tell the whole story unless I open more charts separately.The question remains, why the need to see what data is there in the first place. If I run a test for Jan-Feb and have previously downloaded data only for Jan, the system should use the existing data for Jan + download data for Feb and show trades for both months instead of just for Jan.

      There is more to it than just the trades. When testing a 'mixed' period the test takes longer, irrespective of the period length. Sometimes it doesn't run at all, or maybe I just wasn't patient enough to wait (I did wait, long). The system clearly doesn't like it when a period is mixed, which in turn requires one to very carefully map (download) the data. For example if I downloaded Jan 1 to 14 and then Jan 16 to 31, I would never know that Jan 15 is missing.

      I suppose once all the data is mapped it is no longer an issue and soon I won't care enough to inquire about it. Nonetheless, the behavior of the system during mapping strikes me as being more developer-friendly than user-friendly.

      Having said that, there is plenty for me yet to discover and so I take into account that I'm still missing something here.

      If I may, a conceptual question to get a better grip on things: what is the difference between importing from file and using a live-feed (as far as historical data is concerned)? Is live-feed just another way to import? Is there no difference to do with the 'live' in live-feed?

      Thanks.

      Kind regards

      Comment


        #4
        Hello
        If I run a test for Jan-Feb and have previously downloaded data only for Jan, the system should use the existing data for Jan + download data for Feb and show trades for both months instead of just for Jan.
        Correct, if you are not seeing that we could look into that situation. The platform will generally download data so long as you are connected at that time to a provider which offers the data being requested. This should happen when using a chart or the other tools that require historical data.

        When testing a 'mixed' period the test takes longer, irrespective of the period length. Sometimes it doesn't run at all, or maybe I just wasn't patient enough to wait (I did wait, long).
        If the test does not run at all, that is a good indication that one of the series did not load data or that the logic the script uses was somehow not being met to place trades. You noted you use 4 series, so all 4 series would need to have data downloaded for that time period to run.

        For example if I downloaded Jan 1 to 14 and then Jan 16 to 31, I would never know that Jan 15 is missing.
        If you are testing specifically those date ranges, no you wouldn't know that the date was missing as it was not included. Generally if you plan to test over multiple areas in a span of time, it is best to just download the larger set of data. For example open a chart from Jan 1 to March 1 and then there is no issue with data not being downloaded. The next time you test it will just use the data you have or request it if there is missing data from the dates you choose. You can use a chart or the historical data manager to download large periods of data at once. You can also request the full length of time in a backtest to include all data to be downloaded.


        If I may, a conceptual question to get a better grip on things: what is the difference between importing from file and using a live-feed (as far as historical data is concerned)? Is live-feed just another way to import? Is there no difference to do with the 'live' in live-feed?
        When importing from file you only have that data once meaning if anything were to ever be updated on the actual data held by the historical data servers, you would not get that update. The historical data servers can be queried from a live data feed and then download the new data including any updates. Live data streamed in realtime is just saved as historical data and collected.

        The only major difference is that the servers can include updates to the data for example if some data was missing for some reason and was later inserted. Aside from that data by file may be available for further back periods, the servers generally contain a running total of data which never exceeds a set number of days/years depending on the type of data. By file you can import saved data as far back as you have data for, this could be much longer than what is available on the server depending on where you downloaded the data.

        I look forward to being of further assistance.

        JesseNinjaTrader Customer Service

        Comment


          #5
          Jesse, thank you for the detailed reply. I have a better grasp now for sure. My experience, meanwhile, is different from what you describe. Please see below screenshots -

          The first is a test Jan 1 - Jan 31 (daily chart). There are trades all throughout the period.
          Then I move the period two weeks forward in time to be Jan 15 - Feb 15. At this point, before the test is ran, there is downloaded data for January for all four series, there is downloaded data for February for the main price series, and there is no downloaded data yet for February for the other three price series.
          The result of this test as can be seen in the second screenshot - there are trades in January but not in February. The data for February for the three other series are never downloaded, as confirmed by checking the historical data manager.

          Click image for larger version

Name:	chart1.png
Views:	1
Size:	58.6 KB
ID:	1067267

          Click image for larger version

Name:	chart2.png
Views:	1
Size:	41.3 KB
ID:	1067268

          When I move the start of the test period to be Jan 31, the result is still the same. When I move it to be Feb 1, only then the data is downloaded and trades show up. This is what I meant by having to be precise when setting the dates.

          I think there might be a scenario here the developers have not considered.

          As for downloading everything in one go, that would be ideal but I would like to have data for four series starting as far back as provided, and I don't have the option to leave the machine running all day, so I have to do it piece by piece.

          As for the connection to the provider, it is not the problem. I have tested the above scenario more than once. The connection is always on and the result is always the same.

          Thanks

          Comment

          Latest Posts

          Collapse

          Topics Statistics Last Post
          Started by Newtrader101, Today, 03:54 PM
          1 response
          15 views
          0 likes
          Last Post NinjaTrader_ShawnB  
          Started by williamzz, Today, 03:28 PM
          0 responses
          7 views
          0 likes
          Last Post williamzz  
          Started by Robert87, Today, 11:07 AM
          0 responses
          13 views
          0 likes
          Last Post Robert87  
          Started by phta123, Today, 10:24 AM
          0 responses
          17 views
          0 likes
          Last Post phta123
          by phta123
           
          Started by sidviciousX, Today, 06:47 AM
          0 responses
          16 views
          0 likes
          Last Post sidviciousX  
          Working...
          X