• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Importing data from a very large csv excel file

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Importing data from a very large csv excel file

    Hi Ninjas,

    I've found a reliable source of historical data which I'd like to import in my database. This file comes as a CSV Excel file, and its data comes in a non-format NinjaTrader, therefore I have to edit it before converting it to a .txt file and later importing it into NT's database.

    So, this is the situation: I need to edit each line of that file cause it comes in the following format: EUR/USD,20130401 00:00:00.258,1.29885,1.2989

    In order to edit this data into NT format data as: 20130408 040000;1.29885;1000000

    I'd need a software capable to open a massive csv file ( 467 MG ), and editing automatically for millions of lines. I've seen software capable to open massive csv files but to edit some lines "on demand" and not the whole file.

    Is the original data format, some format known by you ( by NT ) which could easily import ? or must I look for to do all this job of editing, converting...etc ?

    I'd like to hear any suggestions from your experience and knowledge to achieve this

    Thanks in advance
    Last edited by pstrusi; 09-24-2013, 06:23 AM.

    #2
    Hello pstrusi,

    I am not sure what you mean by: "Is the original data format, some format known by you ( by NT ) which could easily import ?" but the only import formats that NinjaTrader accepts natively is the ones on our Help Guide at the following link that it looks like you are aware of.

    http://www.ninjatrader.com/support/h.../importing.htm


    The only other way you can import it without converting the file would be to write an Import Type that will read a file like that so when you get data in that format you just have to select that Import Type. A few members have written some Import Type files that you may view at the following link.
    http://www.ninjatrader.com/support/f...ks.php?catid=6

    Let me know if this makes sense or if you can clarify about the "known format".
    JCNinjaTrader Customer Service

    Comment


      #3
      Thanks for your response JC.

      I'm sorry not to have properly explained what I meant. So, let me explain myself:

      When I export data from NT, the coming file is a .txt file with the following format:
      20130408 040000;1.29885;1000000 That's what I called the "original NT format". Now, in the market there are several types of format, many of them come in CSV files, with a format like this:
      EUR/USD,20130401 00:00:00.258,1.29885,1.2989

      I've read the NT help guide and basically it accepts the typical .txt file with the NT classic format, or others like Metastock or TradeStation; so my conclusion is that I need ac .txt file with the classic NT format, so I had to edit this massive file that I have as CSV excel file, but it's so large that I cannot do it through excel...so I ask you for your help:

      What software can I use ?

      Comment


        #4
        Hello pstrusi,

        Thanks for the clarification.

        I have seen people use "Notepad++" to be able to do this but I am not sure what the exact size limit is which you may try to use since it is free at the following link:
        http://notepad-plus-plus.org/

        The one that is the most common that I use as well for large files is going to be "Visual Studios Express 2012" which is free to use.
        http://www.microsoft.com/visualstudi...d-2012-express

        Happy to be of further assistance.
        JCNinjaTrader Customer Service

        Comment


          #5
          Thank you JC for your help, hopefully these suggestions could help me

          Comment

          Latest Posts

          Collapse

          Topics Statistics Last Post
          Started by faustf, Today, 05:00 AM
          0 responses
          5 views
          0 likes
          Last Post faustf
          by faustf
           
          Started by NinjaCustomer, Today, 03:19 AM
          1 response
          6 views
          0 likes
          Last Post NinjaTrader_ChrisSch  
          Started by MarianApalaghiei, Today, 02:54 AM
          1 response
          5 views
          0 likes
          Last Post MarianApalaghiei  
          Started by RandanAL, Yesterday, 11:58 PM
          0 responses
          5 views
          0 likes
          Last Post RandanAL  
          Started by Kabua, Yesterday, 11:24 PM
          0 responses
          5 views
          0 likes
          Last Post Kabua
          by Kabua
           
          Working...
          X