Go Back   NinjaTrader Support Forum > NinjaTrader 7 > General Development

General Development General NinjaScript development questions.

Reply
 
Thread Tools Display Modes
Old 05-16-2011, 01:40 PM   #1
8DTK8
Senior Member
 
Join Date: Jun 2008
Posts: 179
Thanks: 2
Thanked 0 times in 0 posts
Default Corrupt data files

I haven't done any backtesting since the upgrade to NT7.
After a few days of getting back into it for the first time in a while, I noticed that large chunks of my data are missing. I never checked to see if the database migrated properly when I went from 6.5 to 7.

The last backtests I ran were working great with both daily and minute data since 1995. Now There are random periods as long as a month missing and it will only go back to 2007.

Any ideas what happened or more imoportantly how I can fix this?

Alternatively, if someone can inform me as to where I can get day and minute data for (ES, NQ, GC, ZB, ZN, and most of the currency) futures, I would greatly appreciate it.

Thanks in advance.
8DTK8 is offline  
Reply With Quote
Old 05-16-2011, 02:07 PM   #2
NinjaTrader_RyanM1
NinjaTrader Customer Service
 
NinjaTrader_RyanM1's Avatar
 
Join Date: Sep 2009
Location: Denver, CO
Posts: 8,127
Thanks: 249
Thanked 434 times in 427 posts
Default

Hello 8DTK8,

Unfortunately there's no way to know what happened to your data. Historical data is provided by our supported brokers/data providers or can be imported if you already have txt files available You will get the most historical data with a data only provider like Kinetick, eSignal, or BarChart.

A complete list of our providers and the level of data supported for each is available here:
http://www.ninjatrader.com/support/h...rical_data.htm

If you have text files available, you can import if the're formatted correctly for NinjaTrader:
http://www.ninjatrader.com/support/h.../importing.htm

To prevent running into situations where you can potentially lose important data, we advise to run regular backups via our inbuild backup feature in NinjaTrader. You can include historical chart data in this backup. http://www.ninjatrader.com/support/h...up_archive.htm
NinjaTrader_RyanM1 is offline  
Reply With Quote
Old 05-16-2011, 05:20 PM   #3
8DTK8
Senior Member
 
Join Date: Jun 2008
Posts: 179
Thanks: 2
Thanked 0 times in 0 posts
Default

Thanks RyanM.

I just spoke with someone at eSig that told me that their daily data goes back 25 years, but their min. data only goes back to 2007.

Realizing tha my laziness will come shining though with my next request, does anyone know who has the most comprehensive (goes the furthest back) daily and minute data for futures out of the providers listed? http://www.ninjatrader.com/support/h...rical_data.htm

Thanks in advance.
8DTK8 is offline  
Reply With Quote
Old 05-19-2011, 02:41 PM   #4
NinjaTrader_Tim
NinjaTrader Customer Service
 
NinjaTrader_Tim's Avatar
 
Join Date: Nov 2009
Location: Denver, CO
Posts: 1,424
Thanks: 0
Thanked 3 times in 3 posts
Default

Hi 8DTK8,

Kinetick offers extensive historical data. Including at least 120 days of historical tick data, two years of minute data, and ten years of daily data. Much more for popular instruments, for example the ES minutes data goes back to 2005.
NinjaTrader_Tim is offline  
Reply With Quote
Old 05-19-2011, 04:22 PM   #5
Camdo
Senior Member
 
Join Date: Aug 2009
Location: Massachusetts, USA
Posts: 225
Thanks: 11
Thanked 1 time in 1 post
Default

My data file was corrupted in similiar manner. The cause was traced to reloading data from a chart with a range beyond what the data provider provides (ie a three year chart was reloaded with right click, but the data provided only supplies 6 months of minute data). The solution is to reduce the range of the chart, then reload, then expand the range back to what it was. My provider is MB Trading.

This is a bug in NT which should be addressed.
Last edited by Camdo; 05-19-2011 at 04:25 PM.
Camdo is offline  
Reply With Quote
Old 05-19-2011, 04:47 PM   #6
NinjaTrader_JoshP
NinjaTrader Product Manager
 
NinjaTrader_JoshP's Avatar
 
Join Date: May 2007
Location: Denver, CO
Posts: 17,459
Thanks: 1
Thanked 293 times in 98 posts
Default

Camdo,

This is not a bug, but unfortunately is indeed a limitation to the way NinjaTrader requests data. When you explicitly try to do a download of historical data based on a date range, NT achieves this in several steps.

1. Removes old data from the range you chose
2. Downloads what it can download from the provider
3. Repopulates the range with what it was able to download

If you request a range longer than what the data provider offers it would not repopulate those sections.

There is no logic in place to allow us to detect what the boundary is for what a certain data provider does or doesn't offer.
NinjaTrader_JoshP is offline  
Reply With Quote
Old 05-19-2011, 05:24 PM   #7
Camdo
Senior Member
 
Join Date: Aug 2009
Location: Massachusetts, USA
Posts: 225
Thanks: 11
Thanked 1 time in 1 post
Default

AhHA !
In that case I would suggest a warning message pop up to remind the user of the range issue and lost data consequenses you mentioned above, and allow the user a chance to opt out before the reload is executed. It is too easy to forget chart range could corrupt a database.
Camdo is offline  
Reply With Quote
Old 05-20-2011, 12:06 PM   #8
NinjaTrader_JoshP
NinjaTrader Product Manager
 
NinjaTrader_JoshP's Avatar
 
Join Date: May 2007
Location: Denver, CO
Posts: 17,459
Thanks: 1
Thanked 293 times in 98 posts
Default

Camdo,

Thank you for the suggestion. This limitation is already on our list for things we would like to remove in a future version of NT.
NinjaTrader_JoshP is offline  
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
corrupt data deanldavis475 Platform Technical Support 1 02-26-2010 05:55 AM
Has anyone compiled Gain's weekly tick data files into larger files? bluelou Platform Technical Support 0 06-18-2009 10:07 PM


All times are GMT -6. The time now is 04:02 PM.

Futures, foreign currency and options trading contains substantial risk and is not for every investor. An investor could potentially lose all or more than the initial investment. Risk capital is money that can be lost without jeopardizing ones financial security or lifestyle. Only risk capital should be used for trading and only those with sufficient risk capital should consider trading. Past performance is not necessarily indicative of future results. View Full Risk Disclosure.

CFTC Rules 4.41 - Hypothetical or Simulated performance results have certain limitations, unlike an actual performance record, simulated results do not represent actual trading. Also, since the trades have not been executed, the results may have under-or-over compensated for the impact, if any, of certain market factors, such as lack of liquidity. Simulated trading programs in general are also subject to the fact that they are designed with the benefit of hindsight. No representation is being made that any account will or is likely to achieve profit or losses similar to those shown.

This website is hosted and operated by NinjaTrader, LLC (“NT”), a software development company which owns and supports all proprietary technology relating to and including the NinjaTrader trading platform. NT is an affiliated company to NinjaTrader Brokerage (“NTB”), which is a NFA registered introducing broker (NFA #0339976) providing brokerage services to traders of futures and foreign exchange products. This website is intended for educational and informational purposes only and should not be viewed as a solicitation or recommendation of any product, service or trading strategy. No offer or solicitation to buy or sell securities, securities derivative or futures products of any kind, or any type of trading or investment advice, recommendation or strategy, is made, given, or in any manner endorsed by any NT affiliate and the information made available on this Web site is not an offer or solicitation of any kind. Specific questions related to a brokerage account should be sent to your broker directly. The content and opinions expressed on this website are those of the authors and do not necessarily reflect the official policy or position of NT or any of its affiliates. 

Vendors along with their websites, products and services, collectively referred to as (“Vendor Content”), are independent persons or companies that are in no manner affiliated with NT or any if its affiliates. NT or any of its affiliates are not responsible for, do not approve, recommend or endorse any Vendor Content referenced on this website and it’s your sole responsibility to evaluate Vendor Content. Please be aware that any performance information provided by a vendor should be considered hypothetical and must contain the disclosures required by NFA Rule 2-29(c). If you are interested in learning more about, or investigating the quality of, any such Vendor Content you must contact the vendor, provider or seller of such Vendor Content. No person employed by, or associated with, NT or any of its affiliates is authorized to provide any information about any such Vendor Content. Visit the CFTC resources for education regarding the industry and signs of fraud.