Unable to import historical tick data

pfdtrader24 · 07-16-2008, 07:44 PM

I am unable to import historical tick data that I purchased from Disktrading on the web. It will begin importing the data and at some random percentage uploaded, it will give the same error message and need to abort. Not sure what is causing this or what to do. 1 min and daily historical data from the same source import just fine, no problems.

NinjaTrader_Ray · 07-16-2008, 07:49 PM

Please check the log tab, any specific error messages?

pfdtrader24 · 07-16-2008, 08:38 PM

I get one of 3 different errors.

1. Volume in line 1 (or 2) is invalid since it is <= 0. This line and subsequent invalid volume lines have been modified and replaced by a value of 1.

2. Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.

3. $EURUSD: Date/Time format error in line 32322846: Exception of type 'System.OutOfMemoryException' was thrown.: '20040123 152000; 1.2592,0'

NinjaTrader_Josh · 07-16-2008, 09:00 PM

Hi pfdtrader24,

How is your data formatted? From the last error message I think it is not formatted correctly. Please see this article: http://www.ninjatrader-support.com/H...V6/Import.html

NinjaTrader_Dierk · 07-16-2008, 10:36 PM

On (2) and (3): Try to split up the import data files, since you hit the wall due to RAM excess.

pfdtrader24 · 07-17-2008, 04:58 PM

Here is the format of the tick data (AUDUSD)

20000630 110100 0.6003 0

It all looks correct to me (except the 0 for volume?)

The file sizes are very large and I tried to split the file up in Excel, but couldn't get it to import more than the first 1,048,576 rows (which is only the first 2 years of so, out of an approx. 10 year file). I am not experienced with Excel, so forgive me if I am overlooking something obvious. Thanks in advance for the help.

NinjaTrader_Ray · 07-17-2008, 05:31 PM

Put in some dummy value in replace of the zero.

pfdtrader24 · 07-17-2008, 05:35 PM

Ray, I was just going to put a "1" in place of all zeros. Any idea as to how I can split up these large files to re-format/re-save them?

NinjaTrader_Ray · 07-17-2008, 07:35 PM

Just use any text editor, cut chunks out, put it into another file and save.

mason · 07-18-2008, 03:57 AM

Tip: use BK ReplaceEm http://www.orbit.org/replace/
It is a lot faster than notepad.

07-16-2008, 07:44 PM	#1
pfdtrader24 Junior Member Join Date: Jul 2008 Posts: 5 Thanks: 0 Thanked 0 times in 0 posts	Unable to import historical tick data I am unable to import historical tick data that I purchased from Disktrading on the web. It will begin importing the data and at some random percentage uploaded, it will give the same error message and need to abort. Not sure what is causing this or what to do. 1 min and daily historical data from the same source import just fine, no problems.
	Facebook Twitter YouTube

07-16-2008, 07:49 PM	#2
NinjaTrader_Ray Administrator Join Date: Nov 2004 Location: Denver, CO, USA Posts: 11,163 Thanks: 6 Thanked 45 times in 32 posts	Please check the log tab, any specific error messages? Ray NinjaTrader Customer Service Use Kinetick, NinjaTrader’s preferred market data service - Learn More Free online training events - View Schedule
	Facebook Twitter YouTube

07-16-2008, 08:38 PM	#3
pfdtrader24 Junior Member Join Date: Jul 2008 Posts: 5 Thanks: 0 Thanked 0 times in 0 posts	I get one of 3 different errors. 1. Volume in line 1 (or 2) is invalid since it is <= 0. This line and subsequent invalid volume lines have been modified and replaced by a value of 1. 2. Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown. 3. $EURUSD: Date/Time format error in line 32322846: Exception of type 'System.OutOfMemoryException' was thrown.: '20040123 152000; 1.2592,0'
	Facebook Twitter YouTube

07-16-2008, 09:00 PM	#4
NinjaTrader_Josh NinjaTrader Product Manager Join Date: May 2007 Location: Denver, CO Posts: 17,458 Thanks: 1 Thanked 106 times in 70 posts	Hi pfdtrader24, How is your data formatted? From the last error message I think it is not formatted correctly. Please see this article: http://www.ninjatrader-support.com/H...V6/Import.html Josh NinjaTrader Customer Service Use Kinetick, NinjaTrader’s preferred market data service - Learn More Free online training events - View Schedule
	Facebook Twitter YouTube

07-16-2008, 10:36 PM	#5
NinjaTrader_Dierk Administrator Join Date: Mar 2005 Location: Bamberg, Germany Posts: 9,994 Thanks: 0 Thanked 6 times in 6 posts	On (2) and (3): Try to split up the import data files, since you hit the wall due to RAM excess. Dierk NinjaTrader Customer Service Use Kinetick, NinjaTrader’s preferred market data service - Learn More Free online training events - View Schedule
	Facebook Twitter YouTube

07-17-2008, 04:58 PM	#6
pfdtrader24 Junior Member Join Date: Jul 2008 Posts: 5 Thanks: 0 Thanked 0 times in 0 posts	Here is the format of the tick data (AUDUSD) 20000630 110100 0.6003 0 It all looks correct to me (except the 0 for volume?) The file sizes are very large and I tried to split the file up in Excel, but couldn't get it to import more than the first 1,048,576 rows (which is only the first 2 years of so, out of an approx. 10 year file). I am not experienced with Excel, so forgive me if I am overlooking something obvious. Thanks in advance for the help.
	Facebook Twitter YouTube

07-17-2008, 05:31 PM	#7
NinjaTrader_Ray Administrator Join Date: Nov 2004 Location: Denver, CO, USA Posts: 11,163 Thanks: 6 Thanked 45 times in 32 posts	Put in some dummy value in replace of the zero. Ray NinjaTrader Customer Service Use Kinetick, NinjaTrader’s preferred market data service - Learn More Free online training events - View Schedule
	Facebook Twitter YouTube

07-17-2008, 05:35 PM	#8
pfdtrader24 Junior Member Join Date: Jul 2008 Posts: 5 Thanks: 0 Thanked 0 times in 0 posts	Ray, I was just going to put a "1" in place of all zeros. Any idea as to how I can split up these large files to re-format/re-save them?
	Facebook Twitter YouTube

07-17-2008, 07:35 PM	#9
NinjaTrader_Ray Administrator Join Date: Nov 2004 Location: Denver, CO, USA Posts: 11,163 Thanks: 6 Thanked 45 times in 32 posts	Just use any text editor, cut chunks out, put it into another file and save. Ray NinjaTrader Customer Service Use Kinetick, NinjaTrader’s preferred market data service - Learn More Free online training events - View Schedule
	Facebook Twitter YouTube

07-18-2008, 03:57 AM	#10
mason Member Join Date: Feb 2008 Posts: 47 Thanks: 0 Thanked 0 times in 0 posts	Tip: use BK ReplaceEm http://www.orbit.org/replace/ It is a lot faster than notepad.
	Facebook Twitter YouTube

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Use standard CSV format for historical data import	Iggy22	Suggestions And Feedback	8	10-17-2010 10:43 PM
Import Historical Data	cls71	Miscellaneous Support	11	05-29-2009 06:36 AM
Historical data import question	iplaygames	Miscellaneous Support	7	05-13-2008 08:01 AM
Issue Import Historical Data	cls71	Miscellaneous Support	2	04-23-2008 12:25 PM
import/export historical data	ATI user	Miscellaneous Support	11	04-17-2008 10:39 AM