Chart not displaying tick data after historical tick data import

[Big D]

Problem: Chart will not display sub-minute data from a synthetic symbol created to hold imported tick data. Sub-minute data is anything less than 1 minute, ie tick, volume or second bars. Anything greater than 1 minute, ie 1 minute, 5 minute, 60 minute displays fine.

Summary #1: Created synthetic symbol, ZES 12-07, to hold imported ES tick data and avoid conflicts with real-time ES 12-07 symbol. Tick data ranges from Jan 1st 2006 to Sept 30 2007. (Had to create 3 separate import files to import that much tick data to avoid "out of memory" errors during the import.)

Summary #2: Data was imported with no errors. Data exports with no errors. All the tick data exists in Ninja Trader database. Entire 19 months of data is stored in one synthetic symbol ZES 12-07. In the instrument manager, ZES is equivalent to ES except, the exchange is set to default and the mapping is set to a non-existing symbol. This prevents Ninja from attempting to download real-time data. Data was imported with the intention of using it for backtesting and optimization.

Summary #3: My guess is that charts work with 1 minute or greater intervals because 1 minute bars were greated during the import by Ninja. Perhaps, tick based chart intervals will not work because of the shear amount of data, 34 million ticks over 19 months.

Question: If tick data is imported into a synthetic symbol, can a chart display bars using tick data and not just 1-minute or greater intervals? Is this a known problem in Ninja or am I doing something wrong?

[NinjaTrader_Ray]

If the symbol holds tick data, then it should display tick based bars.

To confirm, if you export data for this synthetic symbol (Tools > Historical Data > Export) and select Bar Type "Tick" does the exported file contain data?

[Big D]

Yes, tick data exports fine. It is not a database issue.

I'm now convinced that it is a chart memory issue. I've imported 3 instruments, ES, ER2, and YM. The YM has the least amount of ticks, and I am able to view some (around 10 months) of the 19 months of tick data in tick, volume, or second bars. The ER2 has more ticks and I am able to view a little less (around 4 months). The ES has the most ticks and I am able to view only 1 or 2 months.

When trying to view more data I usually receive a "Please select a smaller time frame" error message or a "Historical data is corrupt" error message, which it is not, or I receive no message at all.

When trying to load lots of data there is no indication that the chart is doing anything. I have no idea if it's hung or if it's taking a long time.

Is there some hard-coded limit with the amount of ticks that can loaded into a chart? If there is, I would recommend removing this limit or allowing the user to set this limit. Now I realize that a normal 32-bit OS has an effective 2GB limit per app, however, Ninja never comes close to this limit. If I really push Ninja, it peaks at 1.5 GB. With 64-bit OS coming into their own now, 4GB is easily available to an app.

Also, it is not possible to view this synthetic symbol (even with 1 minute or greater intervals) when connected to a real-time broker, such as Zen-Fire. When opening a chart and selected the sythetic, the charts attempts to load data and never gives control back to the user. I thought that by not giving a correct mapping symbol in instrument manager, the load would eventually stop, but I guess not.

So any thoughts on how to deal with lots of tick data. The primary purpose is to historically backtest 1-2 years of tick based bar types.

[NinjaTrader_Dierk]

>> The primary purpose is to historically backtest 1-2 years of tick based bar types.
Unfortunately this will not work, since it's too much data to handle. See also here: http://www.ninjatrader-support.com/v...ght=tick+bytes

Minute data is fine though, the problem only is with the masses of tick data.

[Big D]

Ray and Dierk,

I read the post concerning the amount of memory a tick bar requires and I understand that dealing with large amounts of data in an application is a difficult thing to do, however I would highly recommend that you program/redesign Ninja to be able to handle larger amounts of data. Why? Because that is what traders need and that is where this platform space is going.

Ninja is a very good application for manual trading and real-time automated system strategies, however it is not very good for backtesting/optimization, primarily because it's database design and memory usage is poor.

I applaud you for giving us the ability to create custom bar types in the next release (v6.5), however the one thing that custom bars need is raw tick data. Even the built-in bar types, tick and volume bars, need raw tick data. As traders, we do not want to deal with large amounts of data, we are forced to. Using non-time based bars allows us to cut down on the amount of data displayed on a chart, however if Ninja cannot handle even 1 year of ES tick data, how can we create custom bars and do a simple 1 year backtest?

There are 2 things that Ninja can do to deal with larger amounts of data:

1) Go to 64-bit. I currently use XP 64-bit and have 8 GB of RAM. I'm frustrated that none of my apps can take advantage of this RAM, because they're all still 32-bit. I know it's a pain to deal with 2 code bases and I know none of the data vendors provide 64-bit plug-ins, but you must work around these issues. Traders have the money to buy good hardware, and you must design software that allows us to take advantage of this hardware.

2) Create a database conversion tool. Storing ticks on disk is no problem, but storing it in RAM is. If Ninja could store raw ticks and then convert these ticks into non-time based bars, ie tick or volume or custom, before loading into RAM for display on a chart, this would solve most of the problems. (You already do this by converting tick to 1 minute.) Remember this is mostly a backtesting issue. (Real-time trading needs at most 5 days of tick data.) The conversion could be a one time event, ie non-time based bars are converted and stored in the DB, or the conversion could be on demand, ie the non-time based bars are converted and stored in a temp table, then displayed on the chart. Either way, the ticks are not loaded in RAM.

In summary, Ninja must be programmed to handle larger amounts of tick data for the following reason: We traders need this functionality to stay competitive and providing this will allow Ninja to stay competitive in this real-time trading/backtesting platform space.

[NinjaTrader_Dierk]

Thanks for your suggestions. We'll add them to the list.

>> I applaud you for giving us the ability to create custom bar types in the next release
Unfortunately this is beyond the scope of what we support.

[Big D]

Dierk,

As a work around, I am able to create my own custom bars outside of Ninja, however I cannot use the historical data import tool to import these bars because:

Ninja assumes, during import, that all OHLCV bars are 1 minute bars. My custom bars, for example 100 tick bars, will have instances where 2 successive bars will have the same minute timestamp. Of course most of the time, they will have different second timestamps, but theoretically could have the same second timestamp. During the import, Ninja gives an error message: "ES 12-07: Time stamp line '57' is smaller/equal than timestamp of previous line." In this particular case, line 56 = 072104 and line 57 = 072140.

Would you give us the ability to selectively allow/disallow this timestamp checking? For normal 1 min imports, it is important to not have duplicate timestamps, however for custom bar imports, not checking at the minute level or perhaps not checking at all is desired.

As 6.5 beta is not yet released, I feel you could throw this feature in very easily, while the time is right. Otherwise, as it stands, Ninja is useless for backtesting on tick, volume, or custom bars, as you can only test 6 months or less of any heavily traded symbol, ie ES, ER2, Euro Stoxx, etc.

[NinjaTrader_Dierk]

Thanks for your suggestions. We'll add them to the list.

[Big D]

Is anyone else frustrated with Ninja's inability to import large amounts of tick data or import custom bar types? Does this affect your ability to backtest in Ninja? Or does everyone pretty much use other tools for backtesting and only use Ninja for real-time trading?

The sooner Ninja provides a 1-stop shop for backtesting and real-time trading, the sooner we can all stop using 2 platforms and wasting time converting code from our backtesting platform to our real-time platform.

Now is your chance to let Ray and Dierk know about your concerns. They do listen to their customers.

[PolarBear]

Is This Fixed?

Hi all,

I notice these posts were a few years ago. I'm assuming / hoping that this problem has been fixed. I've purchased a bunch of tick data for the ES for the past 5 years, and have spent all day generating the files required for the import into ninja. I've finally done this, and had a successful import, however when i goto the chart to look at the tick data i don't see anything.
I've imported tick data from 2003-2009 so there's lots of data,
However, i was just trying to do look at a chart say around mid 2005, of 5000 bars, 233 Tick chart.
Nothing comes up.

I tried the export thing to verify data was loaded. Found that there was no data loaded for the later timeframes, however the data from earlier in 2003 was there (ie: i could export it). I then tried to plot that timeframe and i got something up.

So,
my queries are:
1) i'm wondering if there's anything i should know about importing large tick files (ie: this is data from 2003-2009 and contains 145,000,000 lines of data) - will there be a problem importing this amount of data?? I did import it all, it took awhile, but went through all the dates, etc, and i had the success msg at the end, so it appeared to wrk. However it seems like i'm missing data.
2) Are the above problems fixed, and are there any problems using tick data for backtesting?? The whole idea was i wanted to create and backtest strats, using tick charts rather than the minute charts i've been testing with so far.

thanks in advance

[Big D]

PolarBear,

NinjaTrader is not a very robust platform for backtesting. NinjaTrader backtesting can only handle a few days of tick data for a fast market like the ES for any simple optimization run. I highly recommend you use some other program. I use Amibroker and a SQL Server database and that works for me, but there are others out there.

In fact, not very many programs will work with the 145 million lines of tick data that you are looking at. You will at an absolute minimum need to convert this tick data to some other bar type in order to use in pretty much any backtesting platform. Dealing with large amounts of raw ES tick data is just too overwhelming.

BigD

[PolarBear]

thanks for the reply... .

i REALLY hope this is not the case. How can a platform marketed as a trading development platform only be able to handle a few days of tick data. This seems unbelievable.

wrt to the comments in the previous post in particular. I did load up the tick data (150 tick chart) from about a year's worth of data 2003-4. So it may be possible now in the later version since you were previously testing. However, looks like some memory problem, or something's gone funny during the import process.

I purchased the lifetime license of ninja trader, and have been developing 100's of indicators, strategies, money mangament techniques etc in C# ninja code over the past 2 years, primarily looking at between 1->60min bar data.

I've only just made the connection, that a lot of my techniques could work better if i focus on tick charts, for example 233 tick chart i see popping up a lot around the place, from others' discussions.

This will really, REALLY put me back timewise, and money wise, if i have to convert to a different platform.
So far I like everything about ninja trader.

Can some ninja staff please comment on this tick issue?? I need to get this resolved asap.
thanks.

PS. restarted ninja, and tried to reload the data file when i went to bed last night. Checked on my pc this morning, and it had this out of memory error in the log:

6/12/2009 1:23:44 AM,Default,ES ##-##: Date/Time format error in line 32997303: Exception of type 'System.OutOfMemoryException' was thrown.: '20050830 090407;01208.75;50',

Can't seem to see any tick data now when i try to plot the graph.

[Big D]

Realtime trading development and backtesting/optimization (bt/opt) are two different things. NinjaTrader is great for realtime trading, the development of realtime trading strategies, charts, indicators, etc., but not so good for backtesting/optimization. NinjaTrader can easily create a chart with many days of tickdata, but it cannot run a bt/opt with any normal amount of parameter permutations and any normal amount of tickdata, say 1 month of ES tickdata. NinjaTrader will quickly run out of memory as you've already found out. The only solution is to restart NinjaTrader and that gets really old after a while.

There are a couple problems with bt/opt: 1) NinjaTrader doesn't store tickbars natively. It stores tickdata and creates tick bars on the fly. 2) There's a memory leak, where each bt/opt run doesn't reuse the memory or even release the memory after each run and you eventually hit the 32-bit app memory limit of 1.4 GB.

NinjaTrader is attempting to address these issues in their version 7, which is still in private beta. It's up to you to decide if you want to wait for version 7 or move to a new bt/opt platform.

[PolarBear]

Hi Big D,
thx for the quick reply, and interesting read.

I wonder if the ninja staff can comment on this issue wrt the new version 7. I know we've been waiting for awhile for it's release.
Any info on:
- when this might be?
- will the memory tick issues be addressed?
- will the new version be more suited to backtesting and optimisation (in particular using tick charts)
- Can I get a pre-release download of the beta 7 version so i can determine whether it will be adequate for my testing.

thanks

[PolarBear]

Could the ninja trader moderators please address my questions.

thanks