Database file too large

[LTWCI]

I have Win 7 also and I install manually. Yesterday NT worked fine that is why I am suspicious of the update.

[egan857]

Hi Textheis,

You asked the question, would we ever run out of database space in a live account? My answer is that it would happen much less frequently. When I have the system running flat-out in simulation mode, it places more trades in one day than I placed all last year using NT and manual trades. But I cannot foresee a situation where I'll run NT in live mode exclusively for an entire year; I always run sims while I'm actively trading, and then I run the sims with looser constraints (running flat-out) when I'm not trading. In flat-out sim mode I probably eat up 175MB a day; in sim+live trading, closer to 50MB a day; in live-only, about 10MB a day. In live-only mode, I'd be resetting the database at least twice a year.

I don't want an unlimited database size - it's good to do some housecleaning every once in a while to keep the system running smoothly. I just don't want to have to do it every day.

[textheis]

my sim mimicks my real trades

E,
That's some great food for thought. I pondered this much. I'm still wondering though....my sim action is no where near yours. In fact, I only sim what I would actually trade live + about 20% more activity. That comes to somewhere between 10-30 trades a day depending on price action. Given that we arrived at the same DB constraint under two different "use models", I'm still not sure what would actually happen in a live trading environment.

My questions still linger.

And with your set of circumstances being different than mine, I add:
Why did this happen to me at this particular time?
Given the same amount of sim activity over the past year, is there anything that I'm unknowingly doing with NT that has kept me from running into this cap? i.e. what causes the DB to stay right sized (if anything) since you have identified what causes it to overflow 257M?

[egan857]

Hi Textheis,

My database usage increased dramatically when I changed my strategy to float offers constantly a few cents above the current price (for a long position) until I'm ready to get out, when it moves the offer down to the bid. It doesn't move the exit price on every single price move, but often enough to be a lot of activity that NT has to track.

If you have made changes to your strategy recently that increase the number of attempts to enter a position, even if it doesn't actually get in, or updates the limit prices before exiting, that would have increased your usage enough to blow past the 256MB limit. Similarly, if you recently went from tracking 100 symbols on Kinetick to 200 or 500 symbols, your activity might have increased dramatically. Or, you could have just crept up on the 256MB gradually over the past year and just happened to hit it now.

My strategy is very very noisy. It's literally 150+ entry strategies, and about 20 different exit strategies, all in one program. My program is 11,000 lines long, and about 4,000 of those are just for creating all kinds of indicators that are used in my entry strategies.

One thing I don't do anymore is print statements to the log (I send them out to text files, which is why I have no problem wiping the database clean daily). It's possible that doing Print() statements will add to the database usage - I know they add to the size of the trace files and increase memory usage. If you've added a bunch of prints to find an error and then never disabled them, that might have had an impact on your database usage.

If your database has been growing at an increasing rate, my money is on an increased number of changes to your limit orders. Also, chances are very high that you'll never get to the point I'm at with database housecleaning.

[textheis]

Tried deleting the DB....

Egan,
After a nice Memorial Day observance, I came back this morning only to find the DB over sized again. This time I chose to restore using your method of deleting and restarting. That works really well, except I lose my ATM strategies (with only a handful it's easy enough to recreate) and my custom session templates which I noticed after the open - that stopped me in my tracks. So, I went back to rename old to current and started. Repaired and reset and I'm now running a DB less than 200M. Why the clean up decimated more than 65M of items I have no idea, but it ran smooth the rest of the day.

As it was a good question earlier as to whether a change in my strategy creates more entries is not the cause as I haven't really changed a whole lot other than adding some different ATM's and not cleaning up the list. So "why me" still remains in my mind an unanswered question.

Having said all that, my version of NT is stable and now I understand a whole lot more about this program than I thought I would need - but I believe it's better to know about this during the calm of sim than be unpleasantly surprised with real bullets. Many thanks for your discussion - it's really helpful when you're on our own with this chosen activity.

[egan857]

Hi Textheis,

Sorry for the delay. If after restoring you have over 200MB, that's a problem - you're still beating your head on the ceiling. I'd recommend practicing backing up and recovering the ATM strategies and templates (and indicators and workspaces?), and then wipe the database clean and recover the other stuff. Unfortunately I don't use ATM strategies so I don't know what's involved. It may take a while to do the backup and recovery, but at your burn rate it'll probably be another year before you need to do it again. You'll never do anything with that 200MB so why not throw it away?

I don't know why the db repair wiped out 65MB - probably just getting rid of temp tables or removing inefficiencies from too many extents. It was probably all just blank or unusable space. My suspicion is that NT was designed with a different kind of user in mind - someone who trades one currency or e-mini futures and only places a few trades a day. That's probably why they thought 250MB would be plenty of space. Oh well, it works great for equities traders like me who have never met a stock they didn't like. I would prefer if they increased the database limit or removed all but the bare minimum of information after a trade so we could track the historical performance, but NT is a whole lot better than the alternatives (AmiBroker, SmartQuant).

[kujista]

Hello egan did you solve the problem? i reach 256MB of db every time a run strategy and do not know how to solve this...

[NinjaTrader_Jason]

Hello kujista,

Your strategy likely submit many orders, makes many order modifications, etc. I can only suggest to limit these actions, so the database will not grow that big that fast.

You could rename the database to start fresh as mentioned previously in this thread.

[kujista]

No no... DB grows even when strategy makes NO trade and DB is empty.. this all grows just when I generate the strategy object (list of c# objects inside)... no trades, no data.. pure ninja and added strategy on last 10 days...

[NinjaTrader_Bertrand]

kujista, have just replied to your other thread on the topic with our thoughts.