I work on a python based video downloader/player and I'm working on
replacing a pickle based on disk database with a berkely db with a
pickle per object (so that I can update each object without having to
rewrite the whole database.)

So, I started by creating my database without using any of the
transaction stuff, but then I had some corruption problems. So I
turned on transactions and logging and such, and now it seems to be
proof against crashing, but the log files get very large very fast. As
in 450MB within an hour or two of actively using the program.

The question is, what am I doing wrong and how do I fix it? Creating
such a huge database is not acceptable, and neither is losing data.
Can berkeley-db fill my needs in this case?

So, how do I currently create the database? I create the dbenv and I
call set_flags with DB_AUTO_COMMIT | DB_TXN_NOSYNC. Then I open the
environment with DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER

I use a transaction the first time I create the database, but after
that I depend on DB_AUTO_COMMIT. I use a cursor when loading the
database, but close it when done.

My behavior during use consists of lots and lots of writing of the same
key with updating values.

Finally, I close the db and the dbenv on program shutdown.

* christopher james lahey:

If losing data is not acceptable, you really should archive those log
files. 8-)

You need to run checkpoints periodically. After that, some log files
can be safely removed (if necessary after archival). See:

<http://www.sleepycat.com/docs/ref/transapp/checkpoint.html>
<http://www.sleepycat.com/docs/ref/transapp/archival.html>

Thanks for the pointers. This helps tremendously, but I wanted to
confirm my solution below, if you don't mind checking it out.

Florian Weimer wrote:
Well, it's not acceptable to lose data because of a crash or a machine
losing power. Dealing with disk corruption is beyond the scope that I
have to deal with. It's all about user expectations. The user can
copy all the files of the database while the app is not running and
that will suffice.

So, I've coded this up and all looks good, but there is much talk of
catastrophic failure. I'm pretty sure this isn't a big deal as in the
case of catastrophic failure, the enduser wouldn't have the tools to
fix the problem anyway. Does catastrophic failure mean disk
corruption? The sequence I've coded is stop database transactions,
dbenv.txn_checkpoint(), db.sync(), delete all files returned by
dbenv.log_archive(), allow database transactions again. Is this safe
in the cases I described above (crash, power outage)?

Hubert Schmid wrote:
I tried this but it didn't seem to remove them. The code I have to
remove them by hand works just fine.

The database access is single threaded anyway, so stopping transactions
is automatic, but that's useful to know anyway.

The final thing I'd like to do is to make the log files smaller. As it
is, it builds up to 10MB, but if I could delete the log file every 1MB
or 5MB, that would be more space efficient, in exchange for being a bit
less time efficient. I looked and didn't find anything, so I suspect
this isn't possible, so if not, it might be something useful to add in
the future. If it is possible, even better. However, this is just
icing on the cake. How it is now is great.

Thanks again for all the help. This has made using berkeley-db
possible for this project.

It sounds like you are looking for set_lg_max(size), search for the API
name here for more info:
http://pybsddb.sourceforge.net/bsddb3.html

- Alex

christopher.james.lahey (AT) gmail (DOT) com writes:

You are doing lots of modifications to the database. Transaction logs
grow with each modification, and at a rate roughly proportional to the
length of the changed bytes in the database.

Do less modifications, or smaller ones.

best regards
Patrick

Alex wrote:
Ah, I didn't find that because I was looking at
http://www.sleepycat.com/docs/api_c/env_list.html . From now on I'll
use http://www.sleepycat.com/docs/api_c/api_core.html for my list of
available APIs or perhaps the python page you listed. Thanks for the
pointer on set_lg_max. This is working great. It's awesome that every
time I ask for a feature, you guys just say, "That's already there." I
think I've got it all working exactly as I want it.

Thanks much,
Chris

* christopher james lahey:

Log removal can only occur after checkpoints.

I hope you query Berkeley DB to get a list of obsolete log files,
instead of guessing which ones to delete. 8-)

Call DB_ENV->set_lg_max, or put an appropriate directive into the
DB_CONFIG file. See <http://www.sleepycat.com/docs/ref/log/config.html>