Dear all, I dont understand why the log generated by Berkeley DB is so
huge. I run the program that only has one transaction but the log is
10MB big. I know I can set log size but is thee a way to set it to the
necessary minimum? For ex if only 10KB is necessary then set it to 10KB
automagically. I'd like to see complete log records but a huge log file
often gives rise to "File size limit exceeded" the second time I run
the program. Maybe I am doing so mething wrong.

Thank you so much~

Hi,

Berkeley DB preallocates log files (by creating the file then writing a
byte at the end), which leads to much better performance on some
systems. Most systems support this kind of thing without allocating
disk space for all of the empty space in the middle of the file.

If you are on a Unix-like system, try the "du" command to work out how
much disk space is really being used.

Can you check whether you have a file size limit set with "ulimit -f"?
If so, try either increasing that limit or reducing the log file size
with DB_ENV->set_lg_max.

Regards,
Michael.

Thanks for your reply. I dont know why, but I run into this error very
frequently:
Filesize limit exceeded (core dumped)

I only execute the following operations in one transaction (Unix
running java):
cursor.put(theKey, theData)
cursor.getNext(theKey, theData, LockMode.DEFAULT)

When I run it the first time, it's fine although getNext() returns
failure which makes sense because put() hasn't flushed result to
database yet. However when I run it the second time it gives the
aforementioned error. I really dont know which file it is talking
about..

Sorry about bothering you but I am really lost. Thank you.

Hi,

That error is not generated by Berkeley DB, it must be some constraint
on your system. Does running "ulimit -a" give any clues about the
problem? Do you have a disk quota on this system (try the command
"quota"). If not, is there a system administrator you can contact?

If there is some limit on the size of files that you can create (you
could try creating a 100MB file using standard tools like "dd" and see
whether that fails in the same way), and you believe that log files are
the culprit, try using the DB_ENV->set_lg_max method to reduce the size
of individual log files.

Regards,
Michael.

Thanks for reply. Well there is no "ulimit" command on my unix machine.
Since I use java, I need to use setMaxLogFileSize(1000000) //1MB
But still I got "Filesize limit exceeded (core dumped)" after I run it
second time. I check the directory where stuff is put in and here is
the contents:

-rw------- 1 savior ugrad 16384 Feb 8 08:54 __db.001
-rw------- 1 savior ugrad 5251072 Feb 8 08:54 __db.002
-rw------- 1 savior ugrad 98304 Feb 8 08:54 __db.003
-rw------- 1 savior ugrad 311296 Feb 8 08:54 __db.004
-rw------- 1 savior ugrad 106496 Feb 8 08:54 __db.005
-rw------- 1 savior ugrad 1000000 Feb 8 08:54 log.0000000001
-rw------- 1 savior ugrad 32768 Feb 8 08:54 testdb.db

I dont know what __db.00x are. If the culprit is __db.002 (since it's
the biggest), how do i reduce its size? It seems to me i have no
control over it.

Thank you so much~




Michael Cahill wrote:

Hi,

__db.002 is the cache region. Try calling set_cachesize with a smaller
value.

Alternatively, can you try just creating a large file by some other
means (for example, "dd if=/dev/zero of=bigfile bs=1M count=64"). As I
said earlier, I suspect that this is some limit on your system
unrelated to Berkeley DB.

You still haven't given any information about what sort of system you
are using, but if you are using Linux, run "cat /proc/sys/fs/file-max"
and see whether that tells you anything. Some Linux distributions have
a file /etc/security/limits.conf that might be the source of the
constraint.

Regards,
Michael.

Thanks for your reply. Now the problem is gone somehow: I no longer get
filesize exceeded error - thanks to you. However I don't understand why
sometimes the program pauses without giving any output. It seems kind
of random. For ex I run 2 threads each running read(theKey) and
write(theKey) only. I run these threads several times and it works
fine. However when I run it the 20th time, it simply pauses. I do ^C to
terminate it and run it again. Now threads are started but do nothing
but pause. I tried many combinations of read() and write() and most of
them give me this "pause indefinitely" problem. Please advise.

Again, I can't thank you enough. By the way I am using Fedora Core 2,
and use BDB's java version.

Hi,

The most likely cause of this error is if your application exits
without closing the environment cleanly.

If you kill your process (with ^C, say), you must run db_recover or
open the environment with the DB_RECOVER flag, before using it again.

Regards,
Michael.

I don't quite understand stand what __db.001, __db.002, __db.003,
__db.004, and __db.005 's functionalities are? I know now that __db.001
may have someting to do with mutex, __db.002 with cache.. the rest I
have no idea, what are they meant to show us? Thanks a lot for advices.

I don't quite understand stand what __db.001, __db.002, __db.003,
__db.004, and __db.005 's functionalities are? I know now that __db.001
may have someting to do with mutex, __db.002 with cache.. the rest I
have no idea, what are they meant to show us? Thanks a lot for advices.