Since upgrading my application to 4.29.29 of BerkeleyDB I have found
that the application is crashing every few days. I have not yet
isolated the source of the problem and it may very well be not related
to the database code.

The problem is that every couple of crashes the database just refuses
to recover.

At first I was getting errors indicating that a lot of the files has
invalid block sizes, then when I tried to run a db_verify I get got the
following:

db_verify: Page 122: item 0 of unrecognizable type
db_verify: Page 122: item 1 of unrecognizable type
.....
db_verify: Page 122: item 20 of unrecognizable type
db_verify: Page 122: gap between items at offset 880
db_verify: Page 122: item order check unsafe: skipping
db_verify: Page 123: bad page number 1507328
db_verify: Page 123: invalid next_pgno 1013055488
db_verify: Page 124: bad page number 1291911184
db_verify: Page 124: invalid next_pgno 3338666000
db_verify: Page 125: bad page number 16777220
db_verify: Page 125: nonzero level 207 in non-btree database
db_verify: Page 125: overflow page has zero reference count
db_verify: Page 126: bad page number 16777343
db_verify: Page 126: bad page type 77
db_verify: Page 126: unknown page type 77
db_verify: Page 127: bad page number 1614618624
db_verify: Page 127: bad page type 84
db_verify: Page 127: unknown page type 84
db_verify: Page 128: partially zeroed page
db_verify: Page 128: invalid next_pgno 1025572864
db_verify: Page 129: bad page number 805306368
db_verify: Page 129: invalid next_pgno 1684614732
db_verify: Page 130: bad page number 251658241
db_verify: Page 130: invalid next_pgno 3573547008
db_verify: Page 131: bad page number 16781312
db_verify: Page 131: invalid next_pgno 4241
db_verify: Page 132: bad page number 2263285767
db_verify: Page 132: invalid next_pgno 2236547073
db_verify: Page 133: bad page number 5242888
db_verify: Page 133: bad page type 128
db_verify: Page 133: unknown page type 128
db_verify: Page 134: bad page number 1012465664
db_verify: Page 134: bad page type 110
db_verify: Page 134: unknown page type 110
db_verify: Page 135: bad page number 3305111568
db_verify: Page 136: bad page number 3388999321
db_verify: Page 136: invalid next_pgno 21325
db_verify: 0490DL.idv: DB_VERIFY_BAD: Database verification failed

I was unable to db_dump the file without using the -R option.

What could be causing this, I was under the impression that in a
transactionally protected environment that you were protected from
database corruption.

Here is my DB_CONFIG:

set_cachesize 0 50000000 0
set_lk_max_lockers 20000
set_lk_max_locks 5000
set_lg_dir ../dblogs
set_flags DB_LOG_AUTOREMOVE

Any help appreciated.

Gerard

Some things to try:

1. Kill the application process and attempt recovery before it crashes.
See if the database is already corrupt at that point. This might tell
you something about when the corruption arises, and hence what
is causing it.

2. Verify that your application is not performing invalid memory
writes.
e.g. use Purify or Valgrind. It's just possible that your application
is
writing to a buffer before its written to disk, thereby corrupting the
database. BDB has various measures in place to protect itself
from this kind of attack, but I don't believe they're 100% fullproof.