Hi,

my name's Luca and I work for an italian software house.

Our main application works in UNIX-like systems (Linux, AIX, Sun, etc.)
and is based on "C-Isam" databases (one table = one pair of file
".dat", ".idx") accessed by multiple processes without transactions.
Every process is an executor of a program written in our proprietary
language and is single-threaded.

We are planning to replace "C-Isam" with another database
implementation and we choosed DB-Berkeley.

However, for a lot of reason, we cannot change our application
structure. So the replacement is at a very low-level (from our
programmers point of view nothing changes), but this have some impact
on the database architecture.

In practice, there is an environment per server (our application works
with databases distribuited on multiple servers) and there is one
file/database per table. All must work with multiple processes
accessing the databases/files (when I say multiple processes, I intend
in order of thousand of processes, even if most of them waiting user
input). The enviroment is created once at server startup with these
flags: DB_CREATE, DB_INIT_LOCK, DB_INIT_MPOOL, DB_INIT_LOG (the log is
optional, as we don't use transactions). All the processes then open
the environment with the DB_JOINENV.

After a lot of work/testing we resolved most of the problems but now we
have a problem with "lock getting" that seems hard to resolve (maybe
impossible!?!).

All the "low level" code is written in "C". The machine used for the
test, is a "Linux" server with 4 processor (can this be a problem?).
The library of the DB Berkeley are always compiled in relase mode and
linked statically.

Before analyzing the problem, I must inform that we had to do a change
of the source code, to resolve the problem of a process that died
"brutally" ("kill -9" and yes we have to deal with it): if this process
leaved some mutex acquired, I free them. I don't think it has impacts
for the real problem, but who knows?
I changed the source of the file "mut_tas.c" with these lines (I used
the code of the "is_alive" feature and I must admit that before last
version of DB Berkeley this was another great issue):

/*

* Yield the processor.

*/

__os_yield(NULL, ms * USEC_PER_MS);

if ((ms <<= 1) > max_ms)

ms = max_ms;


// My code start here
if (ALIVE_ON(dbenv) &&

!dbenv->is_alive(dbenv, mutexp->pid, mutexp->tid)) {

__db_tas_mutex_unlock(dbenv, mutex);

}
// My code stop here


/*

* We're spinning. The environment might be hung, and somebody
else
* has already recovered it. The first thing recovery does is
panic
* the environment. Check to see if we're never going to get
this
* mutex.

*/


In fact, we don't tested it very well, but it seemed to work.

Another change that might have some impact, is that I used the function
"DBENV->set_lk_conflicts" to modify the behaviour in case of READ/WRITE
conflict. This is due to the fact that we must use "always open" cursor
and we permit to write a record currently pointed from a cursor (our
high level language doesn't have an equivalent of the "get current
record of the cursor", so it is not so badly).

The code is below:

if ((r = dbenv->get_lk_conflicts( dbenv, (const u_int8_t
**)&conflicts, &nmodes)) == 0)
conflicts[nmodes + 2] = 0;

r = dbenv->set_lk_conflicts( dbenv, conflicts, nmodes );

}


Here comes the real problem: when there are some intensive processes
(let say 15/20) working on the same files, after some time the DB
Berkeley "goes" in panic for this reason (this is the output of the
file setted with the function "DBENV->set_errfile"):

DB_LOCK->lock_put: Lock is no longer valid
Cursor position must be set before performing this operation
Unexpected lock status: 6
PANIC: Invalid argument

The first two lines seems to be only warning and, I suppose, are
correctly managed. They seem not to be linked with the problem.
The third line is the problematic one! Lock status "6" is
"DB_LSTAT_WAITING": I think that the lock can't be granted in time and
so its leaved in an invalid status.
Only recently I discovered that I have to set the "lock detector" with
the function DBENV->set_lk_detect; if not so, this error came out wery
quickly. So I even added the "db_deadlock" deamon (with time of 5
seconds, 100 msec or 1 msec), but even so there was the error.

Is there a way to resolve this situation? Have we forgotten something?

Thanks in advance.
ciao, Luca