Hi,

I'm tryring to use berkeleydb on a single database across multiple
machines. Each machine also has multiple processes reading/updating the
database. The database file is mounted over NFS. Each process on each
machine is locking the database file after it opens it, performs its
reads/writes and then unlocks the file before closing it.
I'm using CDB while opening environements on individual machines.
But occasionally, I get an error 'DB_RUNRECOVERY: Fatal error, run
database recovery' on one of the machines while calling DB->put().
How can I ensure database file integrity across multiple machines over
a remote filesystem, inspite of locking the database file before
reading/updating it.

Any help would be appreciated.

Thanks,
Shashank

skavishwar (AT) gmail (DOT) com wrote:
Note [ http://www.sleepycat.com/docs/ref/env/remote.html ] where in
the middle of the third paragraph it states:

"it is difficult (or impossible) for modifiable databases on remote
filesystems to be accessed from multiple systems simultaneously."

Firstly, you'll need to change this order so it's:

lock the file
DB->open
...
DB->close
unlock the file

Otherwise, DB->open may see the file in an inconsistent state, and
pages that are flushed to the file during DB->close may not be seen by
the next process to open the file.

Further, for this to work, you must make sure that nothing is open when
the file is unlocked. That includes an environment handle. It doesn't
really make sense to use CDB locking for this application, since you
have to ensure that access is single-threaded for it to work.

Lastly, this also requires that all of the systems involved have
correct NFS implementations (so that fsync has POSIX semantics, among
other things).

Regards,
Michael.

Thanks for your help Michael. Yesterday I tried something else:

i) Create a .lck file on the remote FS and lock it.
ii) Create a DB environment.
iii) Open the environment using DB_JOINENV (as multiple processes
exist,
iv) Create a DB handle
v) Open the DB database file
vi) Perform reads/updates
vii) Close the DB database file
viii) Close the DB handle
ix) Close the DB environment
x) Close the .lck file (Closing it should release all locks on it)

So now, multiple processes, whether on the same machine or across
different machines should stall on step i) untill the .lck file is
unlocked. (Locking the DB database file should now be irrelevant)
But I'm still getting a DB_RUNRECOVERY error during a DB->put() in step
vi). This does not happen immediately, but generally seems to occur
when the database file size is around 3~4MB.

Why do you create an environment at all? Do any processes share the
same environment directory?

Even if it's a single process, the environment directory should not be
stored on NFS.

Lastly, is there any periodic access to the environment (such as the
db_checkpoint utility) that could be causing this problem (by not
waiting to get the lock)?

Michael.

I create an environment directory, as it is shared by multiple
processes on the same machine.

I have mutiple machines with multiple such processess runnning on them,
but all the environment directories on each machine are stored locally.

No, there is no periodic access to the environment through a checkpoint
environment. After a process updates the db file, it syncs the db and
then releases the file lock.