I'm quite a newbie when it comes to using Berkeley DB. I am using BDB
dbxml.

I have an application that fetches data from an external source and
writes that into dbxml. The data fetch takes a long time, so I start a
new thread to do this. All writes into dbxml for that batch are then
done in that same new thread.

When I have a few batches running at the same time, after variable
amounts of time, the application will hang. When this happens my only
option is to run db_recover.

The problem happens when I have more than one of these: multiple writes
in multiple threads. I am protecting everything with transactions, but
am getting lock conflicts.

The output of db_stat -CA includes this:

1 Total number of locks not immediately available due to
conflicts

80000275 READ 28 HELD RefData page
3105
80000275 WRITE 55 HELD RefData page
3105
80000276 READ 1 WAIT RefData page
3105

Unfortunately, I don't know how to interpret this. How do I tell where
the conflict is happening? What sort of pattern should I be following
to prevent this from happening?

Many thanks,

PC

* Petra Chong:

Do you run any kind of deadlock detector?

Florian Weimer wrote:
I do- but it didn't help.

However, I have solved the problem like this:

(I am using dbxml)

1. All queries use XmlQueryContext.DeadValues except where live values
are truly necessary. This reduced the read-write lock conflicts
considerably

2. All queries specify DB_RMW if they are selecting nodes to be
modified later (read-modify-write pattern)

This got me to the point where I was getting write-write lock
conflicts. This was easy to solve by:

3. Making all writes in my application single-threaded. I realise that
this solution is probably not acceptable for other applications, but my
usage pattern is such that single-threaded writes are acceptable.
Basically I am running a series of batch uploads where the source data
comes from an external source that takes a long time to return. The
data is only useful to me after it has completed uploading; it doesn't
matter whether 5 batches run sequentially or are interleaved- I only
care about when they are all uploaded completely.

Regards

PC