hello all,

my application is using secondary databases.
it seems that berkeley db can cause itself to deadlock when trying to
retrieve data
using 'pget' in one thread and at the same time trying to insert data,
using 'put' in another thread.

after a glance at the DB source code it seems like those functions are
working in the following manner:

put:
-----
1. acquire exclusive lock on the page in the primary database and
putting the data.
2. acquire exclusive lock on the page in the secondary database and
putting the data.

pget:
-------
1. acquire shared lock on the secondary database.
2. acquire shared lock on the primary database.

the two functions are blocking each other.
in order to avoid deadlock, the two functions need to access the data
in the same order.
is this a bug?
can you see the problem?

waiting for your reply,
moshe.

Hi Moshe,

Deadlock is certainly possible in that situation, so your application
will need to deal with it (by running deadlock detection and handling
deadlock errors).

Berkeley DB does try to minimize deadlock in this case: nothing can be
done about secondary gets, since they have to read the secondary to
find the primary key. So primary put operations try hard to access the
secondaries before the primary. That is possible in many cases, but
not if doing a DB->c_put with the DB_CURRENT flag, or for puts with the
DB_DBT_PARTIAL flag set on the data.

Can you explain what flags you're using for the put, including the DBT
flags?

Regards,
Michael.

Hi,
We use the C++ API wrapping of the Berkeley DB Interface, after
examining our code we discovered we get the deadlock when using the
DB_NOOVERWITE flag (with the C++ interface). We've managed to bypass
this deadlock by avoiding the use of this flag (which is very limiting,
and narrows down our application's capabilities), but we've encountered
another deadlock now, when deleting a record from the primary DB
(similar to writing without overwriting...)
It is not clear to us, why the Berkeley DB doesn't provide atomicity
for these actions, we expect the infrastructure to be deadlock free
when performing single instructions.
It would be of great help if you could come up with a solution for this
problem

As for now, the only solution we have in mind, is to maintain a locking
mechanism of our own...

Thanks in advance,
Ron & Moshe.

Herman.Ron (AT) gmail (DOT) com writes:

Where does this expectation come from?

Handle deadlocks.

best regards
Patrick

Hi Patrrick
I was very upset after reading your reply... if you have nothing to
say, then don't say anything
I think that it is a very reasonable assumption to think the Berkeley
DB actions should be atomic as to one another... and on top of that,
the documentation doesn't mention anything about this limitation, I
personaly thought, that the same locking order in implemented on every
instruction the infrastructure supplies (in order to avoid such
dealocks!!!)

P.S :
A. handling dealock is not an option for us in our
application, due to pre-defined/pre-designed interface, which we're
basing on...
B. we tried using the supplied locking machanism, but the
documentation isn't clear regarding the " Dbt *object " pointer
supplied to these methods

Regards,
Ron & Moshe

Herman.Ron (AT) gmail (DOT) com writes:

I am sorry to hear that.

That is very good advise. I had to say two things. I said them.

all the best
Patrick

Hi Ron and Moshe,

That makes sense -- DB_NOOVERWRITE has to check whether the item
exists, which means it will lock the primary first.

Again, that makes sense -- DB can't generate the secondary key(s) until
it has the record from the primary that is being deleted.

If you don't want to handle deadlock errors in your application, you
should consider the Concurrent Data Store product (open your
environment with DB_INIT_CDB instead of DB_INIT_LOCK | DB_INIT_LOG |
DB_INIT_TXN).

It implements a locking scheme similar to what you are describing
(coarse-grained read/write locking so that these operation would not
interleave). It is deadlock free. When used with secondaries, you
should also set the DB_CDB_ALLDB flag on the environment.

The major disadvantage of CDS is that it does not provide logging and
recovery, so you need to verify database integrity after application or
process failures, and potentially restore data from backups.

Regards,
Michael.

* Patrick Schaaf:

This expectation is quite reasonable if you've got a certain SQL
background. Typical SQL databases, when used in auto-commit mode,
automatically recover from deadlocks.

Florian Weimer <fw (AT) deneb (DOT) enyo.de> writes:

Aha. Thank you. That would explain that. I have no SQL background
at all, so how could I know...

best regards
Patrick