Re: BerkeleyDB CDB

#1

Nathan,

I don't quite understand your email. You ask if there is a way
to have non-blocking locks in CDB, but then you say that at the
point I would expect a block, you do not block but get an EINVAL
error return.

There is no way with CDB to request non-blocking locks. This option
is only supported when starting a transaction. Transactions are not
part of CDB.

I don't know why you are getting an EINVAL return.

Michael Ubell
Sleepycat Software

#2

In article <36b1555f.0308131509.20cf365e (AT) posting (DOT) google.com>,
ubell (AT) sleepycat (DOT) com (Michael Ubell) writes:

Quote:

I don't quite understand your email. You ask if there is a way
to have non-blocking locks in CDB, but then you say that at the
point I would expect a block, you do not block but get an EINVAL
error return.

I was theorizing that if there was a non-blocking option somewhere that
was getting set by the BerkeleyDB perl module, then this might explain
why the second call to db_cursor() fails instead of blocking. I can't
think of any other reason.

I am looking into this more. I thought that I had verified this behavior
on my FreeBSD machine, but now it appears that the problem is only on OSX
(Darwin?). I'll include my test code at the end of this post. Basically
I run it once and it seems to work. Then I run it again and the second
one should block but instead returns an error. Are there any known issues
with CDB on OSX?

/Nathan.

#!/usr/bin/perl5 -Tw
# This is a perl script to test database locking

use strict;
use Socket;
use BerkeleyDB;

# some variables
my $dbhome = ".";
my $dberr = "err";
my $dbname = "db";
my ($env, $db, $cursor);

# create a new database environment
$env = new BerkeleyDB::Env -Home => $dbhome
, -ErrFile => $dberr, -Flags => (DB_CREATE | DB_INIT_CDB | DB_INIT_MPOOL);
if (!$env) { die "could not create env.\n"; }

# open the database
$db = new BerkeleyDB::Btree -Env => $env, -Filename => $dbname
, -Subname => "data"
, -Flags => DB_CREATE;
if (!$db) { die "Cannot open database '$dbname': $! '$BerkeleyDB::Error'\n"; }

# acquire write lock
print STDERR "$$:Getting write lock cursor.\n";
$cursor = $db->db_cursor(DB_WRITECURSOR);
if (!$cursor) { die "Could not get cursor.\n"; }
print STDERR "$$:Got write lock cursor.\n";

# wait for kill
eval {
local $SIG{INT} = sub { die "interrupted"; };
local $SIG{TERM} = sub { die "interrupted"; };
while (1) { sleep 2; }
};

# loose the lock and die
$cursor->c_close();
print STDERR "$$:Lock released.\n";
$db->db_close();
undef $env;
exit 0;

#3

In article <22fa329d.0308132355.1316b9bd (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

Quote:

Nathan,

there is no no-blocking option with the Perl interface with CDB.

If you are getting a EINVAL error, it sounds like you are passing
db_cursor an invalid argument. Can we have a look at the code?

cheers
PAul

I've posted the test code elsewhere on this thread. It's definately
not a parameter that I send because it works the first time that I
run it. The problem only appears when the call should block. Maybe
it has to do with OSX? Are you aware of any issues with BerkeleyDB
and OSX?

Also, what is the variable "ix"? I can't find its declaration or when
it is assigned. From BerkeleyDB.xs line 2811:

if (ix == 1 && db->cdb_enabled) {
#ifdef AT_LEAST_DB_3
flags = DB_WRITECURSOR;
#else
flags = DB_RMW;
#endif
}

My tests on FreeBSD appear to be working though, so I don't think that
DB_WRITECURSOR is actually being set on every cursor? That would not
be good I don't think?

/Nathan.

#4

In article <22fa329d.0308150020.7c2cd9b3 (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

Quote:

I'm not aware of any OSX issues with my Perl interface. Whether there
are any problems with Berkeley DB itself, I don't know. Perhaps
someone from Sleepycat could comment.

I've tried your code on Soaris and it seems to work fine. When I fire
up multiple instances of the script, only one will get the lock. When
that process is killed, one of the other blocked processes acquires
the lock.

That is what happens when I test on FreeBSD as well. But when I test it on
OSX then the second invocation of the script fails as described earlier on
this thread. This is discouraging since I have been developing on OSX based
on Apples sales pitch "it's based on FreeBSD".

Quote:

Also, what is the variable "ix"? I can't find its declaration or when
it is assigned. From BerkeleyDB.xs line 2811:

if (ix == 1 && db->cdb_enabled) {
#ifdef AT_LEAST_DB_3
flags = DB_WRITECURSOR;
#else
flags = DB_RMW;
#endif
}

That code is used internally when you associate a Perl hash with a
Berkeley DB database and you write something like this

%hash = () ;

Application code like yours that uses db_cursor will not execute that
code (the ix variable will always be 0).

That is good to know. Thanks.

Quote:

My tests on FreeBSD appear to be working though, so I don't think that
DB_WRITECURSOR is actually being set on every cursor? That would not
be good I don't think?

If you are referring to the code you posted, then DB_WRITECURSOR *is*
being set on every cursor. Is there a a reason why you think it isn't
behaving properly on FreeBSD?

Sorry that I was not very clear on this. Since I did not know the
functionality of ix, then I was concerned that the DB_WRITECURSOR was
being set for me by BerkeleyDB on every cursor. I have a second script
that is identical to the first but without the DB_WRITECURSOR to test
if read and write locks play together properly.

Still no joy on OSX though. Any hints as to where to look would be
helpful, but I can't spend a lot of time spinning my wheels here. I may
have to dump OSX for development work for a while. It has been a little
problematic in other ways as well.

/Nathan.

#5

hackett (AT) gardi (DOT) home.rapdat.com (Nathan Hackett) wrote in message news:<DvednWjriI_HyqCiRVn-gw (AT) speakeasy (DOT) net>...

Quote:

In article <22fa329d.0308150020.7c2cd9b3 (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

I'm not aware of any OSX issues with my Perl interface. Whether there
are any problems with Berkeley DB itself, I don't know. Perhaps
someone from Sleepycat could comment.

I've tried your code on Soaris and it seems to work fine. When I fire
up multiple instances of the script, only one will get the lock. When
that process is killed, one of the other blocked processes acquires
the lock.

That is what happens when I test on FreeBSD as well. But when I test it on
OSX then the second invocation of the script fails as described earlier on
this thread. This is discouraging since I have been developing on OSX based
on Apples sales pitch "it's based on FreeBSD".

....

Quote:

My tests on FreeBSD appear to be working though, so I don't think that
DB_WRITECURSOR is actually being set on every cursor? That would not
be good I don't think?

If you are referring to the code you posted, then DB_WRITECURSOR *is*
being set on every cursor. Is there a a reason why you think it isn't
behaving properly on FreeBSD?

Sorry that I was not very clear on this. Since I did not know the
functionality of ix, then I was concerned that the DB_WRITECURSOR was
being set for me by BerkeleyDB on every cursor. I have a second script
that is identical to the first but without the DB_WRITECURSOR to test
if read and write locks play together properly.

ok

Quote:

Still no joy on OSX though. Any hints as to where to look would be
helpful, but I can't spend a lot of time spinning my wheels here. I may
have to dump OSX for development work for a while. It has been a little
problematic in other ways as well.

When you got the EINVAL error, was that from $! or $BerkeleyDB::Error
? Can you run the test again and see what the values of both are
please?

I wonder if this is a signal handling issue? Lets see if we can remove
it from the equation.

Here is a variation on your code, that replaces the signal handler
with a prompt for input - so the script will hold the lock until you
press return. Try running multiple instances and see if there is the
same problem.

#!/usr/bin/perl5 -w
# This is a perl script to test database locking
use strict;
use Socket;
use BerkeleyDB;
# some variables
my $dbhome = ".";
my $dberr = "err";
my $dbname = "db.tmp";
my ($env, $db, $cursor);
# create a new database environment
$env = new BerkeleyDB::Env -Home => $dbhome
, -ErrFile => $dberr, -Flags => (DB_CREATE | DB_INIT_CDB |
DB_INIT_MPOOL);
if (!$env) { die "could not create env.\n"; }
# open the database
$db = new BerkeleyDB::Btree -Env => $env, -Filename => $dbname
, -Subname => "data"
, -Flags => DB_CREATE;
if (!$db) { die "Cannot open database '$dbname': $!
'$BerkeleyDB::Error'\n"; }

# acquire write lock
print STDERR "$$:Getting write lock cursor.\n";
$cursor = $db->db_cursor(DB_WRITECURSOR);
if (!$cursor) { die "Could not get cursor: $! '$BerkeleyDB::Error'\n";
}
print STDERR "$$:Got write lock cursor.\n";

print "Press RETURN to release the lock\n" ; <> ;

# loose the lock and die
$cursor->c_close();
print STDERR "$$:Lock released.\n";
$db->db_close();
undef $env;
exit 0;

Paul

#6

In article <22fa329d.0308160049.7936a943 (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

Quote:

When you got the EINVAL error, was that from $! or $BerkeleyDB::Error
? Can you run the test again and see what the values of both are
please?

Yea, I meant to mention that. It shows up in niether. The value of
$BerkeleyDB::Error is "Successful return: 0" and $! is empty. The
only reason that I know of the error is becuase I recompiled BerkeleyDB
with -DTRACE and it gives

"db_cursor db[0x10f50], txn[0x0], flags[36]"
"db_cursor returned 22: Invalid argument, Successful return: 0".

I may have added the first line myself? I can't remember except that I know
that I added more output for TRACE in the vacinity of _db_cursor() to try
and understand this.

Quote:

I wonder if this is a signal handling issue? Lets see if we can remove
it from the equation.

I don't think that it is, but I will run your script and post back.

Thanks,

/Nathan.

#7

In article <22fa329d.0308160049.7936a943 (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

Quote:

Here is a variation on your code, that replaces the signal handler
with a prompt for input - so the script will hold the lock until you
press return. Try running multiple instances and see if there is the
same problem.

O.K. So I ran your modified script and this brings up another odity in

OSX. The first time I run the script it runs exactly as I would expect
(the db file already exists before this first invocation). Then, when
I run it the second time, it gives error 22 on db_create(). It does this
on my OSX machine when I pass DB_CREATE as a flag to ::Btree. I can't
figure this one out either. The first one runs fine, then the second one
won't run unless I remove DB_CREATE.

Anyway, after I removed DB_CREATE flag, then I get exactly the same thing:

"db_cursor returned 22: Invalid argument, Successful return: 0"

/Nathan.

P.S. I wonder if this has to do with my db installation on OSX.
what I did was steel the tarball from my FreeBSD ports collection and
just build it on OSX. It seemed to build fine, but I'm going to try a
fresh build and see if that helps.

#8

Is it possible that when I built the BerkeleyDB perl module that it
linked against the static libraries instead of the dynamic ones and
that is causing my trouble?

How can I verify this?

/Nathan.

#9

hackett (AT) gardi (DOT) home.rapdat.com (Nathan Hackett) wrote in message news:<1amdnfCLZLTPM6OiRVn-jg (AT) speakeasy (DOT) net>...

Quote:

In article <22fa329d.0308160049.7936a943 (AT) posting (DOT) google.com>,
Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) writes:

Here is a variation on your code, that replaces the signal handler
with a prompt for input - so the script will hold the lock until you
press return. Try running multiple instances and see if there is the
same problem.

O.K. So I ran your modified script and this brings up another odity in
OSX. The first time I run the script it runs exactly as I would expect
(the db file already exists before this first invocation). Then, when
I run it the second time, it gives error 22 on db_create(). It does this
on my OSX machine when I pass DB_CREATE as a flag to ::Btree. I can't
figure this one out either. The first one runs fine, then the second one
won't run unless I remove DB_CREATE.

Anyway, after I removed DB_CREATE flag, then I get exactly the same thing:

"db_cursor returned 22: Invalid argument, Successful return: 0"

/Nathan.

P.S. I wonder if this has to do with my db installation on OSX.
what I did was steel the tarball from my FreeBSD ports collection and
just build it on OSX. It seemed to build fine, but I'm going to try a
fresh build and see if that helps.

If you mean that you ran the Berkeley DB configure script on FreeBSD,
tarred up the resulting source tree, then untarred it on your OSX box
and just typed "make", it is very likely going to cause a problem.

Paul

#10

hackett (AT) gardi (DOT) home.rapdat.com (Nathan Hackett) wrote in message news:<cdadnabPIZ5RaaOiRVn-gg (AT) speakeasy (DOT) net>...

Quote:

Is it possible that when I built the BerkeleyDB perl module that it
linked against the static libraries instead of the dynamic ones and
that is causing my trouble?

How can I verify this?

Find the file BerkeleyDB.so and run the ldd command against it
(assuming OSX has the ldd command)

This is what I get on my Linux box, so mine is not statically lined to
the Berkeley DB library

[paul@linux BerkeleyDB-0.07]$ ldd
../blib/arch/auto/BerkeleyDB/BerkeleyDB.so
libdb-4.1.so =>
/home/paul/perl/ext/BerkDB/BerkeleyDB-0.07/libraries/4.1
..25/lib/libdb-4.1.so (0x4002b000)
libc.so.6 => /lib/i686/libc.so.6 (0x400e4000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x80000000)