hi, i'm using the BerkeleyDB perl module with berkelydb 4.2.

i am have an application that stores images of about 40K each. and
often, there will be concurrent (?) writes, but not to the same key...
(forgive my terminology if off)

my current env is:
my $env = new BerkeleyDB::Env
-Cachesize => 131072,
-Flags => DB_CREATE | DB_INIT_LOCK | DB_INIT_MPOOL;

with -PageSize => 65536 using BerkeleyDB::Hash

my first question is about -Page and -Cache size. do these seem
reasonable values? or can i
optimize? i cant find too much docs about larger db entries.

2nd question about locking: i tried switching to BerkeleyDB::Btree
because entries with similar keys will tend to be requested at the same
time. however, it seems that when multiple writes (to different keys)
occur at near the same time, some just fail. i tried changing the env
flags to:
-Flags => DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB
but same problem. must i acquire a lock on the entire file?

any help or direction to pertinent docs is appreciated,
-brent

<bpederse (AT) gmail (DOT) com> wrote

I assume that there will be concurrent reads whilst the writes are taking
place?

That's a generic Berkeley DB question, which I'm not best placed to answer.

Switching from Hash to Btree won't make any difference if you have an
underlying concurrency problem.

Your seconf flags setting

-Flags => DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB

should be enough to allow safe concurrent access. Your previous flags
setting won't work.

Can you post more information on your setup (Operating System name, versions
of Perl, Berkeley DB etc) and some more details on how it is failing? Some
code would help as well. Also check that the environment file isn't stored
on a network drive.

Paul

hi, thanks for the reply.
perl -v: This is perl, v5.8.8 built for i386-linux
berkeleyDB. 4.2, latest BerkeleyDB.pm
on an x86 linux (gentoo machine).

there may be concurrent reads. while writing. yes.
with these flags:
-Flags => DB_CREATE | DB_INIT_LOCK | DB_INIT_MPOOL;
and BerkeleyDB::Hash i actually do get good results as things seem to
get stored and
retrieved correctly. 99% of the time.
with these flags:
-Flags => DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB
and Btree. Bad values (images) get saved in the correct keys and
when the image is retrieved, i see a broken image in my web page. fails
~ 60% of time. something is stored in the db, because subsequent calls
do not fill that entry and return an actual image (which they would if
nothing was stored for a given key).

i should also add that i am doing this in a web environment, and using
a URL as the db key.

the code is part of a larger size module, but here are the pertinent
parts:

my $env = new BerkeleyDB::Env
-Cachesize => 131072,
# this one works best.. but not great
-Flags => DB_CREATE | DB_INIT_LOCK | DB_INIT_MPOOL;
# -Flags => DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB;

$db = new BerkeleyDB::Hash
-Filename => $cache_file,
-Pagesize => 65536,
-Flags => DB_CREATE,
-Env => $env
or die "Cannot open $cache_file: $!\n check that directory is
web-writable\n"

# meanwhile in another method:

if($force_retile){
$db->db_del($param_url);
}

# cache BerkeleyDB returns 0 on success
my $in_cache = ! $db->db_get($param_url, $IMG);
return $IMG if $in_cache;

my $tmp = LWP::Simple::get($full_url);
return $tmp if $bypass_cache;

# if we made it this far, it's not in the cache;
# but it wants to be.

my $success = ! $db->db_put($param_url, $tmp);
$db->db_sync();
$this->{db_has_changed} = 1;
if($this->DEBUG()){
print STDERR 'status of put: ', $success,"\n";
}
return $tmp;
}



then in DESTROY

$db->db_sync()
....