Hello all,

In my software (http://rrfw.sourceforge.net) I'm intensively using
Concurrent Data Store, and it crashes or hangs at the times
when mod_perl user frontend causes intensive concurent access.

See the detailed bug description at
http://sourceforge.net/mailarchive/m...msg_id=5894020

Currently the opening/closing operations are super-paranoid: with
every HTTP GET, the environment gets opened, and then closed,
along with the database handles. At least, the Perl interface tells
so. db_stat shows that there are no lockers or open locks left
when mod_perl cycle finishes.

I'm very reluctant to migrate to another backing store, but currently
the project is blocked by this problem. Any help or advice to
track and fix the problem are highly appreciated.

With regards,
Stanislav

"Stanislav Sinyagin" <ssinyagin (AT) yahoo (DOT) com> schreef in bericht
news:9e2f1b32.0308310712.11cf1d36 (AT) posting (DOT) google.com...

ssinyagin (AT) yahoo (DOT) com (Stanislav Sinyagin) wrote in message news:<9e2f1b32.0308310712.11cf1d36 (AT) posting (DOT) google.com>...
I had a very quick look at your code (sorry, I don't have the time at
the moment to run it - I don't have mod_perl available). I don't see
anything obviously wrong, but it's difficult to work out how you are
using your RRFW:B without spending more time analysing the code in
detail.

Is mod_perl multi-threaded these days? - it's been a while since I
looked at it. If it is, you will have problems with my BerkeleyDB
module. It does not support running multi-threaded.

In the email you sent me, and I've only just seen, you say that
db_stat hangs when your code locks up. That sounds like your code has
deadlocked the database somehow. For each HTTP GET, does your code
only write to a single database? If you have multiple databases, you
might want to try using the DB_CDB_ALLDB flag when you are opening the
environment (specify -Property => DB_CDB_ALLDB)

RandomCollector I assume is a way to generate events to test your
system? Just how much traffic will it generate? You may be hitting the
limits of the CDS, which isn't designed to cope with extreme
concurrent access, especially if you open the environment & database
for every HTTP request. You might want to try caching the environment
& database objects, rather than opening & tearing theme down every
time.

cheers
Paul

ssinyagin (AT) yahoo (DOT) com (Stanislav Sinyagin) wrote in message news:<9e2f1b32.0308310712.11cf1d36 (AT) posting (DOT) google.com>...
We first started using BerkeleyDB and Perl in combination more than
three years ago at which time we were using CDB, mainly because it was
the easiest option to begin with, as the library handles concurrency,
and we didn't need transactions.

At peak times, when the system was under stress, we found that CDB
would sometimes "lockup" or else the system would become unresponsive.
At these times, the solution was to delete and re-create the
environment and restart the application.

We later switched to using the locking subsystem, i.e.,
"DB_INIT_LOCK", and made the appropriate changes to the code to handle
expected return codes, such as "DB_LOCK_NOTGRANTED" and
"DB_LOCK_DEADLOCK". In addition, we now run the standard deadlock
detector to handle deadlocks.

Since making the switch, we haven't experienced problems of the kind
we experienced with CDB. We have the odd time-out, or
"DB_LOCK_NOTGRANTED" error, which we handle by retrying the operation
a pre-defined no. of times but otherwise the system performs well.

Obviously, it depends on the nature of your application but in my
experience I would certainly recommend using the locking subsystem in
preference to CDB for all but the simplest applications.

Good luck with it!

Peter

Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) wrote in message news:<22fa329d.0309010340.5df37ff3 (AT) posting (DOT) google.com>...

In short, on every HTTP GET, several databases are opened,
two of them in r/w, others in readonly mode. The database responsible
for HTTP renderer cache is updated with the cache details.
The other database, config_readers.db, is used to register the readers
of the main configuration database, in order to prevent configuration
update when there are active readers.

After HTTP request is processed, and we have a filename for output,
the databases get closed. In this development snapshot, I close the environment
too. In previous releases, I kept one opened environment per mod_perl
process. Now I made it as paranoid as possible, in order to track down the
problem (it persists with the old RRFW release too).


no, it's not multi-thread. Apache master process forks out several child
processes, and they live independently.

Exactly. On FreeBSD, iy just locks up. On Solaris, the process gets bus error.
I tested only the previous releases on Solaris.

It writes to two databases, and opens in readonly mode about 5-6
other databases, and reads them intensively.

Tried it, as long as somem other flags.
With DB_CDB_ALLDB, it lives longer. But locks up the same way after few
hundred HTTP GET requests.


Yes, it just generates lots of random data.

With the test described in that bug description, the HTML page has
around 250 images, each generated with RRFW renderer. It means
there are 4 Apache processes generating this output (as far as I know,
4 is the standard limit of IE simultaneous reqests).

The environment object is caching in the release versions, and
that's where this problem appeared. I'll try to see if it's possible
to cache the open database handles.

Thanks,
Stan

Hello all

Paul.Marquess (AT) btinternet (DOT) com (Paul Marquess) wrote in message news:<22fa329d.0309010340.5df37ff3 (AT) posting (DOT) google.com>...
I tried this, and it really helped. Now the DB handles are opened
once per process, and get closed at the process exit.
The stress tests are passed, and db_stat -c shows no deadlocks
or forgotten lockers. So, RRFW's future is safe now.

Still, it's an open problem for BDB that it locks up under
heavy opening/closing activity.
I've received a patch from Michael Ubell @sleepycat,
and will try to test it with the old configuration soon.

Thanks,
Stan