Hallo everybody, i'm a newbie.

I'm trying to test Berkeley DB Java Edition to understand if is the
right technology for my purpose.

I'll try to explain what is the environment and what i'm testing.

The server is a hp server dual cpu with 4 GB of ram running linux os
(red hat distribution).

I got a multithread test, in which every thread has its own Berkeley
environment, with its own DB (around 6 PrimaryDb and other 2 Secondary
DB for environment)

Every thread running (200) got 2,5 MB of cache
....
environmentConfig.setCacheSize((5 * 1000 * 1024)/2));
....

and handles a lot of domain objects (around 20000 different objects) in
6 primaryDB and 2 secondaryDB.

The JVM maximum heap size is set to 1024 MB.

The problem that occurs sometimes is that when we monitor the server
with the top command during the creation and the insert in
DBEnvironment of our beans we got this strange behaviour:

CPU states:
cpu user nice system irq softirq iowait idle
total 2.7% 0.0% 97.2% 0.0% 0.0% 0.0% 0.0%

As you can see after vmstat command at a certain moment the system
(kernel) process goes at maximum load.

procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us
sy id wa
4 0 0 2652940 56556 552516 0 0 0 4324 217 995 78
3 20 0
7 1 0 2607224 57216 599432 0 0 0 5641 234 1017 85
2 13 0
108 0 0 2601400 57792 625236 0 0 0 3031 197 677 46
43 11 0
109 0 0 2601400 57812 625236 0 0 0 17 141 194 2
98 0 0
108 0 0 2601400 57836 625236 0 0 0 17 141 134 2
98 0 0

I was supposing to got a lot of disk work (iowait) and not this other
"kernel" work during the test.
Moreover this behaviour is not systematic and I cannot understand which
operation is guilty.
I think that cache size is not responsible because I can see that the
iowait is very low.

Moreover once happened this exception:

[java] 14:29:46;/var/berkeley/VM_0_120;722156;180114;52116
[java] <Checkpointer name="Checkpointer"/> caught exception:
com.sleepycat.je.DatabaseException: fetchTarget of 0x2/0x15cd50
IN=960004 state=0 com.sleepycat.je.log.LogFileNotFoundException:
0x2/0x15cd50 Couldn't open file /var/berkeley/VM_0_40/00000002.jdb:
/var/berkeley/VM_0_40/00000002.jdb (Too many open files)
[java] com.sleepycat.je.DatabaseException: fetchTarget of
0x2/0x15cd50 IN=960004 state=0
com.sleepycat.je.log.LogFileNotFoundException: 0x2/0x15cd50 Couldn't
open file /var/berkeley/VM_0_40/00000002.jdb:
/var/berkeley/VM_0_40/00000002.jdb (Too many open files)
[java] at com.sleepycat.je.tree.IN.fetchTarget(IN.java:922)
[java] at
com.sleepycat.je.tree.Tree.searchSubTreeUntilSplit (Tree.java:1994)
[java] at
com.sleepycat.je.tree.Tree.searchSubTreeSplitsAllo wed(Tree.java:1933)
[java] at
com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tre e.java:1637)
[java] at
com.sleepycat.je.tree.Tree.findBinForInsert(Tree.j ava:3088)
[java] at com.sleepycat.je.tree.Tree.insert(Tree.java:2263)
[java] at
com.sleepycat.je.dbi.CursorImpl.putLN(CursorImpl.j ava:964)
[java] at
com.sleepycat.je.cleaner.UtilizationProfile.insert FileSummary(UtilizationProfile.java:1118)
[java] at
com.sleepycat.je.cleaner.UtilizationProfile.putFil eSummary(UtilizationProfile.java:724)
[java] at
com.sleepycat.je.cleaner.UtilizationProfile.flushF ileSummary(UtilizationProfile.java:664)
[java] at
com.sleepycat.je.recovery.Checkpointer.flushUtiliz ationInfo(Checkpointer.java:497)
[java] at
com.sleepycat.je.recovery.Checkpointer.doCheckpoin t(Checkpointer.java:411)
[java] at
com.sleepycat.je.recovery.Checkpointer.onWakeup(Ch eckpointer.java:218)
[java] at
com.sleepycat.je.utilint.DaemonThread.run(DaemonTh read.java:189)
[java] at java.lang.Thread.run(Thread.java:595)
[java] Continuing

Any idea of what is going on?

Thank you

Mazi

Hello Mazi,

Sorry for the slow reply, we don't normally use the newsgroup for the
JE product. In the future I suggest using the bdbje mailing list, see
here:

http://dev.sleepycat.com/community/discussion.html

Please see my comments below.

M4zi wrote:

This is a very unusual configuration. Can you use a single environment
with a 250 MB cache, and split up your data by database? You will get
much better cache utilization with a single larger cache than with many
small caches.

I suspect that the cache size is the problem. Since you have 4 GB
physical memory, if it is not used by other processes then my guess is
that all of your database files are in the file system cache. Because
you are using a small JE cache size for each JE environment, the system
may be spending a lot of its time moving data between the JE cache and
the file system cache. This doesn't require real device I/O, but is
still doing a lot of work in the kernel to move data out of the file
system cache.

Please try increasing the size of the 200 caches to see if the behavior
changes. For example, double your Java heap size and use a 8 MB cache
for each environment. JE will then use 1.8 GB out of a 2 GB heap.

Or even better, use a single enviornment with a large cache. Doing
this, you can probably get away with a smaller cache -- perhaps a 750
MB cache in a 1 GB heap would suffice.

Another possibility is that you have too many threads active at the
same time, and the kernel is spending a lot of time switching between
threads. If you only have two CPUs, it is best to have a comparable
number of concurrent threads. I suggest limiting the number of active
concurrent threads to four or less, by using a thread pool.

The "Too many open files" in the error above is the clue. One solution
is to increase the maximum number of open files in your OS.

Or, you can reduce the size of JE's file handle cache. It is 100 by
default. For example, you can change it to 10 by putting this line in
your je.properties file (in your environment directory):

je.log.fileCacheSize=10

Even better, if you use a single environment instead of 200
environments, you'll reduce the number of open files by a factor of
200, and then you won't need to change the size of JE's file handle
cache.

Mark