dbTalk Databases Forums  

Can't get fill factor high enough

comp.databases.berkeley-db comp.databases.berkeley-db


Discuss Can't get fill factor high enough in the comp.databases.berkeley-db forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Sue
 
Posts: n/a

Default Can't get fill factor high enough - 05-31-2006 , 07:23 PM






I'm using Sleepycat 4.4.20, C API. I've been struggling trying to get
the fill factor for one of my databases higher than 51%, and failing
miserably. Hopefully someone has some ideas.

Database is a Btree, running on an Opteron Solaris 10 system. Page
size is set to 16384. Key size is 12 (1 32-bit integer and 1 64-bit
integer), and I've got an insert sort function that appears to be
working correctly. Data size is 28, if that matters. Duplicates are
not allowed in this database.

The key is an object ID (64-bit), with a 'container' ID (32-bit).
Neither is unique by itself, but the combination creates a unique ID.
I originally tried sorting on the 64-bit integer first, then the 32-bit
integer. This resulted in about a 55% fill rate, but was exceptionally
slow - presumably because there are a LOT more Objects than Containers.
Switching to using the container ID as the primary sort sped things up
by an order of magnitude (!), but the fill factor sank even lower, to
50%. Dumping the database didn't show anything that looked
extraordinary - getting 314 entries/leaf most times.

Questions are:
1. Any idea why there is such a huge speed difference when sorting on
the container ID vs. the object ID? (note: For retrieval purposes, it
may actually be better to keep it sorting this way, so I'm not
terrifically bothered by this, but I am very curious)
2. What could be causing such a low fill factor, and what can I do to
fix it?

Thank you,


Reply With Quote
  #2  
Old   
Ron
 
Posts: n/a

Default Re: Can't get fill factor high enough - 06-01-2006 , 10:29 AM






Sue wrote:
Quote:
I'm using Sleepycat 4.4.20, C API. I've been struggling trying to get
the fill factor for one of my databases higher than 51%, and failing
miserably. Hopefully someone has some ideas.

Database is a Btree, running on an Opteron Solaris 10 system. Page
size is set to 16384. Key size is 12 (1 32-bit integer and 1 64-bit
integer), and I've got an insert sort function that appears to be
working correctly. Data size is 28, if that matters. Duplicates are
not allowed in this database.

The key is an object ID (64-bit), with a 'container' ID (32-bit).
Neither is unique by itself, but the combination creates a unique ID.
I originally tried sorting on the 64-bit integer first, then the 32-bit
integer. This resulted in about a 55% fill rate, but was exceptionally
slow - presumably because there are a LOT more Objects than Containers.
Switching to using the container ID as the primary sort sped things up
by an order of magnitude (!), but the fill factor sank even lower, to
50%. Dumping the database didn't show anything that looked
extraordinary - getting 314 entries/leaf most times.

Questions are:
1. Any idea why there is such a huge speed difference when sorting on
the container ID vs. the object ID? (note: For retrieval purposes, it
may actually be better to keep it sorting this way, so I'm not
terrifically bothered by this, but I am very curious)
2. What could be causing such a low fill factor, and what can I do to
fix it?

If you search on the newsgroup that you posted to you will find
several entries discussing integer keys on Little Endian Systems. I
believe that is the problem you are observing. There is also an FAQ in
our Developer Zone which discusses the problem and the solution.

The faq posting can be found here:

http://dev.sleepycat.com/resources/f...ction%3Dsearch

Alternatively you can search in this newsgroup and review a couple of
the responses.

Once you are set with that you may want to review this thread on
performance:
http://tinyurl.com/p6tes

Ron



Reply With Quote
  #3  
Old   
Sue
 
Posts: n/a

Default Re: Can't get fill factor high enough - 06-01-2006 , 01:49 PM



Ron wrote:
Quote:
If you search on the newsgroup that you posted to you will find
several entries discussing integer keys on Little Endian Systems. I
believe that is the problem you are observing. There is also an FAQ in
our Developer Zone which discusses the problem and the solution.

The faq posting can be found here:

http://dev.sleepycat.com/resources/f...ction%3Dsearch

Alternatively you can search in this newsgroup and review a couple of
the responses.

Once you are set with that you may want to review this thread on
performance:
http://tinyurl.com/p6tes

Ron
Thank you Ron, but as I pointed out in the initial message, I've
already got the little endian issue addressed and I've read everything
here, on the Sleepycat website and elsewhere on the web without
success. I'm looking for out-of-the-ordinary suggestions here, as
I've tried everything else.



Reply With Quote
  #4  
Old   
Ron
 
Posts: n/a

Default Re: Can't get fill factor high enough - 06-02-2006 , 10:28 AM




Sue wrote:
Quote:
Ron wrote:

If you search on the newsgroup that you posted to you will find
several entries discussing integer keys on Little Endian Systems. I
believe that is the problem you are observing. There is also an FAQ in
our Developer Zone which discusses the problem and the solution.

The faq posting can be found here:

http://dev.sleepycat.com/resources/f...ction%3Dsearch

Alternatively you can search in this newsgroup and review a couple of
the responses.

Once you are set with that you may want to review this thread on
performance:
http://tinyurl.com/p6tes

Ron

Thank you Ron, but as I pointed out in the initial message, I've
already got the little endian issue addressed and I've read everything
here, on the Sleepycat website and elsewhere on the web without
success. I'm looking for out-of-the-ordinary suggestions here, as
I've tried everything else.
Sue,

If you can send us a test program (hopefully not to large and does not
require installation of other packages) that will compile cleanly and
some sample data I will review your application and try to figure out
what is going on. Please send the program and data as well as telling
us the version of Berkeley DB you are running and the OS. Send this
information to support... I'm not typing the @sleepycat.com due to
Spam.


Ron



Reply With Quote
  #5  
Old   
Sue
 
Posts: n/a

Default Re: Can't get fill factor high enough - 06-05-2006 , 04:52 PM




Ron wrote:
Quote:
Sue,

If you can send us a test program (hopefully not to large and does not
require installation of other packages) that will compile cleanly and
some sample data I will review your application and try to figure out
what is going on. Please send the program and data as well as telling
us the version of Berkeley DB you are running and the OS. Send this
information to support... I'm not typing the @sleepycat.com due to
Spam.


Ron
Terrific. I think I've got a decent reproducer written, just verifying
that it truly
shows the same strangeness before sending it on.

Sue



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.