"Secondary index corrupt: not consistent with primary" when deleting data

gerol80 · #1

Hi,

I have a big problem when I try to delete data in a primary database that has an associated secondary database. The key in the primary database is an unique id (integer), the key in the secondary database is an integer number, too, but there can be duplicates (so I use flags DB_DUB and DUB_DUPSORT for the secondary database).

Inserting and retrieving data is working fine, so I think the callback function for the secondary keys works.

But when I try to delete data I get an DB_SECONDARY_BAD error with the error message: "Secondary index corrupt: not consistent with primary". It makes no difference whether I delete data from the primary or the secondary database. I've put some debug outputs into the callback function for the secondary keys --- the function is called and it sets the right value for the secondary key.

I tried to delete the data with or without a cursor, with an explicit transaction handle or with DB_AUTO_COMMIT --- it's always the same. Finally I cannot imagine where the mistake could be. Perhaps it's a bug?

#2

Hi,

Quote:

I have a big problem when I try to delete data in a primary database
that has an associated secondary database. The key in the primary
database is an unique id (integer), the key in the secondary database
is an integer number, too, but there can be duplicates (so I use flags
DB_DUB and DUB_DUPSORT for the secondary database).

Can you explain how the secondary keys are generated, or better still,
post the code for the secondary key callback?

Do you set any other callbacks, such as key or duplicate comparison
functions?

Lastly, what version of Berkeley DB are you using, and what programming
language API (C, C++, Java, etc.) are you using?

Quote:

Inserting and retrieving data is working fine, so I think the callback
function for the secondary keys works.

That isn't enough to say for sure -- a common issue with the callback
is that it must generate the original secondary key when given an
existing key/data pair, and that is only tested when an existing entry
is updated or deleted.

Regards,
Michael.

gerol80 · #3

Quote:

Originally Posted by Michael Cahill

Can you explain how the secondary keys are generated, or better still,
post the code for the secondary key callback?

Here's the code for the secondary key callback. The first "field" of the primary db record's data is an integer --- and this integer is used for the secondary key. Both debug outputs print the correct value.

Code:

int KeyOwnerID(Db *sdbp,          // secondary db handle
               const Dbt *pkey,   // primary db record's key
               const Dbt *pdata,  // primary db record's data
               Dbt *skey)         // secondary db record's key
{
  // get data of the primary db record
  int ownerID = *( (int*)pdata->get_data() );

  std::cout << "Key for secondary db: " << ownerID << "\n";

  // now set the secondary key's data to be the owner id
  skey->set_data( &ownerID );
  skey->set_size( sizeof(int) );

  std::cout << "Data of key: " << *( (int*)skey->get_data() ) << "\n\n";

  // return 0 to indicate that the record can be created/updated.
  return 0;
}

Quote:

Originally Posted by Michael Cahill

Do you set any other callbacks, such as key or duplicate comparison
functions?

Yes, there's another callback function:

Code:

int CompareInt(DB* dbp, const DBT* a, const DBT* b)
{
  int ai, bi;

  memcpy(&ai, a->data, sizeof(int));
  memcpy(&bi, b->data, sizeof(int));
  return (ai - bi);
}

I use this function to sort the primary database (with set_bt_compare( CompareInt )) and to sort the duplicates in the secondary database (with set_dup_compare( CompareInt )). But I tried to omit the sorting function and the problem was the same.

Quote:

Originally Posted by Michael Cahill

Lastly, what version of Berkeley DB are you using, and what programming
language API (C, C++, Java, etc.) are you using?

My Berkeley DB version is 4.4.20 and I use the C++ API.

gerol80 · #4

I now discovered something interesting:

If I use the flag DB_DUPSORT for the secondary db --- whether with my own comparison function or with the "built-in" comparison function --- the problem persists. If I use the flag DB_DUP for the secondary db instead I'm able to delete data from the primary db. But with DB_DUP the access via the secondary db doesn't work anymore. I didn't change anything but when I call

Code:

int ret = myCursor->pget(&skey, &key, &data, DB_SET);

ret has the value -30989 (= DB_NOTFOUND). How is that possible? Even if the entries in the secondary db are not sorted the first entry should be found!?

#5

Hi,

I can't see anything in what you've posted that would explain what you
are seeing. The quickest way to resolve this would be if you can write
a small test program that opens the two databases and associates them,
inserts a record, then tries to delete it and gets the error. If you
can send that to support (AT) sleepycat (DOT) com, I should be able to work out
what's going on.

Regards,
Michael.

gerol80 · #6

*grrrrrr* After hours of testing and despairing *g* I found the mistake. I think it was up to 2 different BerkeleyDB versions on my system.

On the Linux system I'm working on there is a global installation of BerkeleyDB 4.3.29. I wanted to use the newest BerkeleyDB version, so I installed version 4.4.20 locally for me. I think there was an mistake when I compiled the c++ source code (with include and/or library directories). I detected this by printing out DbEnv::version. It said Sleepycat Software: Berkeley DB 4.3.29 and not --- as I had expected --- Sleepycat Software: Berkeley DB 4.4.20. So I played around with the include and library directories a little bit --- and now it works :-)

I think the problem was that the compiler and linker used the version 4.3.29 but at runtime the shared library of version 4.4.20 was used. Perhaps it would be advantageous if the right version could somehow be checked at runtime.

gerol80 · #7

Sorry for the inconvenience *g* ... but now I have a new problem:

I have 8 database files which I open in a single environment. With the first 5 files everything is ok. But when my application tries to open the 6th file, there is an error:

file spiroTrials.db (meta pgno = 0) has LSN [1][17643].
end of log is [1][649]
BerkeleyDB/spiroTrials.db: unexpected file type or format

I understood that the LSN is the log sequence number, but I have no idea why it should be 17643 for this file. The found end of log 649 makes more sense to me.

Opening the database only works if the database file doesn't exist or the database contains no entries. When I put a entry into the database, exit the application and run it again, the error occurs --- but only with this one file. I tried to delete all database files several times, so that the log information should be deleted, too, but the error remains the same.

Additionally the error only occurs, if I use

Code:

environment->set_flags(DB_LOG_INMEMORY, 1);

But I don't want to write the logs to the disk because the application is being designed for an embedded device where disk space is limited.

I think it's up to the file size. The affected file is by far the largest file. I put more entries into the file and then the error message was

file spiroTrials.db (meta pgno = 0) has LSN [1][244442].
end of log is [1][649]
BerkeleyDB/spiroTrials.db: unexpected file type or format

So the [1][244442] is larger, the [1][649] is the same. And if I put even more data into the db files, the problem occurs with the other files, too. The only way to avoid the problem is to use logging in files.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

"Secondary index corrupt: not consistent with primary" when deleting data

comp.databases.berkeley-db comp.databases.berkeley-db

"Secondary index corrupt: not consistent with primary" when deleting data - 05-10-2006 , 09:19 AM

Re: "Secondary index corrupt: not consistent with primary" when deleting data - 05-10-2006 , 05:20 PM

05-11-2006 , 01:19 AM

05-11-2006 , 01:47 AM

Re: "Secondary index corrupt: not consistent with primary" when deleting data - 05-11-2006 , 02:18 AM

05-11-2006 , 05:02 AM

05-11-2006 , 07:28 AM