I have played with the cryptography of BDB, only found that when being
given a wrong password, the DB->open method just returns -1, which
means: "Unknown Error".
Then I want to know is there any other mechanism to let me know it's
the wrong password that makes the opening failure instead of all other
possibilies. Or at least let me know it's some kind of format error or
checksum error rather than others.

Are you checking the return value of db->set_encrypt() ?

Parameters
flags
The flags parameter must be set to 0 or the following value:
DB_ENCRYPT_AES
Use the Rijndael/AES (also known as the Advanced Encryption Standard
and Federal Information Processing Standard (FIPS) 197) algorithm for
encryption or decryption.
passwd
The passwd parameter is the password used to perform encryption and
decryption.
Errors

The DB_ENV->set_encrypt method may fail and return one of the following
non-zero errors:
EINVAL
If the method was called after DB_ENV->open was called; or if an
invalid flag value or parameter was specified.
EOPNOTSUPP
Cryptography is not available in this Berkeley DB release.

MorningStar wrote:
You need to enable informational error messages and it will give you
additional
output to tell you what the problem it detect was. See
docs/api_c/env_set_errfile.html or env_set_errpfx.html in your
documentation set.

DB will detect bad passwords, no passwd in an encrypted env, etc.

Sue LoVerso
Sleepycat Software