Could someone explain me please how should I work with DB_RECOVERY and
DB_AUTO_COMMIT?

My problem is that after recovery procedure, my db is empty.
My env: Windows XP SP2, VS 2005, console app., DB version 4.4.20
My test:
1) open db
2) add 1 million records into DB in a loop
3) abort program in the middle of the loop
4) try to restore all the data and find a record

my code:
pDBEnv = new DbEnv(0);
pDBEnv->set_cachesize(0, 512*1024*1024, 1);
pDBEnv->open(dir_path, DB_CREATE | DB_THREAD | DB_INIT_MPOOL |
DB_INIT_TXN | DB_INIT_LOG | DB_INIT_LOCK | DB_RECOVER, 0);

pDBEnv->set_flags(DB_AUTO_COMMIT, 1);

pDB = new Db(pDBEnv, 0); // Instantiate the Db object
pDB->set_flags(DB_DUP);

pDB->open(NULL, // Transaction pointer
symbols.db, // Database file name
NULL, // Optional logical database name
DB_BTREE, // Database access method
DB_CREATE | DB_THREAD | DB_AUTO_COMMIT, // Open flags
0); // File mode (using defaults)

....
//write records
// ...

and terminate app.


I see log file and a bunch of __db files

I run: db_stat.exe -d symbols.db
it stalls without any output. I close it.

After that, I run my program again, but now instead of writing records,
I try to find a record I wrote first, and I could not find anything.

If I now run db_stat.exe -d symbols.db, I can see output but 0 records.

If I run db_recover.exe instead of running my program the second time,
it will delete all log files, db_stat.exe will show output with 0
records.

What do I do wrong? why the written data is empty? I do use
DB_AUTO_COMMIT flag, but it does not seem to work.

One work around I found is to make transactions
DbTxn *tx;
pDBEnv->txn_begin(0, &tx, DB_TXN_SYNC);
....
tx->commit(DB_TXN_SYNC);
But system works really slow then.

Any suggestions, ideas?

Dima

Would you please post the working program?

When did you teminate the application?


Ron

Hey, Ron!

I terminate the program (while of cause it is inserting records):
1) just at any time but before sync() or DB close. Crash happens not
all the time, just occasuaonly in this case. When I run to read data, I
get this message: Dbc::get: DB_PAGE_NOTFOUND: Requested page not found.
Verify, dump and recover utilities do not help at all. I spent 3 days
just reading docs and trying solutions. Nothing could restore the data.
2) when I do:
a) close db
b) sync()
Just when db writes the data into disk. Then 99% that data will be
currupted despite it was opened as transactional db with logs and
recovery features.

Sources (It is already big...)





#include "CSymbolDataBase.h"

CSymbolDataBase::CSymbolDataBase()
{
pDB = NULL;
pDBEnv = NULL;
}

CSymbolDataBase::~CSymbolDataBase(void)
{
CloseDB();
}

void ErrFunction(const DbEnv *, const char *a, const char *b)
{
int z = 0;
}

BOOL CSymbolDataBase::OpenDB(char *dir_path, char *db_name)
{
CloseDB();

pDBEnv = new DbEnv(0);

try
{
pDBEnv->set_errcall(&ErrFunction);
pDBEnv->set_cachesize(0, 512*1024*1024, 1);
pDBEnv->open(dir_path,
DB_CREATE | DB_THREAD | DB_INIT_MPOOL | DB_INIT_TXN | DB_INIT_LOG |
DB_INIT_LOCK | DB_RECOVER,
0);
pDBEnv->set_flags(DB_AUTO_COMMIT, 1);
// pDBEnv->set_flags(DB_TXN_WRITE_NOSYNC, 1);

pDB = new Db(pDBEnv, 0); // Instantiate the Db object
pDB->set_flags(DB_DUP);

pDB->open(NULL, // Transaction pointer
db_name, // Database file name
NULL, // Optional logical database name
DB_BTREE, // Database access method
DB_CREATE | DB_THREAD | DB_AUTO_COMMIT, // Open flags
0); // File mode (using defaults)

}
catch(DbException &e)
{
printf("<ERROR DbException> %s", e.what());
return FALSE;
}
catch(std::exception &e)
{
printf("ERROR: %s", e.what());
return FALSE;
}

return TRUE;
}

void CSymbolDataBase::CloseDB()
{
if(pDB)
pDB->close(0);
pDB = NULL;

if(pDBEnv)
pDBEnv->close(0);
pDBEnv = NULL;
}

void CSymbolDataBase::Flush()
{
pDB->sync(0);
}

void CSymbolDataBase::CheckPoint()
{
pDBEnv->txn_checkpoint(0, 0, DB_FORCE);
}

BOOL CSymbolDataBase::PutBar(char *Symbol, BAR_INFO *pBar)
{
Dbc *pCursor = NULL;
BOOL bError = FALSE;

// DbTxn *tx;
// pDBEnv->txn_begin(0, &tx, DB_TXN_SYNC);

try
{
// pDB->cursor(tx, &pCursor, 0);
pDB->cursor(0, &pCursor, 0);

Dbt key(Symbol, strlen(Symbol)+1);
Dbt data(pBar, sizeof(BAR_INFO));

Dbt data_get(pBar, sizeof(BAR_INFO));

// get last inserted record
int ret = pCursor->get(&key, &data_get, DB_SET);
if(ret == DB_NOTFOUND)
{
ret = pCursor->put(&key, &data, DB_KEYFIRST);
}
else
{
BAR_INFO *bi = ((BAR_INFO *)data_get.get_data());

// compare time, time should be bigger
if(bi->time < pBar->time)
ret = pCursor->put(&key, &data, DB_KEYFIRST);
}
}
catch(DbException &e)
{
bError = TRUE;
printf("<ERROR DbException> %s", e.what());
}
catch(std::exception &e)
{
bError = TRUE;
printf("ERROR: ");
}

if(pCursor)
pCursor->close();

// tx->commit(DB_TXN_SYNC);

return !bError;
}






// BerkeleyDB.cpp : Defines the entry point for the console
application.
//

#include "stdafx.h"
#include "conio.h"
#include <time.h>
#include "windows.h"
#include "CSymbolDataBase.h"

LARGE_INTEGER tBefore, tAfter, i64PerfFrequency;
__int64 dif;
double m_gDataBasePerfomance;

#define TIMER_START() \
QueryPerformanceFrequency( &i64PerfFrequency ); \
Sleep( 0 ); \
QueryPerformanceCounter( &tBefore )

#define TIMER_END() \
QueryPerformanceCounter( &tAfter ); \
dif = tAfter.QuadPart - tBefore.QuadPart; \
m_gDataBasePerfomance = (double)dif / i64PerfFrequency.QuadPart

char _random(int n)
{
static int val = 0;

val = rand();
return val % n;
}

int _tmain(int argc, _TCHAR* argv[])
{
srand(time(0));

CSymbolDataBase db;

db.OpenDB("db", "symbols.db");

char tmp[4];
// create symbols
for(int j = 0; j < 1; j++)
{
TIMER_START();

for(unsigned int i=0; i<1000000; i++)
{
char *symbol = strcpy( tmp, "IBM" );
symbol[ 0 ] = 'A' + _random( 25 );
symbol[ 1 ] = 'A' + _random( 25 );
symbol[ 2 ] = 'A' + _random( 25 );

BAR_INFO bar;
bar.time = i + rand() % 5000;

if(db.PutBar(symbol, &bar) == FALSE)
break;

if(i%100000 == 0)
{
//db.CheckPoint();
srand(time(0));
printf("%d\n", i);
}
}

TIMER_END();
printf("\n\ntime = %f sec\n", m_gDataBasePerfomance);
}
}





My goal was to make the program robust to crashes, but fast enough. To
commit transactions after every record - is a way too slow. Make sync -
too slow when cache (512mb) is full (all the time). Do checkpoint also
slow, but still does not help to avoid "Dbc::get: DB_PAGE_NOTFOUND" if
I terminate app on writing.

I already switched all my logic to MS SQL (I figured out how to use
bcp_ functions from c++). Just simple comparision: insert 1.000.000 of
random records (with complete commit) with the same conditions:
Berkeley DB: 72 sec, MS SQL 2000: 46 sec.

Anyway, I'm interested why Berkeley DB was not able to restore my data
and get rid of DB_PAGE_NOTFOUND automatically even having all kind of
logs.

Dima

Hi Dima,

The DB_AUTO_COMMIT flag does not apply to cursors. The documentation
about this is not clear and is being correct. You will need to use an
explicit transaction with the cursor.

Sorry for the inconvenience!

Ron