Hi,

I am working on a program to do analysis of data from the Wikipedia
(http://wikipedia.org) database using MySQL++.

The program reads a very large result (up to several gigabytes) from the
database and therefore uses mysqlpp::ResUse. However while analysis of
this large result set, it have to insert the results into other tables
in the database as they are found.

I discovered that you cannot send new queries on a connection while it
have an unfinished query using mysqlpp::ResUse - it gives a "Commands
out of sync; you can't run this command now"-error when you try.

To work around this, I created a second mysqlpp::Connection object to do
inserts and updates to the database while the first mysqlpp::Connection
object was occupied by the select query with the large result set.
However, after doing that my program began to misbehave (segfaults,
result data being changed in strange ways between two reads etc.).

I put Valgrind (http://valgrind.org/) on the program, and it reports
lots of invalid reads when I access data returned by
mysqlpp::ResUse::fetch_row(). Like this:

==23222== Invalid read of size 1
==23222== at 0x1B905488: strlen (mac_replace_strmem.c:189)
==23222== by 0x804B6D6: check_len(char const*, unsigned) (create.cpp:155)
==23222== by 0x804BB28: handle_link(unsigned, char const*)
(create.cpp:266)
==23222== by 0x804C19F: main (create.cpp:457)
==23222== Address 0x1BE35A98 is 24 bytes inside a block of size 37 free'd
==23222== at 0x1B906237: operator delete(void*) (vg_replace_malloc.c:156)
==23222== by 0x1B9D2DE4:
std::string::_Rep::_M_destroy(std::allocator<char> cons
t&) (in /usr/lib/libstdc++.so.6.0.3)
==23222== by 0x1B9D30CD: std::string::~string() (in
/usr/lib/libstdc++.so.6.0.3)
==23222== by 0x804CB14:
mysqlpp::ColData_Tmpl<mysqlpp::const_string>::~Col Data_T
mpl() (create.cpp:426)

So my question is: Do MySQL++ have a limitation so it only works with
one actice Connection object at a time, or do I have to do something
special to make it work with several active Connection objects?

And another question: Do you think that it would be more efficient to
store inserts + updates in for example a local file, and not send these
queries to the database until after the mysqlpp::ResUse object is deleted.

Thanks for any help.

Best regards,
Byrial Jensen

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Byrial Jensen wrote:
I regularly use multiple Connection objects and have not had any kind of
a problem with them. Are you using threading?

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Byrial Jensen wrote:
If it does, it's considered a bug. Patches thoughtfully considered.

Probably not. If anything, I'd make the second database local, and do
your final analysis there. It sounds like both databases are remote, in
which case your bottleneck is the bandwidth of the link to the database.
Unless you've got serious pipe, your disk bandwidth will be a lot higher.

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Warren Young wrote:
Thanks to Warren Young and Earl Miles for the replies. I don't have a
patch, but I do have a very much cut down program to demonstrate the
problems.

When I run the program below with the cur table from the database dump
found at
http://download.wikimedia.org/wikipe...r_table.sql.gz,
I get an abort in check_len() which seems to indicate that data returned
from ResUse::fetch_row() have been changed. I suspect that this may be
due to a mysql++ bug. It may of course also be due to misunderstandings
on my side. If so, please tell me what's wrong with the program.

When run with valgrind, I get thousands of invalid read errors. All
about reads inside blocks previously free'd by the
ColData_Tmpl<mysqlpp::const_string> destructor. Like this one:

==32674== Invalid read of size 1
==32674== at 0x1B905491: strlen (mac_replace_strmem.c:189)
==32674== by 0x804B16E: check_len(char const*, unsigned) (cr.cpp:47)
==32674== by 0x804B771: main (cr.cpp:145)
==32674== Address 0x1BE315AD is 13 bytes inside a block of size 41 free'd
==32674== at 0x1B906237: operator delete(void*) (vg_replace_malloc.c:156)
==32674== by 0x1B9D2DE4:
std::string::_Rep::_M_destroy(std::allocator<char> cons
t&) (in /usr/lib/libstdc++.so.6.0.3)
==32674== by 0x1B9D30CD: std::string::~string() (in
/usr/lib/libstdc++.so.6.0.3)
==32674== by 0x804BDD6:
mysqlpp::ColData_Tmpl<mysqlpp::const_string>::~Col Data_T
mpl() (cr.cpp:121)


============ Start code ============
#include <mysql++/mysql++.h>
#include <string>
#include <sstream>
#include <iostream>

using namespace std;
using namespace mysqlpp;

#define DATABASE_HOST NULL
#define DATABASE_USER NULL
#define DATABASE_PASSWD NULL
#define DATABASE_NAME "da_20050516"
#define DATABASE_PORT 0
#define DATABASE_SOCKET NULL

Connection con (true /* use exceptions */);
Connection con2 (true /* use exceptions */);

void do_query (const char *q)
{
Query query = con.query ();
query << q;
try {
query.execute ();
}
catch (exception& err) {
cerr << "\nError 1: " << err.what ();
exit (1);
}
}

void do_query_2 (ostringstream& o)
{
const string s = o.str ();
Query query = con2.query ();
try {
query.exec (s);
}
catch (exception& err) {
cerr << "\nError 2:" << err.what ();
exit (1);
}
}

void check_len (const char *start, size_t length)
{
size_t new_length = strlen (start);
if (new_length != length) {
cout << "check_len(): length=" << length
<< ", new_length=" << new_length << endl;
abort ();
}
}

const char *handle_link (unsigned id, const char *text)
{
ostringstream query;
query << "INSERT INTO links (from_id, link) VALUES(" << id << ",'";
for (; {
if (! *text) {
// Unmatched [[
return text;
}
if (*text == ']') {
break;
}
else {
if (*text == '\'' || *text == '\\') {
query << '\\';
}
query << *text;
}
++text;
}
query << "')";
do_query_2 (query);
return text + 1;
}

int main ()
{
try {
con.real_connect (DATABASE_NAME, DATABASE_HOST,
DATABASE_USER, DATABASE_PASSWD,
DATABASE_PORT,
false /* compress */,
10 /* timeout */,
DATABASE_SOCKET,
0 /* client_flag */
);

con2.real_connect (DATABASE_NAME, DATABASE_HOST,
DATABASE_USER, DATABASE_PASSWD,
DATABASE_PORT,
false /* compress */,
10 /* timeout */,
DATABASE_SOCKET,
0 /* client_flag */
);

}
catch (exception& err) {
cerr << "Connect error: " << err.what () << endl;
return 1;
}

do_query ("DROP TABLE IF EXISTS links");
do_query
("CREATE TABLE links ("
"from_id mediumint(7) unsigned NOT NULL,"
"link varchar(255) binary NOT NULL default '',"
"INDEX ( from_id )"
") ENGINE=MyISAM");

Query query = con.query ();
query << "SELECT cur_id, cur_text FROM cur";
try {
size_t count;
ResUse res = query.use ();
while (Row row = res.fetch_row ()) {
unsigned int id = row[0];
const char *text = row[1];
long unsigned *lengths = res.fetch_lengths ();
unsigned text_length = lengths[1];

if (memchr (text, '\0', text_length)) {
cout << "Warning: Article " << id
<< " contains NUL characters. Skiping it." << endl;
continue;
}
if (text[text_length] != '\0') {
cout << "Error: text not NUL terminated." << endl;
abort ();
}

const char *saved_start = text;
check_len (saved_start, text_length);
while (*text) {
if (*text == '[' && text[1] == '[') {
text = handle_link (id, text + 2);
}
else {
++text;
}
check_len (saved_start, text_length);
} // While text;
if (++count % 100 == 0) cout << '.' << flush;
} // While row
}
catch (BadQuery&) {
cout << " done" << endl;
}
catch (exception& err) {
cerr << "\nError: " << err.what ();
exit (1);
}
return 0;
}
============ End code ============

I wasn't clear. I only use a local database.
I use a PC with Mandriva Linux 2005 with MySQL 4.1.11,
gcc 3.4.3 and glib 2.3.4

mysql++ 1.7.35 is installed from locally build RPMs.

Best regards
Byrial

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Byrial Jensen wrote:
From looking this over, I think your problem might be string manipulation.

const char *text = row[1];

I'm not sure row[] is guaranteed to return a null terminated string.
Without a null terminator, strlen() is guaranteed to fail and fail
badly. I believe what you really want is this (slightly longer than it
has to be for clarity):

std::string tempString = row[1];
const char *test = tempString.c_str();

Really, what you likely want is to avoid char* where you can and use
std::string, which is a remarkably efficient string library, if somewhat
large. And mysql++ is using it all over the place, so you lose nothing
by using it, and gain a *great deal* of string safety by doing so.

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

On Wed, May 25, 2005 at 11:37:07AM -0700, Earl Miles wrote:
Yep, Earl is on the right track. See the declaration of operator[] in
row.h:

const ColData operator [] (size_type i) const;

This returns a temporary, which disappears after that line of code.
You need to make a copy for yourself.

- Chris


--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

On Wed, May 25, 2005 at 03:43:54PM -0400, Chris Frey wrote:
Replying to myself... ahem. :-)

As an alternative to makeing a copy, you can use "row[1]" everywhere,
instead of saving a pointer.

Note that ColData provides casting conversions, so you can cast that data
to std::string, int, long, etc, and ColData will try to make the conversion
for you. The temporary will still disappear though, so making your own copy is
likely the most efficient method.

- Chris


--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Chris Frey wrote:
Tho casting it to std::string automatically copies it; (of course, using
a temporary and then using c_str() off of that will still give you a
bogus result that will mostly work except when it won't)



--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Chris Frey wrote:
Now I have studied the mysql++ source code. row[] returns a temporary
ColData object and that object have a function to do conversion to
"const char *" which indeed gives a pointer to a NUL-terminated string.

However - as Chris points out - the ColData object is destructed
immidiately after the conversion, leaving the string pointer pointing to
freed memory, which caused all the trouble.

I replaced the line

const char *text = row[1];

with

const ColData text_col = row[1];
const char *text = text_col;

and that fixed all observed problems in my programs. Thank you very much
for the suggestion.

I don't know if it is considered a bug that you cannot write

const char *text = row[1];

but even if it isn't, it might be an idea to warn about the construction
in the manual.

Thanks to all helpful people in this thread.

Best regards
Byrial

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Byrial Jensen wrote:
The bug is in the construct, not in MySQL++. To make that work in
MySQL++, we'd have to return a pointer (or reference) to a static
object, and that opens up a whole new class of risks.

MySQL++ behaves reaonably in this regard. The question is how we make
people understand what the reasonable behavior is.

The only place an outright warning about this particular error will make
sense is in the reference manual, for Row's operator[]. But it might
also be good to have a general discussion of concepts in the userman.
Any ideas of where it should go, and how to approach it?

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw