Hello,

I've been using mysql++ for some time, and I've run into a very annoying
performance issue. This email contains an example demonstrating the
problems (and Makefile for build and python script to create the db).

Fetching the records from a query takes more time per record for each
record retrieved. I've stumbled across this when iterating a resultset
containing 500k+ records. The first ones take somewhere around 40 us per
record, but that time increases linearly with the number of records
retrieved to 2371 us after record 49000. (This behaviour is observed on
multiple machines).

I think the problem may be caused by the const_subscript_container as a
base for Result, but I have no idea how to go about solving this.

I hope any of you can help me,
Thanks
Erwin

--
erwin (AT) leastwanted (DOT) nl

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

On Thu, Apr 28, 2005 at 11:01:20PM +0200, Erwin wrote:
[snip]

This sounds like fun to track down, but unfortunately I'm kinda busy this
week with paying work. :-) Hopefully soon.

Until then, have you tried store()/use() differences? Do they both have
the same performance? I suspect that with a huge data set, you'd be
using use(), but would be interesting to see if there is any difference.

- Chris


--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Erwin wrote:

Chris nailed it: you should be using Query::use(), and therefore ResUse
instead. Instead of trying to store all the records in memory, a "use"
query grabs the records from the server one at a time, and you process
them linearly. It's like the difference between a random access
iterator into a vector and an input iterator into an I/O stream.

I'm working on the documentation now, and will add a section to make
this clearer.

P.S. Documentation team, yer Fired! ...until I finish hacking on the
docs anyway. Don't want to have duplicate effort...

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Warren Young wrote:
Ok by me. I haven't had a lick of time to start on it, and it isn't looking good
soon.

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Yes, that completely solved it. However I used the Result because it has
the nice iterator, which ResUse does not have. So, I'm writing one in
stead, which I will submit later this week. (If anyone's interested).

That would be very nice; the documentation was a bit 'lacking'.

Erwin

--
erwin (AT) leastwanted (DOT) nl

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Erwin wrote:

I'm not sure what you're saying. Did you do this:

Result r = q.use(...)

? If so, that't not legitimate. You can't force a 'use' query result
to be random-access.

"One" what? Do you mean you're adding an iterator to ResUse? If so, be
sure it works like an STL input or forward iterator. I'm not sure
exactly what the semantics should be, but random-access is certainly out
of the question. Be sure it doesn't conflict with Result's iterator,
too, since it subclasses from ResUse.

Here's a direct link:

http://tangentsoft.net/mysql++/doc/u...html#id2805203

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Hello Warren e.a.

Sorry for being unclear. I was using Query::store(), which returns a
Result, because I liked the fact that Result has an iterator. However,
as this is a random-access iterator, it apparently slows down for each
row used (funnily enough, without increasing memory use, and O(n^2),
instead of O(n) which I would expect).

I am creating an inputiterator, since semantics for a forwarditerator
state that you can 'start from the beginning', and get the same result.
Currently, I don't see how that behaviour can be implemented without
redoing the query, or storing the results. Also, the iterator I need
only needs inputiterator semantics.

I will, but thanx for the reminder.

Thank you.

Erwin
--
erwin (AT) leastwanted (DOT) nl
The human genome is about 3 gigabases long, which boils down to 750
megabytes. Depressingly enough, this is only 2.8 Mozilla browsers."
-- Jamie Zawinski

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw

Erwin wrote:
I suspect that what's happening is that the memory management overhead
is getting to you. "store" queries are just not designed to cope with
large amounts of data.

--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe: http://lists.mysql.com/plusplus?unsu...ie.nctu.edu.tw