 | |
Reasoning behind process instead of thread based arch?
comp.databases.postgresql.general comp.databases.postgresql.general
 |
| | Thread Tools | Display Modes |
#1
 
| |||
| |||
|
Reasoning behind process instead of thread based arch? -
10-27-2004
, 10:56 AM
#2
| |||
| |||
|
|
"The developers agree that multiple processes provide more benefits (mostly in stability and robustness) than costs (more connection startup costs). The startup costs are easily overcome by using connection pooling. " Please explain why it is more stable and robust? |
corrupt the entire system by writing to the wrong part of memory.
With separate processes, the only data that is shared is that which is
meant to be shared, which reduces the potential for such damage.
Quote:
|
"Also, each query can only use one processor; a single query can't be executed in parallel across many CPUs. However, several queries running concurrently will be spread across the available CPUs." And it is because of the PostgreSQL process architecture that a query can't be executed by many CPU:s right? |
multiple helper processes, but no one's implemented that feature--it
would be a pretty major job.
Quote:
|
Also, MySQL has a library for embedded aplications, the say: "We also provide MySQL Server as an embedded multi-threaded library that you can link into your application to get a smaller, faster, easier-to-manage product." Do PostgreSQL offer anything similar? |
this.
-Doug
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)
#3
| |||
| |||
|
|
Hello! I have a couple of final ( I hope, for your sake ) questions regarding PostgreSQL. I understand PostgreSQL uses processes rather than threads. I found this statement in the archives: "The developers agree that multiple processes provide more benefits (mostly in stability and robustness) than costs (more connection startup costs). The startup costs are easily overcome by using connection pooling. " Please explain why it is more stable and robust? More from the above statement: |
archives (I use google to do that, but YMMV with the postgresql site's
search engine.)
Basically, there are a few issues with threading that pop up their ugly
heads. One: Not all OSes thread libraries are created equal. There
was a nasty bug in one of the BSDs that causes MySQL to crash a couple
years ago that drove them nuts. So programming a threaded
implementation means you have the vagaries of different levels of
quality and robustness of thread libraries to deal with. Two: If a
single process in a multi-process application crashes, that process
alone dies. The buffer is flushed, and all the other child processes
continue happily along. In a multi-threaded environment, when one
thread dies, they all die. Three: Multi-threaded applications can be
prone to race conditions that are VERY hard to troubleshoot, especially
if they occur once every million or so times the triggering event
happens.
On some operating systems, like Windows and Solaris, processes are
expensive, while threads are cheap, so to speak. this is not the case
in Linux or BSD, where the differences are much smaller, and the
multi-process design suffers no great disadvantage.
Quote:
|
"Also, each query can only use one processor; a single query can't be executed in parallel across many CPUs. However, several queries running concurrently will be spread across the available CPUs." And it is because of the PostgreSQL process architecture that a query can't be executed by many CPU:s right? Although I wonder if this is the case in MySQL. It only says in their manual that each connection is a thread. |
still be true, because the postgresql engine isn't designed to split off
queries into constituent parts to be executed by seperate threads or
processes. Conversely, if one wished to implement it, one could likely
patch postgresql to break up parts of queries to different child
processes of the current child process (grand child processes so to
speak) that would allow a query to hit multiple CPUs.
Quote:
|
Also, MySQL has a library for embedded aplications, the say: "We also provide MySQL Server as an embedded multi-threaded library that you can link into your application to get a smaller, faster, easier-to-manage product." Do PostgreSQL offer anything similar? |
extension, your database. Now, I'd argue that if I had to choose
between which database to have crash in the middle of transactions, I'd
pick PostgreSQL, it's generally considered a bad thing to have a
database crash mid transaction. PostgreSQL is more robust about crash
recovery, but still...
That's another subject that shows up every x months, an embedded version
of PostgreSQL. Basically, the suggestion is to use something like
SQLlite, which is built to be embedded, and therefore has a much lower
footprint than PostgreSQL could ever hope to achieve. No one wants
their embedded library using up gobs of RAM and disk space when it's
just handling one thread / process doing one thing. It's like
delivering Pizzas with a Ferrari, you could do it, it just eouldn't make
a lot of sense.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo (AT) postgresql (DOT) org
#4
| |||
| |||
|
|
I understand PostgreSQL uses processes rather than threads. I found this statement in the archives: "The developers agree that multiple processes provide more benefits (mostly in stability and robustness) than costs (more connection startup costs). The startup costs are easily overcome by using connection pooling." Please explain why it is more stable and robust? |
A critical problem in a thread could terminate the entire process
or corrupt its data. If the database server were threaded, such
problems would affect the entire server. With each connection
handled by a separate process, a critical error is more likely to
affect only the connection that had the problem; the rest of the
server survives unscathed.
Quote:
|
"Also, each query can only use one processor; a single query can't be executed in parallel across many CPUs. However, several queries running concurrently will be spread across the available CPUs." And it is because of the PostgreSQL process architecture that a query can't be executed by many CPU:s right? Although I wonder if this is the case in MySQL. It only says in their manual that each connection is a thread. |
it might simply be using one thread per connection instead of a one
process per connection. If that's the case, then queries executed
by a particular connection are still single-threaded, the same as
in PostgreSQL.
A database that uses a separate process for each connection could
still employ multiple threads within each process if somebody could
figure out a way to distribute a query amongst the threads. I don't
know what the PostgreSQL developers' thoughts on that are.
A disadvantage of threads is that some systems (e.g., FreeBSD 4)
implement threads in userland and threads don't take advantage of
multiple CPUs. On such systems, using multiple processes better
employs additional CPUs.
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly
#5
| ||||
| ||||
|
|
On some operating systems, like Windows and Solaris, processes are expensive, while threads are cheap, so to speak. this is not the case in Linux or BSD, where the differences are much smaller, and the multi-process design suffers no great disadvantage. |
connections or connection pooling to eliminate the process overhead.
Quote:
|
Actually, if it were converted to multi-threaded tomorrow, it would still be true, because the postgresql engine isn't designed to split off queries into constituent parts to be executed by seperate threads or processes. Conversely, if one wished to implement it, one could likely patch postgresql to break up parts of queries to different child processes of the current child process (grand child processes so to speak) that would allow a query to hit multiple CPUs. |
are corner cases but I rarely find that it is the CPU that is doing all
the work, thus splitting the query may not do you any good.
In theory I guess being able to break it up and execute it to different
CPUs could cause the results to process faster, but I wonder if it would
be a large enough benefit to even notice?
Quote:
|
"We also provide MySQL Server as an embedded multi-threaded library that you can link into your application to get a smaller, faster, easier-to-manage product." Do PostgreSQL offer anything similar? |
database you would Embed.
Quote:
|
pick PostgreSQL, it's generally considered a bad thing to have a database crash mid transaction. PostgreSQL is more robust about crash recovery, but still... That's another subject that shows up every x months, an embedded version of PostgreSQL. Basically, the suggestion is to use something like SQLlite, which is built to be embedded, and therefore has a much lower footprint than PostgreSQL could ever hope to achieve. No one wants their embedded library using up gobs of RAM and disk space when it's just handling one thread / process doing one thing. It's like delivering Pizzas with a Ferrari, you could do it, it just eouldn't make a lot of sense. ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo (AT) postgresql (DOT) org |
--
Command Prompt, Inc., home of PostgreSQL Replication, and plPHP.
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd (AT) commandprompt (DOT) com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
#6
| |||
| |||
|
|
Two: If a single process in a multi-process application crashes, that process alone dies. The buffer is flushed, and all the other child processes continue happily along. In a multi-threaded environment, when one thread dies, they all die. |
So this means that if a single connection thread dies in MySQL, all
connections die?
Seems rather serious. I am doubtful that is how they have implemented it.
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)
#7
| |||
| |||
|
|
Two: If a single process in a multi-process application crashes, that process alone dies. The buffer is flushed, and all the other child processes continue happily along. In a multi-threaded environment, when one thread dies, they all die. So this means that if a single connection thread dies in MySQL, all connections die? Seems rather serious. I am doubtful that is how they have implemented it. |
it's the *process* (ie all threads) that receives the signal and it's
the *process* that dies.
Just like a SIGSTOP stops all threads and a SIGTERM terminates them
all. Signals are shared between threads. Now, you could ofcourse catch
these signals but you only have one address space shared between all
the threads, so if you want to exit to get a new process image (because
something is corrupted), you have to close all connections.
And indeed, the one MySQL server I can see is four threads. Nasty.
--
Martijn van Oosterhout <kleptog (AT) svana (DOT) org> http://svana.org/kleptog/
Quote:
|
Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. |
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQFBf+RbY5Twig3Ge+YRAl/lAKCA97RjyQUOgfEXdSYRQf1FG+T8IQCeMCAu
M9tGr4AqyG6XmgT/TFul0BY=
=OrDg
-----END PGP SIGNATURE-----
#8
| |||
| |||
|
|
Two: If a single process in a multi-process application crashes, that process alone dies. The buffer is flushed, and all the other child processes continue happily along. In a multi-threaded environment, when one thread dies, they all die. So this means that if a single connection thread dies in MySQL, all connections die? Seems rather serious. I am doubtful that is how they have implemented it. That all depends on how you define crash. If a thread causes an |
floating point exception, the process will die, hence killing all
threads. But a more advanced multi-threaded environment will install
handlers for such signals that will handle the error gracefully. It
might not even be necesarry to kill the offending thread.
Some conditions are harder to handle than others, such as stack overflow
and out of memory, but it can be done. So to state that multi-threaded
environments in general kills all threads when one thread chrashes is
not true. Having said that, I have no clue as to how advanced MySQL is
in this respect.
Regards,
Thomas Hallgren
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)
#9
| |||
| |||
|
|
... Signals are shared between threads. Now, you could ofcourse catch these signals but you only have one address space shared between all the threads, so if you want to exit to get a new process image (because something is corrupted), you have to close all connections. |
would kill a thread and keep it from killing the whole process, but
this still leaves you with a process memory space that may or may not
be corrupted. Continuing in that situation is not cool, at least not
according to the Postgres project's notions of reliable software design.
It should be pointed out that when we get a hard backend crash, Postgres
will forcibly terminate all the backends and reinitialize; which means
that in terms of letting concurrent sessions keep going, we are not any
more forgiving than a single-address-space multithreaded server. The
real bottom line here is that we have good prospects of confining the
damage done by the failed process: it's unlikely that anything bad will
happen to already-committed data on disk or that any other sessions will
return wrong answers to their clients before we are able to kill them.
It'd be a lot harder to say that with any assurance for a multithreaded
server.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
#10
| |||
| |||
|
|
Right. Depending on your OS you may be able to catch a signal that would kill a thread and keep it from killing the whole process, but this still leaves you with a process memory space that may or may not be corrupted. Continuing in that situation is not cool, at least not according to the Postgres project's notions of reliable software design. There can't be any "may or may not" involved. You must of course know |
It is very common that you either get a null pointer exception (attempt
to access address zero), that your stack will hit a write protected page
(stack overflow), or that you get some sort of arithemtic exception.
These conditions can be trapped and gracefully handled. The signal
handler must be able to check the cause of the exception. This usually
involves stack unwinding and investingating the state of the CPU at the
point where the signal was generated. The process must be terminated if
the reason is not a recognized one.
Out of memory can be managed using thread local allocation areas
(similar to MemoryContext) and killing a thread based on some criteria
when no more memory is available. A criteria could be the thread that
encountered the problem, the thread that consumes the most memory, the
thread that was least recently active, or something else.
Quote:
|
It should be pointed out that when we get a hard backend crash, Postgres will forcibly terminate all the backends and reinitialize; which means that in terms of letting concurrent sessions keep going, we are not any more forgiving than a single-address-space multithreaded server. The real bottom line here is that we have good prospects of confining the damage done by the failed process: it's unlikely that anything bad will happen to already-committed data on disk or that any other sessions will return wrong answers to their clients before we are able to kill them. It'd be a lot harder to say that with any assurance for a multithreaded server. I'm not sure I follow. You will be able to bring all threads of one |
processes. Killing the multithreaded process is more like pulling the plug.
Regards,
Thomas Hallgren
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
|
« Previous Thread
|
Next Thread »
| Thread Tools | |
| Display Modes | |
Linear Mode
| |
Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.