PL/Perl(U) internals curiosity

#1

Hi, just a curiosity, how does PG manage PL/Perl? Is it loaded on the
same process space of PG (a la mod_perl) or is it called via IPC?
This question can be referred to other languages integrated in PG too.
Thank you in advance for explanation.

#2

Anselmo Canfora wrote:

Quote:

Hi, just a curiosity, how does PG manage PL/Perl? Is it loaded on the same process space of PG (a la mod_perl) or is it called via
IPC?
This question can be referred to other languages integrated in PG too.

They all work the same by design. As the documentation explains,
you can even add your own languages.

http://www.postgresql.org/docs/curre...ic/xplang.html has:

The [language] handler itself is a C language function compiled
into a shared object and loaded on demand, just like any other C function.

They are shared objects that are loaded into the backend process.
You can see them in your libdir as pl<name>.<shlib-extension>

Yours,
Laurenz Albe

#3

Il 07/01/2010 11.31, Laurenz Albe ha scritto:
[CUT]

Quote:

They all work the same by design. As the documentation explains,
you can even add your own languages.

http://www.postgresql.org/docs/curre...ic/xplang.html has:

The [language] handler itself is a C language function compiled
into a shared object and loaded on demand, just like any other C function.

They are shared objects that are loaded into the backend process.
You can see them in your libdir as pl<name>.<shlib-extension

Thank you for the clarification, I see these:

root@pluto:/usr/lib/postgresql/8.3/lib# ll pl*
-rw-r--r-- 1 root root 70K 2009-12-15 20:26 plperl.so
-rw-r--r-- 1 root root 148K 2009-12-15 20:25 plpgsql.so

now I am wondering how PG manage the perl interpreter, I am not a C
programmer, so I do not know a lot of things, however I would like to
understand something about the performance of perl stored procedures in
PG, for example if they are managed in a similar manner of
apache/mod_perl, and if they are as fast as it in same tasks.
It seems that the language handler loads the interpreter as shared library:

root@pluto:/usr/lib/postgresql/8.3/lib# ldd pl*
plperl.so:
linux-vdso.so.1 => (0x00007fff083fe000)
libperl.so.5.8 => /usr/lib/libperl.so.5.8 (0x00007fa3ffdd8000)
libdl.so.2 => /lib/libdl.so.2 (0x00007fa3ffbd4000)
libm.so.6 => /lib/libm.so.6 (0x00007fa3ff952000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007fa3ff736000)
libc.so.6 => /lib/libc.so.6 (0x00007fa3ff3d4000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fa3ff19b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa40032e000)
plpgsql.so:
linux-vdso.so.1 => (0x00007ffff5ffe000)
libc.so.6 => /lib/libc.so.6 (0x00007f36ed887000)
/lib64/ld-linux-x86-64.so.2 (0x00007f36ede15000)

And here:
http://www.postgresql.org/docs/curre...FUNC-C-DYNLOAD
it is stated that "Presently, unloads are disabled and will never occur,
but this may change in the future", so is it correct to think that the
libperl.so* is kept in memory after the first invocation? If yes, is it
the perl stored procedure AST compiled on creation and then kept in
memory after the first invocation?
Another question, there are not references to language handlers here?

root@pluto:/usr/lib/postgresql/8.3# ldd bin/post* | grep -i pl
root@pluto:/usr/lib/postgresql/8.3#

sorry for the slew of questions, I am very curious about these things

#4

On Thu, 07 Jan 2010 15:54:40 +0100, Anselmo Canfora wrote:

Quote:

And here:
http://www.postgresql.org/docs/curre....html#XFUNC-C-
DYNLOAD
it is stated that "Presently, unloads are disabled and will never occur,
but this may change in the future", so is it correct to think that the
libperl.so* is kept in memory after the first invocation? If yes, is it
the perl stored procedure AST compiled on creation and then kept in
memory after the first invocation?
Another question, there are not references to language handlers here?

1) The library is opened using dlopen library function which is described
here: http://linux.die.net/man/3/dlopen. This function is defined in
/usr/lib/libdl.so. Essentially, you can read the name of the library
that you want to open from a config or a database file and open it
using dlopen. After that, all the symbols from the library are
available to your program. Run time linker acts behind the scene here.
2) Plperl is probably a hacked version of perl.so, also known as "perl
lib". That is an open source project which makes perl interpreter
available for embedding. It must be hacked, in order to distinguish
between "trusted" and "untrusted' operations. I haven't checked the
source code, but that's what I expect it to be.
3) Perl code is never compiled. It's stored as a string. The entire
string is passed to the embedded perl interpreter which executes
it. Each server process (aka "handler process" maps the embedded perl
interpreter in its address space, so there aren't any multi-threading
problems. I don't use Winduhs so I don't know how are things there.
Winduhs OS uses threads instead of processes which may or may not
present some problems with multi-threading.

--
http://mgogala.byethost5.com

#5

Il 08/01/2010 0.27, Mladen Gogala ha scritto:

Quote:

On Thu, 07 Jan 2010 15:54:40 +0100, Anselmo Canfora wrote:

And here:
http://www.postgresql.org/docs/curre....html#XFUNC-C-
DYNLOAD
it is stated that "Presently, unloads are disabled and will never occur,
but this may change in the future", so is it correct to think that the
libperl.so* is kept in memory after the first invocation? If yes, is it
the perl stored procedure AST compiled on creation and then kept in
memory after the first invocation?
Another question, there are not references to language handlers here?

1) The library is opened using dlopen library function which is described
here: http://linux.die.net/man/3/dlopen. This function is defined in
/usr/lib/libdl.so. Essentially, you can read the name of the library
that you want to open from a config or a database file and open it
using dlopen. After that, all the symbols from the library are
available to your program. Run time linker acts behind the scene here.
2) Plperl is probably a hacked version of perl.so, also known as "perl
lib". That is an open source project which makes perl interpreter
available for embedding. It must be hacked, in order to distinguish
between "trusted" and "untrusted' operations. I haven't checked the
source code, but that's what I expect it to be.
3) Perl code is never compiled. It's stored as a string. The entire
string is passed to the embedded perl interpreter which executes
it. Each server process (aka "handler process" maps the embedded perl
interpreter in its address space, so there aren't any multi-threading
problems. I don't use Winduhs so I don't know how are things there.
Winduhs OS uses threads instead of processes which may or may not
present some problems with multi-threading.

Perl interpreter does a sort of "compilation", see for example:
http://www.perlfoundation.org/perl5/...gi?optree_guts
I wish to know if the result of this compilation is kept in memory as it
does for apache mod_perl

#6

On Fri, 08 Jan 2010 10:44:36 +0100, Anselmo Canfora wrote:

Quote:

Perl interpreter does a sort of "compilation", see for example:
http://www.perlfoundation.org/perl5/...gi?optree_guts I wish to know
if the result of this compilation is kept in memory as it does for
apache mod_perl

Of course it is. It is not saved to the disk, if that's what you think.
It's a perl optimization, done by the interpreter on the fly.

--
http://mgogala.freehostia.com

#7

Il 08/01/2010 14.19, Mladen Gogala ha scritto:

Quote:

On Fri, 08 Jan 2010 10:44:36 +0100, Anselmo Canfora wrote:

Perl interpreter does a sort of "compilation", see for example:
http://www.perlfoundation.org/perl5/...gi?optree_guts I wish to know
if the result of this compilation is kept in memory as it does for
apache mod_perl

Of course it is. It is not saved to the disk, if that's what you think.
It's a perl optimization, done by the interpreter on the fly.

mod_perl does not save on disk optrees

do you know mod_perl?

#8

On Fri, 08 Jan 2010 18:01:26 +0100, Anselmo Canfora wrote:

Quote:

Il 08/01/2010 14.19, Mladen Gogala ha scritto:
On Fri, 08 Jan 2010 10:44:36 +0100, Anselmo Canfora wrote:

Perl interpreter does a sort of "compilation", see for example:
http://www.perlfoundation.org/perl5/...gi?optree_guts I wish to
know if the result of this compilation is kept in memory as it does
for apache mod_perl

Of course it is. It is not saved to the disk, if that's what you think.
It's a perl optimization, done by the interpreter on the fly.

mod_perl does not save on disk optrees

do you know
mod_perl?

No, I don't know mod_perl. I am an oracle DBA in the process of becoming
acquainted with Postgres. I must confess to liking PostgreSQL better than
I thought I would.

--
http://mgogala.byethost5.com

#9

Il 08/01/2010 20.28, Mladen Gogala ha scritto:

Quote:

On Fri, 08 Jan 2010 18:01:26 +0100, Anselmo Canfora wrote:

Il 08/01/2010 14.19, Mladen Gogala ha scritto:
On Fri, 08 Jan 2010 10:44:36 +0100, Anselmo Canfora wrote:

Perl interpreter does a sort of "compilation", see for example:
http://www.perlfoundation.org/perl5/...gi?optree_guts I wish to
know if the result of this compilation is kept in memory as it does
for apache mod_perl

Of course it is. It is not saved to the disk, if that's what you think.
It's a perl optimization, done by the interpreter on the fly.

mod_perl does not save on disk optrees

do you know
mod_perl?

No, I don't know mod_perl. I am an oracle DBA in the process of becoming
acquainted with Postgres. I must confess to liking PostgreSQL better than
I thought I would.

Ok, you just can't fulfill my curiosity, thank you for the the attempt.

#10

Anselmo Canfora wrote:

Quote:

now I am wondering how PG manage the perl interpreter, I am not a C programmer, so I do not know a lot of things, however I would
like to understand something about the performance of perl stored procedures in PG, for example if they are managed in a similar
manner of apache/mod_perl, and if they are as fast as it in same tasks.

I don't know much about Perl and nothing about mod_perl, so I cannot
compare the two.

Quote:

It seems that the language handler loads the interpreter as shared library:

Yes, that is the way that you embed Perl into a C program.
See "man perlembed" for details.

Quote:

And here:
http://www.postgresql.org/docs/curre...FUNC-C-DYNLOAD
it is stated that "Presently, unloads are disabled and will never occur, but this may change in the future", so is it correct to
think that the libperl.so* is kept in memory after the first invocation? If yes, is it the perl stored procedure AST compiled on
creation and then kept in memory after the first invocation?

As soon as PL/Perl has been loaded into the backend process, it
will remain there. So it remains loaded for the rest of the session.

There will be one Perl interpreter created per backend, or two
if you use both trusted and untrusted Perl in the same session.

Stored procedures are realised as anonymous perl subroutines and
are cached (you can see that from the source; my Perl knowledge is
not sufficient to tell if that means that they are compiled only
once, but I'd guess so).

To see the PL/Perl implementation, look here:
http://anoncvs.postgresql.org/cvsweb...h_tag=REL8_4_2
(it is currently being rewritten, so it's better to look at a stable branch).

I don't know what AST is, could you explain?

Quote:

Another question, there are not references to language handlers here?

root@pluto:/usr/lib/postgresql/8.3# ldd bin/post* | grep -i pl
root@pluto:/usr/lib/postgresql/8.3#

Yes, because they are not linked with the database server. They
are loaded at runtime as soon as they are needed.

Yours,
Laurenz Albe