dbTalk Databases Forums  

[BUGS] ISM shared memory on solaris

mailing.database.pgsql-bugs mailing.database.pgsql-bugs


Discuss [BUGS] ISM shared memory on solaris in the mailing.database.pgsql-bugs forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Josh Wilmes
 
Posts: n/a

Default [BUGS] ISM shared memory on solaris - 10-24-2003 , 10:16 AM






I hope this is the right place to send this.. the FAQ in the
distribution mentions http://www.PostgreSQL.org/bugs/bugs.php, which
doesn't work.

We've found that postgresql wasn't using ISM shared memory on solaris,
which theoretically would cost performance. The root cause in our case
was that the "solaris" define is not defined by our compilers or by
postgresql itself.

The patch below simple has it check SHM_SHARE_MMU instead, which should
work fine. I verified (with 'pmap') that the database is now using ISM
on its shared memory, after this patch was applied.

--Josh



--- sysv_shmem.c.orig 2002-09-04 13:31:24.000000000 -0700
+++ sysv_shmem.c 2003-10-23 12:52:26.756765000 -0700
@@ -143,7 +143,7 @@
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
-#if defined(solaris) && defined(__sparc__)
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
@@ -323,8 +323,8 @@
shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);
if (shmid < 0)
continue; /* failed: must
be some other app's */
-
-#if defined(solaris) && defined(__sparc__)
+
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else



---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly

Reply With Quote
  #2  
Old   
Bruce Momjian
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-24-2003 , 10:55 AM






--ELM1067010456-13526-0_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII


Yikes! We thought we were already using ISM on Solaris.

Would you test the attached patch? It uses _solaris_ rather than
SHM_SHARE_MMU in the define test. Does that work too?

---------------------------------------------------------------------------

Josh Wilmes wrote:
Quote:
I hope this is the right place to send this.. the FAQ in the
distribution mentions http://www.PostgreSQL.org/bugs/bugs.php, which
doesn't work.

We've found that postgresql wasn't using ISM shared memory on solaris,
which theoretically would cost performance. The root cause in our case
was that the "solaris" define is not defined by our compilers or by
postgresql itself.

The patch below simple has it check SHM_SHARE_MMU instead, which should
work fine. I verified (with 'pmap') that the database is now using ISM
on its shared memory, after this patch was applied.

--Josh



--- sysv_shmem.c.orig 2002-09-04 13:31:24.000000000 -0700
+++ sysv_shmem.c 2003-10-23 12:52:26.756765000 -0700
@@ -143,7 +143,7 @@
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
-#if defined(solaris) && defined(__sparc__)
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
@@ -323,8 +323,8 @@
shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);
if (shmid < 0)
continue; /* failed: must
be some other app's */
-
-#if defined(solaris) && defined(__sparc__)
+
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else



---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly

--
Bruce Momjian | http://candle.pha.pa.us
pgman (AT) candle (DOT) pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

--ELM1067010456-13526-0_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
Content-Disposition: inline; filename="/pgpatches/solaris"

Index: src/backend/port/sysv_shmem.c
================================================== =================
RCS file: /cvsroot/pgsql-server/src/backend/port/sysv_shmem.c,v
retrieving revision 1.21
diff -c -c -r1.21 sysv_shmem.c
*** src/backend/port/sysv_shmem.c 13 Oct 2003 22:47:15 -0000 1.21
--- src/backend/port/sysv_shmem.c 24 Oct 2003 15:46:03 -0000
***************
*** 133,139 ****
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
--- 133,139 ----
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(__solaris__) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
***************
*** 352,358 ****

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else
--- 352,358 ----

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(__solaris__) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else

--ELM1067010456-13526-0_
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

--ELM1067010456-13526-0_--


Reply With Quote
  #3  
Old   
Bruce Momjian
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-24-2003 , 11:19 AM



Josh Wilmes wrote:
Quote:
Nope, __solaris__ is not defined on our system either.
I thought our configure defined __portname__ for every platform, but I
don't see that anywhere, so it seems we rely on the compiler to supply
defines for the cpu and OS.

Does src/tools/ccsym show you your defines? I would like to have
something that identifies Solaris rather than something that checks for
ISM so that if the ISM define isn't found, we throw an error and we hear
about it.

--
Bruce Momjian | http://candle.pha.pa.us
pgman (AT) candle (DOT) pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly


Reply With Quote
  #4  
Old   
Josh Wilmes
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-24-2003 , 12:50 PM




Bruce Momjian wrote:

Quote:
Josh Wilmes wrote:

Nope, __solaris__ is not defined on our system either.


I thought our configure defined __portname__ for every platform, but I
don't see that anywhere, so it seems we rely on the compiler to supply
defines for the cpu and OS.

Does src/tools/ccsym show you your defines? I would like to have
something that identifies Solaris rather than something that checks for
ISM so that if the ISM define isn't found, we throw an error and we hear
about it.
That would be preferable- i didn't know what was safe to assume would
always be defined.

ccsym is pretty neat. Here's what it shows (gcc)

__GNUC__=2
__GNUC_MINOR__=95
sparc
sun
unix
__svr4__
__SVR4
__sparc__
__sun__
__unix__
__svr4__
__SVR4
__sparc
__sun
__unix
system=unix
system=svr4
__GCC_NEW_VARARGS__
cpu=sparc
machine=sparc



---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings


Reply With Quote
  #5  
Old   
Josh Wilmes
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-24-2003 , 12:50 PM



Nope, __solaris__ is not defined on our system either.

--Josh

Bruce Momjian wrote:

Quote:
Yikes! We thought we were already using ISM on Solaris.

Would you test the attached patch? It uses _solaris_ rather than
SHM_SHARE_MMU in the define test. Does that work too?

---------------------------------------------------------------------------

Josh Wilmes wrote:

I hope this is the right place to send this.. the FAQ in the
distribution mentions http://www.PostgreSQL.org/bugs/bugs.php, which
doesn't work.

We've found that postgresql wasn't using ISM shared memory on solaris,
which theoretically would cost performance. The root cause in our case
was that the "solaris" define is not defined by our compilers or by
postgresql itself.

The patch below simple has it check SHM_SHARE_MMU instead, which should
work fine. I verified (with 'pmap') that the database is now using ISM
on its shared memory, after this patch was applied.

--Josh



--- sysv_shmem.c.orig 2002-09-04 13:31:24.000000000 -0700
+++ sysv_shmem.c 2003-10-23 12:52:26.756765000 -0700
@@ -143,7 +143,7 @@
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
-#if defined(solaris) && defined(__sparc__)
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
@@ -323,8 +323,8 @@
shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);
if (shmid < 0)
continue; /* failed: must
be some other app's */
-
-#if defined(solaris) && defined(__sparc__)
+
+#if defined(SHM_SHARE_MMU) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else



---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly




------------------------------------------------------------------------

Index: src/backend/port/sysv_shmem.c
================================================== =================
RCS file: /cvsroot/pgsql-server/src/backend/port/sysv_shmem.c,v
retrieving revision 1.21
diff -c -c -r1.21 sysv_shmem.c
*** src/backend/port/sysv_shmem.c 13 Oct 2003 22:47:15 -0000 1.21
--- src/backend/port/sysv_shmem.c 24 Oct 2003 15:46:03 -0000
***************
*** 133,139 ****
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
--- 133,139 ----
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(__solaris__) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
***************
*** 352,358 ****

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else
--- 352,358 ----

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(__solaris__) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend


Reply With Quote
  #6  
Old   
Bruce Momjian
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-25-2003 , 11:56 PM



--ELM1067143985-1620-0_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII


OK, attached patch applied. I tested for 'sun' rather than 'solaris'.

---------------------------------------------------------------------------

Josh Wilmes wrote:
Quote:
Bruce Momjian wrote:

Josh Wilmes wrote:

Nope, __solaris__ is not defined on our system either.


I thought our configure defined __portname__ for every platform, but I
don't see that anywhere, so it seems we rely on the compiler to supply
defines for the cpu and OS.

Does src/tools/ccsym show you your defines? I would like to have
something that identifies Solaris rather than something that checks for
ISM so that if the ISM define isn't found, we throw an error and we hear
about it.

That would be preferable- i didn't know what was safe to assume would
always be defined.

ccsym is pretty neat. Here's what it shows (gcc)

__GNUC__=2
__GNUC_MINOR__=95
sparc
sun
unix
__svr4__
__SVR4
__sparc__
__sun__
__unix__
__svr4__
__SVR4
__sparc
__sun
__unix
system=unix
system=svr4
__GCC_NEW_VARARGS__
cpu=sparc
machine=sparc



---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

--
Bruce Momjian | http://candle.pha.pa.us
pgman (AT) candle (DOT) pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

--ELM1067143985-1620-0_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
Content-Disposition: inline; filename="/bjm/diff"

Index: src/backend/port/sysv_shmem.c
================================================== =================
RCS file: /cvsroot/pgsql-server/src/backend/port/sysv_shmem.c,v
retrieving revision 1.21
diff -c -c -r1.21 sysv_shmem.c
*** src/backend/port/sysv_shmem.c 13 Oct 2003 22:47:15 -0000 1.21
--- src/backend/port/sysv_shmem.c 26 Oct 2003 04:52:33 -0000
***************
*** 133,139 ****
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
--- 133,139 ----
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(sun) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
***************
*** 352,358 ****

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else
--- 352,358 ----

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(sun) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else

--ELM1067143985-1620-0_
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

--ELM1067143985-1620-0_--


Reply With Quote
  #7  
Old   
Bruce Momjian
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-25-2003 , 11:57 PM



--ELM1067144134-1620-1_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII


Please use new patch. This uses __sun__ rather than sun.

---------------------------------------------------------------------------

Josh Wilmes wrote:
Quote:
Bruce Momjian wrote:

Josh Wilmes wrote:

Nope, __solaris__ is not defined on our system either.


I thought our configure defined __portname__ for every platform, but I
don't see that anywhere, so it seems we rely on the compiler to supply
defines for the cpu and OS.

Does src/tools/ccsym show you your defines? I would like to have
something that identifies Solaris rather than something that checks for
ISM so that if the ISM define isn't found, we throw an error and we hear
about it.

That would be preferable- i didn't know what was safe to assume would
always be defined.

ccsym is pretty neat. Here's what it shows (gcc)

__GNUC__=2
__GNUC_MINOR__=95
sparc
sun
unix
__svr4__
__SVR4
__sparc__
__sun__
__unix__
__svr4__
__SVR4
__sparc
__sun
__unix
system=unix
system=svr4
__GCC_NEW_VARARGS__
cpu=sparc
machine=sparc



---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

--
Bruce Momjian | http://candle.pha.pa.us
pgman (AT) candle (DOT) pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

--ELM1067144134-1620-1_
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
Content-Disposition: inline; filename="/bjm/diff"

Index: src/backend/port/sysv_shmem.c
================================================== =================
RCS file: /cvsroot/pgsql-server/src/backend/port/sysv_shmem.c,v
retrieving revision 1.21
diff -c -c -r1.21 sysv_shmem.c
*** src/backend/port/sysv_shmem.c 13 Oct 2003 22:47:15 -0000 1.21
--- src/backend/port/sysv_shmem.c 26 Oct 2003 04:52:33 -0000
***************
*** 133,139 ****
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
--- 133,139 ----
on_shmem_exit(IpcMemoryDelete, Int32GetDatum(shmid));

/* OK, should be able to attach to the segment */
! #if defined(__sun__) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
***************
*** 352,358 ****

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(solaris) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else
--- 352,358 ----

hdr = (PGShmemHeader *) shmat(*shmid,
UsedShmemSegAddr,
! #if defined(__sun__) && defined(__sparc__)
/* use intimate shared memory on Solaris */
SHM_SHARE_MMU
#else

--ELM1067144134-1620-1_
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

--ELM1067144134-1620-1_--


Reply With Quote
  #8  
Old   
Tom Lane
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-27-2003 , 12:19 PM



Bruce Momjian <pgman (AT) candle (DOT) pha.pa.us> writes:

Quote:
! #if defined(sun) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
I think this is going in the wrong direction. Why isn't the code just

#if defined(SHM_SHARE_MMU)
/* use intimate shared memory on Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else

If the symbol is available I think we probably want to use it. It is an
O/S issue, not a hardware issue, and so the test on __sparc__ seems
quite wrongheaded ...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly


Reply With Quote
  #9  
Old   
Bruce Momjian
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-27-2003 , 12:33 PM



Tom Lane wrote:
Quote:
Bruce Momjian <pgman (AT) candle (DOT) pha.pa.us> writes:

! #if defined(sun) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else

I think this is going in the wrong direction. Why isn't the code just

#if defined(SHM_SHARE_MMU)
/* use intimate shared memory on Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else

If the symbol is available I think we probably want to use it. It is an
O/S issue, not a hardware issue, and so the test on __sparc__ seems
quite wrongheaded ...
What I was hoping to do with the define test was to throw an error if we
don't find intimate shared memory on Solaris, but the define doesn't
work fir i386/Solaris so we are probably better going with the define as
you suggest --- I just hope we don't fail to include a file and somehow
miss it on some version of Solaris.

Change applied.

--
Bruce Momjian | http://candle.pha.pa.us
pgman (AT) candle (DOT) pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html


Reply With Quote
  #10  
Old   
Josh Wilmes
 
Posts: n/a

Default Re: [BUGS] ISM shared memory on solaris - 10-27-2003 , 02:32 PM



Seems like the BEST case would be to have a configure test verify that
it works and define something if it does, but i don't know what such a
test would look like.

--Josh


Bruce Momjian wrote:

Quote:
Tom Lane wrote:

Bruce Momjian <pgman (AT) candle (DOT) pha.pa.us> writes:


! #if defined(sun) && defined(__sparc__)
/* use intimate shared memory on SPARC Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else

I think this is going in the wrong direction. Why isn't the code just

#if defined(SHM_SHARE_MMU)
/* use intimate shared memory on Solaris */
memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else

If the symbol is available I think we probably want to use it. It is an
O/S issue, not a hardware issue, and so the test on __sparc__ seems
quite wrongheaded ...


What I was hoping to do with the define test was to throw an error if we
don't find intimate shared memory on Solaris, but the define doesn't
work fir i386/Solaris so we are probably better going with the define as
you suggest --- I just hope we don't fail to include a file and somehow
miss it on some version of Solaris.

Change applied.



---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings


Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.