Hi all,

IDS 11.50.FC7W3

I found this in the message log (sorry, it's a bit long):

17:36:56 IBM Informix Dynamic Server Started.
17:36:56 Requested shared memory segment size rounded from 116736KB to
131072KB
17:36:56 Shared memory segment will use large pages with intimate
shared memory (ISM) if available
17:36:56 Segment locked: addr=10a000000, size=134217728

Sat Dec 4 17:36:56 2010

17:36:56 Event alarms enabled. ALARMPROG =
'/opt/informix/etc/log_full.sh'
17:36:56 Booting Language <c> from module <>
17:36:56 Loading Module <CNULL>
17:36:56 Booting Language <builtin> from module <>
17:36:56 Loading Module <BUILTINNULL>
17:36:56 SQLTRACE: SQL History Tracing set to 1000 SQL statements.
17:37:02 DR: DRAUTO is 0 (Off)
17:37:02 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled)
17:37:02 Fast poll /dev/poll enabled.
17:37:02 Requested shared memory segment size rounded from 560KB to
1024KB
17:37:02 IBM Informix Dynamic Server Version 11.50.FC7W3 Software
Serial Number AAA#B000000
17:37:02 listener-thread: err = -25572: oserr = 0: errstr = : Network
driver cannot bind a name to the port.

17:37:02 sql_listener: ASF_LISTEN failed
17:37:02 Attempting to bring listener thread down.
17:37:03 Performance Advisory: The physical log size is smaller than
the recommended size for a
server configured with RTO_SERVER_RESTART.
17:37:03 Results: Fast recovery performance might not be optimal.
17:37:03 Action: For best fast recovery performance when
RTO_SERVER_RESTART is enabled,
increase the physical log size to at least 110000 KB. For servers
configured with a large buffer pool, this might not be necessary.
17:37:03 IBM Informix Dynamic Server Initialized -- Shared Memory
Initialized.

17:37:03 Started 1 B-tree scanners.
17:37:03 B-tree scanner threshold set at 5000.
17:37:03 B-tree scanner range scan size set to -1.
17:37:03 B-tree scanner ALICE mode set to 6.
17:37:03 B-tree scanner index compression level set to med.
17:37:03 Physical Recovery Started at Page (1:2760).
17:37:03 Physical Recovery Complete: 0 Pages Examined, 0 Pages
Restored.
17:37:03 Logical Recovery Started.
17:37:03 10 recovery worker threads will be started.
17:37:04 Logical Recovery has reached the transaction cleanup phase.
17:37:04 Logical Recovery Complete.
0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
..
..
..
08:56:16 Dynamically allocated new virtual shared memory segment (size
4096KB)
08:56:16 Dynamically allocated new virtual shared memory segment (size
4096KB).

15:07:31 dynamically allocated 10000 locks
15:57:32 Dynamically allocated new virtual shared memory segment (size
4096KB)
15:57:32 Dynamically allocated new virtual shared memory segment (size
4096KB)
15:57:33 Dynamically allocated new virtual shared memory segment (size
4096KB)
11:19:32 dynamically allocated 20000 locks
11:19:48 Dynamically allocated new virtual shared memory segment (size
5120KB)
11:19:48 dynamically allocated 40000 locks
11:20:12 Dynamically allocated new virtual shared memory segment (size
10240KB)
11:20:12 dynamically allocated 80000 locks
13:06:28 Dynamically allocated new virtual shared memory segment (size
20480KB)
13:06:28 dynamically allocated 160000 locks
13:06:40 Dynamically allocated new virtual shared memory segment (size
40960KB)
13:06:40 dynamically allocated 320000 locks
07:39:36 Dynamically allocated new virtual shared memory segment (size
4096KB)
07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)
07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)
07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)
07:39:38 Dynamically allocated new virtual shared memory segment (size
4096KB)
08:46:26 Dynamically allocated new virtual shared memory segment (size
81920KB)
08:46:26 dynamically allocated 640000 locks
07:41:12 Dynamically allocated new virtual shared memory segment (size
4096KB)
14:16:19 Dynamically allocated new virtual shared memory segment (size
126976KB)
14:16:19 Memory sizes:resident:131072 KB, virtual:382976 KB,
SHMTOTAL:2000000 KB
14:16:20 dynamically allocated 1000000 locks
..
..
Thu Jan 13
shmget: [ENOSPC][28]: key 52574815: The server could not allocate shared
memory because an operating system limit was reached
08:00:58 out of virtual shared memory
08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
08:00:58 rsread_init: buffer error
..
..
..
Mon Jan 17

10:49:16 Requested shared memory segment size rounded from 126500KB to
126976KB
10:49:16 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached
10:49:16 The out of shared memory error has been encountered 5 times
since it was last printed to the log.
10:49:16 out of virtual shared memory

10:49:16 Assert Warning: Lock table overflow - user id 22223, session
id 272389

Today Fri Feb 04 I had to kill the master demon, no reaction from onbar
(log files were full), onmode. Restart to quiescent mode and backup the
log files with ontape brought the server up again.

I count 2,280,000 locks = resident memory for locks = 261 MB + 130 MB
BUFFERS + 380 MB VIRT. SEC. = 772 MB what is fare away from SHMTOTAL.

Operating system is Solaris 10 on SPARC. We have multiple instances of
IDS on this host in parallel.

How can I determine which operating limit was reached.

TIA,
Reinhard

It's either the number of shared memory segments per process or the maximum
shared memory per process.

Art

Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art (AT) iiug (DOT) org)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.



On Fri, Feb 4, 2011 at 5:02 AM, Habichtsberg, Reinhard <
RHabichtsberg (AT) arz-emmendingen (DOT) de> wrote:

On 04/02/2011 10:47, Art Kagel wrote:

1. Why is SHMVIRTSIZE set to be less than "maximum usage at peak time"?
2. Why is SHMADD so small?
3. Why are the locks needed to be dynamically allocated (i.e. should the
process lock the whole table? Or should you have a higher LOCKS value to
start with?)

Jonny,

I have an open ticket with IBM regarding similar issue ("...operating system limit was reached...), but on a RHEL 5.6 32bit and IDS 11.7UC1GE.

Are you using HugePages? If so, how much HugePages is configured and how much of it is available? In RHEL...

# cat /proc/meminfo | grep HugePages

Hi Art,



yes, you are right. All ids instances run under the "default"-project in
Solaris 10. The limit of number of shared mem segs ist 128 (default). We
will create new projects with bigger values.



Thanks to all.



Reinhard.



From: Art Kagel [mailto:art.kagel (AT) gmail (DOT) com]
Sent: Friday, February 04, 2011 11:47 AM
To: Habichtsberg, Reinhard
Cc: informix-list (AT) iiug (DOT) org
Subject: Re: operating system limit was reached



It's either the number of shared memory segments per process or the
maximum shared memory per process.

Art

Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art (AT) iiug (DOT) org)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated
nor those of the entities themselves.




On Fri, Feb 4, 2011 at 5:02 AM, Habichtsberg, Reinhard
<RHabichtsberg (AT) arz-emmendingen (DOT) de> wrote:

Hi all,

IDS 11.50.FC7W3

I found this in the message log (sorry, it's a bit long):

17:36:56 IBM Informix Dynamic Server Started.

17:36:56 Requested shared memory segment size rounded from 116736KB to
131072KB

17:36:56 Shared memory segment will use large pages with intimate
shared memory (ISM) if available

17:36:56 Segment locked: addr=10a000000, size=134217728

Sat Dec 4 17:36:56 2010

17:36:56 Event alarms enabled. ALARMPROG =
'/opt/informix/etc/log_full.sh'

17:36:56 Booting Language <c> from module <>

17:36:56 Loading Module <CNULL>

17:36:56 Booting Language <builtin> from module <>

17:36:56 Loading Module <BUILTINNULL>

17:36:56 SQLTRACE: SQL History Tracing set to 1000 SQL statements.

17:37:02 DR: DRAUTO is 0 (Off)

17:37:02 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled)

17:37:02 Fast poll /dev/poll enabled.

17:37:02 Requested shared memory segment size rounded from 560KB to
1024KB

17:37:02 IBM Informix Dynamic Server Version 11.50.FC7W3 Software
Serial Number AAA#B000000

17:37:02 listener-thread: err = -25572: oserr = 0: errstr = : Network
driver cannot bind a name to the port.

17:37:02 sql_listener: ASF_LISTEN failed

17:37:02 Attempting to bring listener thread down.

17:37:03 Performance Advisory: The physical log size is smaller than
the recommended size for a

server configured with RTO_SERVER_RESTART.

17:37:03 Results: Fast recovery performance might not be optimal.

17:37:03 Action: For best fast recovery performance when
RTO_SERVER_RESTART is enabled,

increase the physical log size to at least 110000 KB. For servers

configured with a large buffer pool, this might not be necessary.

17:37:03 IBM Informix Dynamic Server Initialized -- Shared Memory
Initialized.

17:37:03 Started 1 B-tree scanners.

17:37:03 B-tree scanner threshold set at 5000.

17:37:03 B-tree scanner range scan size set to -1.

17:37:03 B-tree scanner ALICE mode set to 6.

17:37:03 B-tree scanner index compression level set to med.

17:37:03 Physical Recovery Started at Page (1:2760).

17:37:03 Physical Recovery Complete: 0 Pages Examined, 0 Pages
Restored.

17:37:03 Logical Recovery Started.

17:37:03 10 recovery worker threads will be started.

17:37:04 Logical Recovery has reached the transaction cleanup phase.

17:37:04 Logical Recovery Complete.

0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks

..

..

..

08:56:16 Dynamically allocated new virtual shared memory segment (size
4096KB)

08:56:16 Dynamically allocated new virtual shared memory segment (size
4096KB).

15:07:31 dynamically allocated 10000 locks

15:57:32 Dynamically allocated new virtual shared memory segment (size
4096KB)

15:57:32 Dynamically allocated new virtual shared memory segment (size
4096KB)

15:57:33 Dynamically allocated new virtual shared memory segment (size
4096KB)

11:19:32 dynamically allocated 20000 locks

11:19:48 Dynamically allocated new virtual shared memory segment (size
5120KB)

11:19:48 dynamically allocated 40000 locks

11:20:12 Dynamically allocated new virtual shared memory segment (size
10240KB)

11:20:12 dynamically allocated 80000 locks

13:06:28 Dynamically allocated new virtual shared memory segment (size
20480KB)

13:06:28 dynamically allocated 160000 locks

13:06:40 Dynamically allocated new virtual shared memory segment (size
40960KB)

13:06:40 dynamically allocated 320000 locks

07:39:36 Dynamically allocated new virtual shared memory segment (size
4096KB)

07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)

07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)

07:39:37 Dynamically allocated new virtual shared memory segment (size
4096KB)

07:39:38 Dynamically allocated new virtual shared memory segment (size
4096KB)

08:46:26 Dynamically allocated new virtual shared memory segment (size
81920KB)

08:46:26 dynamically allocated 640000 locks

07:41:12 Dynamically allocated new virtual shared memory segment (size
4096KB)

14:16:19 Dynamically allocated new virtual shared memory segment (size
126976KB)

14:16:19 Memory sizes:resident:131072 KB, virtual:382976 KB,
SHMTOTAL:2000000 KB

14:16:20 dynamically allocated 1000000 locks

..

..

Thu Jan 13

shmget: [ENOSPC][28]: key 52574815: The server could not allocate shared
memory because an operating system limit was reached

08:00:58 out of virtual shared memory

08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

08:00:58 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

08:00:58 rsread_init: buffer error

..

..

..

Mon Jan 17

10:49:16 Requested shared memory segment size rounded from 126500KB to
126976KB

10:49:16 shmget: [ENOSPC][28]: key 52574815: The server could not
allocate shared memory because an operating system limit was reached

10:49:16 The out of shared memory error has been encountered 5 times
since it was last printed to the log.

10:49:16 out of virtual shared memory

10:49:16 Assert Warning: Lock table overflow - user id 22223, session
id 272389

Today Fri Feb 04 I had to kill the master demon, no reaction from onbar
(log files were full), onmode. Restart to quiescent mode and backup the
log files with ontape brought the server up again.

I count 2,280,000 locks = resident memory for locks = 261 MB + 130 MB
BUFFERS + 380 MB VIRT. SEC. = 772 MB what is fare away from SHMTOTAL.

Operating system is Solaris 10 on SPARC. We have multiple instances of
IDS on this host in parallel.

How can I determine which operating limit was reached.

TIA,

Reinhard


_______________________________________________
Informix-list mailing list
Informix-list (AT) iiug (DOT) org
http://www.iiug.org/mailman/listinfo/informix-list

On Feb 4, 4:02*am, "Habichtsberg, Reinhard" <RHabichtsb...@arz-
emmendingen.de> wrote:
Well, if you believe the man page for shmget() which is where the
message in the MSGPATH file indicates the error is coming from

ENOSPC A shared memory identifier is to be created
but the system-imposed limit on the maximum
number of allowed shared memory identifiers
system-wide would be exceeded. See NOTES.


I also think it could mean the maximum number of segments per process,
but that man page certainly doesn't make that clear. I did some quick
google searches because pre solaris 10, you could set these limits by
using the /etc/system file with the 2 following parameters:

set shmsys:shminfo_shmmni=100
set shmsys:shminfo_shmseg=100

The shminfo_shmmni is the maximum number of shared memory identifiers,
system wide. So in this example you could only ever see 100 entries
in an ipcs -m output. Then shminfo_shmseg is max number of
identifiers per process.

However, as I mentioned, in solaris 10 it seems that these parameters
could possibly be obsolete, however, if the entries are present in in
the /etc/system file, they will get picked up and used. I have almost
no solaris 10 experience so I'm not sure on the recommended method for
tuning these. But these would have been the parameters you would have
been looking for prior to solaris 10.

Jacques Renaut
IBM Informix Advanced Support
APD Team

Thanks Jacques,

we did some searching in the Solaris 10 manuals and found that the
number of segments allowed is counted per "project". You can list
project with projects -l:

projects -l
system
projid : 0
comment: ""
users : (none)
groups : (none)
attribs:
user.root
projid : 1
comment: ""
users : (none)
groups : (none)
attribs:
noproject
projid : 2
comment: ""
users : (none)
groups : (none)
attribs:
default
projid : 3
comment: ""
users : (none)
groups : (none)
attribs:
group.staff
projid : 10
comment: ""
users : (none)
groups : (none)
attribs:

All informix instances run per default in project "default".

With prctl -i project 3 you can report the values:
For example:
project.max-shm-ids
privileged 128

This is one of the limiting values we have to increase. If we monitor
the current used shm id's with
ipcs -m | wc -l:
123

we see that we are near the limit.

FYI and thanks, Reinhard.