Hi all,

we have an Informix-4ge program that stops with error 104 "too many open
files" while trying to open a violation table. Same program has run for
years without this error. The code has not been changed.

The schema of two tables has changed:

stop violations table for xxx_neu;

drop table xxx_neu_vio;
drop table xxx_neu_dia;

alter table xxx_neu add (verarb_status SMALLINT);

start violations table for xxx_neu;
set indexes xxx_n_1 filtering;

alter table yyy_neu add (anmelde_grund CHAR(2),
abmelde_grund CHAR(2),
svs67 CHAR(2));

Those columns are not in use of the program yet.

Parts of the program runs on new application server (Solaris 10 Zone).
May be the pogram runs faster than before.

The Informix Server is 11.50FC7W3. OS is Solaris 10.

We checked some limits on the appservers and the database server host.
Ulimit is set to 256 for the appusers and for user informix before
starting the databaseserver.

We couldn't see many open files on the appserver. The informix
databaseserver has more than 3000 concurrent sessions, most of them
ontlitcp connections.

What would you suggest?

TIA,
Reinhard.

In additon: The solaris host is configured to hold 65536 open files per
process.



*

* Process Resource Limit Tunables (Current:Maximum)

*

0x0000000000000100:0x0000000000010000 file descriptors

*



I understand that this value is the absolute kernel-limit for a shell
with all child-processes. We did a pfiles command on all
oninit-processes and count more than 63000 open file descriptores. They
can also be found in /proc/oninit-PID/fd.



Most of the file descriptors describe (as I assume) tcp connections:



21: S_IFCHR mode:0000 dev:305,0 ino:52962 uid:0 gid:0 rdev:146,718

O_RDWR|O_NDELAY

/devices/pseudo/tcp6@0:tcp6

22: S_IFCHR mode:0000 dev:305,0 ino:21689 uid:0 gid:0 rdev:146,2440

O_RDWR|O_NDELAY

/devices/pseudo/tcp6@0:tcp6



Most of the oninit processes (about 30) hold more than 3000 of those
descriptors. Do they count when reaching the limit of open files? It's
not very likely because user informix starts the server with an ulimit
of 256 files and this limit is by far exceeded. But who knows...?





From: informix-list-bounces (AT) iiug (DOT) org
[mailto:informix-list-bounces (AT) iiug (DOT) org] On Behalf Of Habichtsberg,
Reinhard
Sent: Wednesday, May 11, 2011 1:04 PM
To: informix-list (AT) iiug (DOT) org
Subject: Too many open files - Isam error 104



Hi all,

we have an Informix-4ge program that stops with error 104 "too many open
files" while trying to open a violation table. Same program has run for
years without this error. The code has not been changed.

The schema of two tables has changed:

stop violations table for xxx_neu;

drop table xxx_neu_vio;

drop table xxx_neu_dia;

alter table xxx_neu add (verarb_status SMALLINT);

start violations table for xxx_neu;

set indexes xxx_n_1 filtering;

alter table yyy_neu add (anmelde_grund CHAR(2),

abmelde_grund CHAR(2),

svs67 CHAR(2));

Those columns are not in use of the program yet.

Parts of the program runs on new application server (Solaris 10 Zone).
May be the pogram runs faster than before.

The Informix Server is 11.50FC7W3. OS is Solaris 10.

We checked some limits on the appservers and the database server host.
Ulimit is set to 256 for the appusers and for user informix before
starting the databaseserver.

We couldn't see many open files on the appserver. The informix
databaseserver has more than 3000 concurrent sessions, most of them
ontlitcp connections.

What would you suggest?

TIA,

Reinhard.

Hello Reinhard,


hmm not much anyway:

-->>Most of the oninit processes (about 30) hold more than 3000 of
those
-->>descriptors. Do they count when reaching the limit of open files?
It's
-->>not very likely because user informix starts the server with an
ulimit
-->>of 256 files and this limit is by far exceeded. But who knows...?

oninit is setuid root so it can increase it if i recall correctly.
oninit procs share filedesc's afaik since threads can switch between
vps.

i would try setting sqlidebug

eq

export SQLIDEBUG=2:/tmp/somefile
run the program and use sqliprint to dump the info.

also you could try and use truss
eq

truss ./yourprogram > somefile 2>&1

maybe you find a hint which may case this.

What does support say about this; i assume the repro is pretty easy??


See you

Superboer




On 11 mei, 15:04, "Habichtsberg, Reinhard" <RHabichtsb...@arz-
emmendingen.de> wrote:

The kernel max_files per process or for the system overall may be exceeded.

Art

Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.



On Thu, May 12, 2011 at 11:19 AM, Superboer <superboer7 (AT) t-online (DOT) de> wrote:

Try running "ulimit -a". Maybe you need to increase your limits.

-LS

Hi,

There are two FD limits in Solaris (Hard and Soft). The hard limit (rlim_fd_max) default and maximum value is 65536 so there is no need to do anything with that, the Soft limit default is 256, add the following to /etc/system and reboot

set rlim_fd_cur=4096 (Or a value you are happy with)



Regards

Colin



There are 10 types of people in the world, those that understand binary and those that don't





From: jjfahey (AT) gmail (DOT) com
To: RHabichtsberg (AT) arz-emmendingen (DOT) de; informix-list (AT) iiug (DOT) org
Subject: RE: Too many open files - Isam error 104
Date: Sat, 14 May 2011 14:10:24 -0400



Message body From: John Fahey [mailto:jfahey (AT) mandm (DOT) net]
Sent: Thursday, May 12, 2011 11:14 PM
To: 'Habichtsberg, Reinhard'; 'informix-list (AT) iiug (DOT) org'
Subject: RE: Too many open files - Isam error 104 Not sure if Solaris supports this; with Linux we put the following in < /etc/security/limits.conf > * soft nofile 131072 * hard nofile 262144 We do a < ulimit –n 256000 > (or so)! inthe Bourne shell script we use to start IDS, before we run < oninit > John Fahey From: informix-list-bounces (AT) iiug (DOT) org [mailto:informix-list-bounces (AT) iiug (DOT) org] On Behalf Of Habichtsberg, Reinhard
Sent: Wednesday, May 11, 2011 7:04 AM
To: informix-list (AT) iiug (DOT) org
Subject: Too many open files - Isam error 104 Hi all,we have an Informix-4ge program that stops with error 104 „too many open files“ while trying to open a violation table. Same program has run for years without this error. The code has not been changed.The schema of two tables has changed:stop violations table for xxx_neu;drop table xxx_neu_vio;drop table xxx_neu_dia;alter table xxx_neu add (verarb_status SMALLINT);start violationstable for xxx_neu;set indexes xxx_n_1 filtering;alter table yyy_neu add (anmelde_grund CHAR(2), abmelde_grund CHAR(2), svs67 CHAR(2));Those columns are not in use of the program yet.Parts of the program runs on new application server (Solaris 10 Zone). May be the pogram runs faster than before.The Informix Server is 11.50FC7W3. OS is Solaris 10. We checked some limits on the appservers and the database server host. Ulimit is set to 256 for the appusers and for user informix before starting the databaseserver.We couldn’t see many open files on the appserver. The informix databaseserver has more than 3000 concurrent sessions, most of them ontlitcp connections.What would you suggest?TIA,Reinhard.
_______________________________________________
Informix-list mailing list
Informix-list (AT) iiug (DOT) org
http://www.iiug.org/mailman/listinfo/informix-list

On May 11, 6:04*am, "Habichtsberg, Reinhard" <RHabichtsb...@arz-
emmendingen.de> wrote:
I'd suggest contacting support. I seriously doubt the 104 error on
the start violations would have anything to do with OS level file
descriptors. The sqlexec thread on the server would not be opening
anything additionally on that command, as a table in the server is not
a unix file construct. All the oninit processes would already have
all the chunks open, so the only OS file descriptors involved would be
the ones for chunks or for the network connections. We have an
internal concept of "file descriptor" that this is likely referring to
that relates to tables, but the limit on the number of these should be
very large, like 32k if I remember right. So I would think that that
error is either getting generated when it shouldn't, or possibly an
error is occurring but we're maybe reporting the wrong error or a
misleading error.

Jacques Renaut
IBM Informix Advanced Support
APD Team

Thanks to Jacques and everybody who answered. We are still trying to find the reason and we don't belief now that it is an issue with file descriptors. The support engineer from Fujitsu gave us the hint to modify the values of DD_HASHSIZE to a prime number. We changed the value from 99 to 503 and the value of DD_HASHMAX from 49 to 8. On Friday we'll restart with the new values and we'll see if the error persists. Until then we stopped violation on the table in question.