Never ending Checkpoint

IDS 9.40.FC4W2
SunOS 5.8

Last Saturday we had a "never ending checkpoint". There are no loggings
about the event except the lines in online.log:

11:41:01 Fuzzy Checkpoint Completed: duration was 2 seconds, 2789 buffers
not flushed,
timestamp: 0x60364810.
11:41:01 Checkpoint loguniq 1497261, logpos 0x18a94d4, timestamp:
0x60364810

11:41:01 Maximum server connections 358
11:41:09 Logical Log 1497261 Complete, timestamp: 0x6036ed0c.
11:41:43 Logical Log 1497262 Complete, timestamp: 0x603a11fd.
11:41:45 Logical Log 1497243 - Backup Started
11:41:54 Logical Log 1497243 - Backup Completed
11:41:54 Logical Log 1497244 - Backup Started
11:41:57 Logical Log 1497244 - Backup Completed
11:41:57 Logical Log 1497245 - Backup Started
11:42:00 Logical Log 1497245 - Backup Completed
11:42:00 Logical Log 1497246 - Backup Started
11:42:03 Logical Log 1497246 - Backup Completed
11:42:03 Logical Log 1497247 - Backup Started
11:42:06 Logical Log 1497247 - Backup Completed
11:42:06 Logical Log 1497248 - Backup Started
11:42:10 Logical Log 1497248 - Backup Completed
11:42:10 Logical Log 1497249 - Backup Started
11:42:13 Logical Log 1497249 - Backup Completed
11:42:13 Logical Log 1497250 - Backup Started
11:42:16 Logical Log 1497250 - Backup Completed
11:42:16 Logical Log 1497251 - Backup Started
13:45:54 Process exited with return code 137: /bin/sh /bin/sh -c
/opt/lib/informix/etc/log_full.sh 2 23 "L
ogical Log 1497262 Complete, timestamp: 0x603a11fd." "Logical
14:26:23 WARNING: Checkpoint appears stalled and may not complete
before the database server shuts down.

14:31:51 IBM Informix Dynamic Server Started.

At 14:26:23 the server was shut down an onmode -ky and then startet with
oninit. The "checkpoint request" lasted nearly three hours yet.

I tried to find helpful advices in c.d.i about long lasting checkpoints and
return code 137 but couldn't find anything that gives me an answer.

Thanks in advance,
Reinhard.

Habichtsberg, Reinhard wrote:

IDS 9.40.FC4W2
SunOS 5.8

Last Saturday we had a "never ending checkpoint". There are no loggings
about the event except the lines in online.log:

....

11:42:16 Logical Log 1497251 - Backup Started
13:45:54 Process exited with return code 137: /bin/sh /bin/sh -c
/opt/lib/informix/etc/log_full.sh 2 23 "L
ogical Log 1497262 Complete, timestamp: 0x603a11fd." "Logical
14:26:23 WARNING: Checkpoint appears stalled and may not complete
before the database server shuts down.

14:31:51 IBM Informix Dynamic Server Started.

At 14:26:23 the server was shut down an onmode -ky and then startet with
oninit. The "checkpoint request" lasted nearly three hours yet.

I tried to find helpful advices in c.d.i about long lasting checkpoints and
return code 137 but couldn't find anything that gives me an answer.

Thanks in advance,
Reinhard.

It is too late to say anything about the nature of the issue as IDS
server has been restarted w/o getting diagnostics.
Next time you observe a hung checkpoint run the command

%> $INFORMIXDIR/etc/evidence.sh 3 XX af.hung 0 x 0 x 1176

as user informix on the command line.

That shall generate an af.hung file that captures most of the important
information. Also a shmem dump will get generated. which might be useful
for IDS support.

Generally hung checkpoints have two possible reasons:
- IDS bug , when some lockjed resources such as mutexes cause a deadlock
between various threads so the chekpoint can never happen .

- IDS waits for the IO subsystem to resturn for the checkpoint to complete.

Short term solution is always an IDS restart .
HTH
Tilman