Hi Group,

OS : RHEL Version 4, update 3
D3 : 7.4.2.33
Hardware: HP ML 310 G4, 2GB RAM, E200 SATA Raid

I am having a problem where d3 is a new install on the above system.
The system is booted, the d3 VM is started (1GB allocated to D3 shared
memory), then I am copying all the RHEL rpm's onto the system. Linux
uses all the remaining memory for io buffers, then when I attempt to
login to D3, I get "Out of Memory" error and cannot connect to the VM.
If I wait a while for the buffers to flush, I can then connect to D3.

The second senerio is, reboot machine, copy all rpms onto the hard
drive, then attempt to start the VM, same error message, "Out of
Memory"

This is a new system, RHEL is installed as a bare bones system plus
required c compiler, not yet in production, the application is not
installed.

I was under the impression that when D3 starts up, and 1GB is setup in
pick0, D3 would grab the shared memory and lock it down, therefore what
ever processing is done at the OS level would never interfere with the
VM. This appears not to be happening.

I have searched this group and the d3 forum for clues, but no joy yet.
Does anyone have any thoughts or advise that may be usefull in
resolving this problem.

TIA

Barry


Does anyone any thoughts

Hi Glen,

Thanks for the input.

First, a correction to the original post, the system is using a HP E200
SAS RAID controller.

I have done further investigations and have made some progress.

Reading through some Oracle documentation and found a comment reporting
that the cfq io scheduler has a bug, Oracle strongly recommend using
the deadline io scheduler when using a smart HBA.

I have reconfigured the kernel to use the deadline io scheduler, and
following some brief testing was unable to reproduce the d3 "Out of
Memory" error.

I do not feel that the problem is with D3, my impression is that
something in linux is just not quite right on this hardware platform. I
will be doing a lot more research and testing today.

Cheers, Barry



Glen B wrote:

Barry,

I have seen situations where disk drivers have caused the sort of
problems you are experiencing (are chasing similar issues at a site
where suddenly 97% of CPU [per TOP] is being spent in I/O wait states)
.... luckily they will be moving to a new platform in a few weeks, so
not looking thaaat hard

Good luck


bgraetz (AT) bigpond (DOT) net.au wrote:

Barry,

I have seen situations where disk drivers have caused the sort of
problems you are experiencing (are chasing similar issues at a site
where suddenly 97% of CPU [per TOP] is being spent in I/O wait states)
.... luckily they will be moving to a new platform in a few weeks, so
not looking thaaat hard

Good luck


bgraetz (AT) bigpond (DOT) net.au wrote:

Hi Ross,

I have tracked the problem down to being the default cfq io scheduler
in
RHEL 4.

After checking the Oracle documentation, doing the rounds on the NET,
plus
my own testing, this is what I found:

The default RHEL (and SLES and CentOS) cfq io scheduler has a bug where

under heavy io will starve the other processes on the system. Also, the
io blocks
are only removed from the list head, not from the io queue, resulting
in poor
performance. This bug impacts file io as well as raw io (which is d3)

I dug up code samples and patches, but the long and the short of it is
that the
developers are still working on a solution in the current 2.6 kernel at
kernel.org
(ie, it is still not fixed)

I can confirm on the web that the broken cfq scheduler is in RHEL 4, U2
and
RHEL 4, U3.

I cannot confirm on the web if the broken cfq scheduler is in RHEL 4,
U4 but I
can confirm that after installing RHEL4, U4 on this HP system I was
able to
re-produce the "Out of Memory" errors, so I guess the cfq scheduler is
not fixed
in RHEL 4, U4.

I found bug reports at Bugzilla, Metalink, and on several linux kernel
threads.

The fix is simple as described at several places on the web, for
database
servers using RHEL4, U2 and U3 (and based on my own testing, U4 as
well),
modify grub.conf and instruct the kernel to use the 'deadline' io
scheduler.

Without going into too much detail, I can confirm that after testing
the deadline
io scheduler and the cfq io scheduler on the HP ML310 and an older IBM
x220,
the results for the deadline scheduler are very positive, I will alway
use the
deadline scheduler in the future on any server that is running the 2.6
kernel.

It relation to the disk HBA driver (and following advise you gave me in
a previous
posting), I have ensured that the cciss drivers, firmware, and bios are
all at the
same level on the HP, as well as on the IBM.

Do you ever get the feeling that you know way to much detail about
something,
well after today this is how I feel about the io schedulers in the 2.6
kernel.

Thanks for taking the time to respond.

Cheers,

Barry


Ross Ferris wrote:

I wonder if the issue you found impacts U1, which is what we install
(am only just now considering U3 - I don't mind being behind the
bleeding edge for some things).

In terms of knowing "too much", lets just say that one of my stock
lines goes something like "Computers are such a great concept that has
appeal to everyone - it's a shame that hardware & software have to come
into the picture to stuff things up, but the concept is GREAT!!"


bgraetz (AT) bigpond (DOT) net.au wrote:

<bgraetz (AT) bigpond (DOT) net.au> wrote
Bummer. As a workaround, maybe it will help to do a
linux sync command just before each d3
command... Is easy to test out, anyway...

If it helps, then make a hijack alias for
d3 to be 'sync ; /usr/lib/pick/d3.orig '