Based on your description, it sounds like you're hitting a table without an index that has grown in size and a single SQL/SPL/UDR is to blame.
I've attached a script called "qio" to the bottom of the post, it will show you the SQL's in the system that are the biggest I/O users. It ain't pretty, but it works.
Jay
Hi,
The processing is OLTP. A SUN 4800 server with 4 CPUs
at 750MHZ each. The overall CPU activity is about 65%.
The server has a total of 1.8TB (18 disks of 73GB).
Out of this 400GB is allocated to the file system and
the rest is configured like this: 6 for RAID 10 and 5
for RAID 5. 2 for hot standby and 2 for parity. Memory
is 8GB.
Well, my worry is the response from the system has
slowed down drastically and the users are beginning to
complain. we have done index rebuilding,
defragmentation on the tables but this has not
improved.
i have attached the results of onstat -d as well.
--- "Konigsberg, Jay" <jaykon (AT) tower (DOT) com> wrote:
Quote:
What, precisely, is it that you are unhappy with?
Are you OLTP, DSS?
How many CPU's?
How many hard drives?
What is your overall CPU activity?
What does you I/O look like?
Etc, etc, etc ...
-----Original Message-----
From: owner-informix-list (AT) iiug (DOT) org
[mailto wner-informix-list (AT) iiug (DOT) org] On Behalf Of
Leona Ankrah
Sent: Thursday, August 18, 2005 7:52 AM
To: informix-list (AT) iiug (DOT) org
Subject: Onstat -P output
File: 77641972-profilereport >> << File:
1832273169-configuration
Hi,
I'm using IDS9.21 UC6 running on a SOlaris 8.0
Operating system.
The output of the onstat -P is not acceptable and I
would appreciate it if I could be helped get a
better
results. The system is not performing well enough.
Attached is the output of the onstat -p.
I have also attached the onconfig file for the
database configuration.
please help.
Leona
Ghana Telecom Limited
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam
protection around
http://mail.yahoo.com |
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Informix Dynamic Server 2000 Version 9.21.UC6 -- On-Line -- Up 1 days 10:14:29 -- 3063808 Kbytes
Dbspaces
address number flags fchunk nchunks flags owner name
b310d7d0 1 0x1 1 5 N informix rootdbs
b3844018 2 0x2001 4 4 N T informix temp1dbs
b3844160 3 0x2001 5 4 N T informix temp2dbs
b38442a8 4 0x1 6 33 N informix datadbs1r5
b38443f0 5 0x1 36 30 N informix datadbs2r10
b3844538 6 0x1 56 30 N informix datadbs3r5
b3844680 7 0x1 85 30 N informix datadbs4r10
b38447c8 8 0x1 106 34 N informix indexdbs
b3844910 9 0x1 126 10 N informix smalldbs
9 active, 2047 maximum
Chunks
address chk/dbs offset size free bpages flags pathname
b310d918 1 1 0 1048575 523224 PO- /dev/online10_1
b314c5f0 2 1 0 1048575 0 PO- /dev/online10_2
b314c760 3 1 0 1048575 54429 PO- /dev/online10_3
b314c8d0 4 2 0 1048575 1046964 PO- /dev/online10_4
b314ca40 5 3 0 1048575 1046204 PO- /dev/online10_5
b314cbb0 6 4 0 1048575 113068 PO- /dev/online5_1
b314cd20 7 4 0 1048575 0 PO- /dev/online5_2
b314ce90 8 4 0 1048575 0 PO- /dev/online5_3
b310da88 9 4 0 1048575 0 PO- /dev/online5_4
b310dbf8 10 4 0 1048575 524286 PO- /dev/online5_5
b310dd68 11 4 0 1048575 0 PO- /dev/online5_6
b313da30 12 4 0 1048575 0 PO- /dev/online5_7
b313dba0 13 4 0 1048575 0 PO- /dev/online5_8
b313dd10 14 4 0 1048575 0 PO- /dev/online5_9
b313de80 15 4 0 1048575 0 PO- /dev/online5_10
b382c018 16 4 0 1048575 0 PO- /dev/online5_11
b382c188 17 4 0 1048575 0 PO- /dev/online5_12
b382c2f8 18 4 0 1048575 0 PO- /dev/online5_13
b382c468 19 4 0 1048575 0 PO- /dev/online5_14
b382c5d8 20 4 0 1048575 0 PO- /dev/online5_15
b382c748 21 4 0 1048575 605928 PO- /dev/online5_16
b382c8b8 22 4 0 1048575 524286 PO- /dev/online5_17
b382ca28 23 4 0 1048575 524286 PO- /dev/online5_18
b382cb98 24 4 0 1048575 0 PO- /dev/online5_19
b382cd08 25 4 0 1048575 524286 PO- /dev/online5_20
b382ce78 26 4 0 1048575 561429 PO- /dev/online5_21
b382d018 27 4 0 1048575 104607 PO- /dev/online5_22
b382d188 28 4 0 1048575 575845 PO- /dev/online5_23
b382d2f8 29 4 0 1048575 0 PO- /dev/online5_24
b382d468 30 4 0 1048575 575845 PO- /dev/online5_25
b382d5d8 31 4 0 1048575 0 PO- /dev/online5_26
b382d748 32 4 0 1048575 1048572 PO- /dev/online5_27
b382d8b8 33 4 0 1048575 1048572 PO- /dev/online5_28
b382da28 34 4 0 1048575 1048572 PO- /dev/online5_29
b382db98 35 4 0 1048575 1048572 PO- /dev/online5_30
b382dd08 36 5 0 1048575 48581 PO- /dev/online10_6
b382de78 37 5 0 1048575 0 PO- /dev/online10_7
b382e018 38 5 0 1048575 0 PO- /dev/online10_8
b382e188 39 5 0 1048575 0 PO- /dev/online10_9
b382e2f8 40 5 0 1048575 99593 PO- /dev/online10_10
b382e468 41 5 0 1048575 0 PO- /dev/online10_11
b382e5d8 42 5 0 1048575 0 PO- /dev/online10_12
b382e748 43 5 0 1048575 0 PO- /dev/online10_13
b382e8b8 44 5 0 1048575 0 PO- /dev/online10_14
b382ea28 45 5 0 1048575 0 PO- /dev/online10_15
b382eb98 46 5 0 1048575 0 PO- /dev/online10_16
b382ed08 47 5 0 1048575 0 PO- /dev/online10_17
b382ee78 48 5 0 1048575 0 PO- /dev/online10_18
b382f018 49 5 0 1048575 0 PO- /dev/online10_19
b382f188 50 5 0 1048575 0 PO- /dev/online10_20
b382f2f8 51 5 0 1048575 524286 PO- /dev/online10_21
b382f468 52 5 0 1048575 524286 PO- /dev/online10_22
b382f5d8 53 5 0 1048575 0 PO- /dev/online10_23
b382f748 54 5 0 1048575 524286 PO- /dev/online10_24
b382f8b8 55 5 0 1048575 561429 PO- /dev/online10_25
b382fa28 56 6 0 1048575 176821 PO- /dev/online5_31
b382fb98 57 6 0 1048575 0 PO- /dev/online5_32
b382fd08 58 6 0 1048575 0 PO- /dev/online5_33
b382fe78 59 6 0 1048575 0 PO- /dev/online5_34
b3830018 60 6 0 1048575 262143 PO- /dev/online5_35
b3830188 61 6 0 1048575 0 PO- /dev/online5_36
b38302f8 62 6 0 1048575 0 PO- /dev/online5_37
b3830468 63 6 0 1048575 0 PO- /dev/online5_38
b38305d8 64 6 0 1048575 0 PO- /dev/online5_39
b3830748 65 6 0 1048575 0 PO- /dev/online5_40
b38308b8 66 6 0 1048575 0 PO- /dev/online5_41
b3830a28 67 6 0 1048575 0 PO- /dev/online5_42
b3830b98 68 6 0 1048575 0 PO- /dev/online5_43
b3830d08 69 6 0 1048575 0 PO- /dev/online5_44
b3830e78 70 6 0 1048575 0 PO- /dev/online5_45
b3833018 71 6 0 1048575 0 PO- /dev/online5_46
b3833188 72 6 0 1048575 517106 PO- /dev/online5_47
b38332f8 73 6 0 1048575 524286 PO- /dev/online5_48
b3833468 74 6 0 1048575 0 PO- /dev/online5_49
b38335d8 75 6 0 1048575 524286 PO- /dev/online5_50
b3833748 76 6 0 1048575 524286 PO- /dev/online5_51
b38338b8 77 6 0 1048575 786429 PO- /dev/online5_52
b3833a28 78 6 0 1048575 0 PO- /dev/online5_53
b3833b98 79 6 0 1048575 1048572 PO- /dev/online5_54
b3833d08 80 6 0 1048575 1048572 PO- /dev/online5_55
b3833e78 81 6 0 1048575 1048572 PO- /dev/online5_56
b3834018 82 6 0 1048575 1048572 PO- /dev/online5_57
b3834188 83 6 0 1048575 1048572 PO- /dev/online5_58
b38342f8 84 6 0 1048575 1048572 PO- /dev/online5_59
b3834468 85 7 0 1048575 222643 PO- /dev/online10_26
b38345d8 86 7 0 1048575 0 PO- /dev/online10_27
b3834748 87 7 0 1048575 0 PO- /dev/online10_28
b38348b8 88 7 0 1048575 0 PO- /dev/online10_29
b3834a28 89 7 0 1048575 524286 PO- /dev/online10_30
b3834b98 90 7 0 1048575 0 PO- /dev/online10_31
b3834d08 91 7 0 1048575 0 PO- /dev/online10_32
b3834e78 92 7 0 1048575 0 PO- /dev/online10_33
b3835018 93 7 0 1048575 0 PO- /dev/online10_34
b3835188 94 7 0 1048575 0 PO- /dev/online10_35
b38352f8 95 7 0 1048575 0 PO- /dev/online10_36
b3835468 96 7 0 1048575 0 PO- /dev/online10_37
b38355d8 97 7 0 1048575 0 PO- /dev/online10_38
b3835748 98 7 0 1048575 0 PO- /dev/online10_39
b38358b8 99 7 0 1048575 524286 PO- /dev/online10_40
b3835a28 100 7 0 1048575 0 PO- /dev/online10_41
b3835b98 101 7 0 1048575 0 PO- /dev/online10_42
b3835d08 102 7 0 1048575 524286 PO- /dev/online10_43
b3835e78 103 7 0 1048575 524286 PO- /dev/online10_44
b3836018 104 7 0 1048575 517106 PO- /dev/online10_45
b3836188 105 6 0 1048575 1048572 PO- /dev/online5_60
b38362f8 106 8 0 1048575 329 PO- /dev/online10_46
b3836468 107 8 0 1048575 12438 PO- /dev/online5_61
b38365d8 108 8 0 1048575 10815 PO- /dev/online10_47
b3836748 109 8 0 1048575 25920 PO- /dev/online5_62
b38368b8 110 8 0 1048575 8264 PO- /dev/online10_48
b3836a28 111 8 0 1048575 72981 PO- /dev/online5_63
b3836b98 112 8 0 1048575 67778 PO- /dev/online10_49
b3836d08 113 8 0 1048575 36683 PO- /dev/online5_64
b3836e78 114 8 0 1048575 108798 PO- /dev/online10_50
b3837018 115 8 0 1048575 91261 PO- /dev/online5_65
b3837188 116 8 0 1048575 3785 PO- /dev/online10_51
b38372f8 117 8 0 1048575 107204 PO- /dev/online5_66
b3837468 118 8 0 1048575 61989 PO- /dev/online10_52
b38375d8 119 8 0 1048575 161060 PO- /dev/online5_67
b3837748 120 8 0 1048575 236157 PO- /dev/online10_53
b38378b8 121 8 0 1048575 136169 PO- /dev/online5_68
b3837a28 122 8 0 1048575 175589 PO- /dev/online10_54
b3837b98 123 8 0 1048575 702648 PO- /dev/online5_69
b3837d08 124 8 0 1048575 500054 PO- /dev/online10_55
b3837e78 125 8 0 1048575 714970 PO- /dev/online5_70
b3838018 126 9 0 1048575 69 PO- /dev/online10_56
b3838188 127 9 0 1048575 110 PO- /dev/online5_81
b38382f8 128 9 0 1048575 600 PO- /dev/online10_57
b3838468 129 9 0 1048575 69900 PO- /dev/online5_82
b38385d8 130 9 0 1048575 412226 PO- /dev/online10_58
b3838748 131 9 0 1048575 1048572 PO- /dev/online5_83
b38388b8 132 9 0 1048575 1048572 PO- /dev/online10_59
b3838a28 133 9 0 1048575 1048572 PO- /dev/online5_84
b3838b98 134 9 0 1048575 1048572 PO- /dev/online10_60
b3838d08 135 9 0 1048575 1048572 PO- /dev/online5_85
b3838e78 136 5 0 1048575 156880 PO- /dev/online10_61
b383d018 137 5 0 1048575 24286 PO- /dev/online10_62
b383d188 138 5 0 1048575 0 PO- /dev/online10_63
b383d2f8 139 5 0 1048575 825845 PO- /dev/online10_64
b383d468 140 5 0 1048575 1048572 PO- /dev/online10_65
b383d5d8 141 5 0 1048575 1048572 PO- /dev/online10_66
b383d748 142 5 0 1048575 1048572 PO- /dev/online10_67
b383d8b8 143 5 0 1048575 1048572 PO- /dev/online10_68
b383da28 144 5 0 1048575 1048572 PO- /dev/online10_69
b383db98 145 5 0 1048575 1048572 PO- /dev/online10_70
b383dd08 146 7 0 1048575 0 PO- /dev/online10_71
b383de78 147 7 0 1048575 0 PO- /dev/online10_72
b383f018 148 7 0 1048575 1048572 PO- /dev/online10_73
b383f188 149 7 0 1048575 1048572 PO- /dev/online10_74
b383f2f8 150 7 0 1048575 1048572 PO- /dev/online10_75
b383f468 151 7 0 1048575 1048572 PO- /dev/online10_76
b383f5d8 152 7 0 1048575 1048572 PO- /dev/online10_77
b383f748 153 7 0 1048575 1048572 PO- /dev/online10_78
b383f8b8 154 7 0 1048575 1048572 PO- /dev/online10_79
b383fa28 155 7 0 1048575 1048572 PO- /dev/online10_80
b383fb98 156 1 0 1048575 622508 PO- /dev/online10_81
b383fd08 157 1 0 1048575 1048572 PO- /dev/online10_82
b383fe78 158 8 0 1048575 0 PO- /dev/online10_86
b3841018 159 8 0 1048575 104858 PO- /dev/online5_71
b3841188 160 2 0 1048575 1048572 PO- /dev/online10_83
b38412f8 161 3 0 1048575 1048572 PO- /dev/online10_84
b3841468 162 8 0 1048575 896408 PO- /dev/online10_87
b38415d8 163 2 0 1048575 1048572 PO- /dev/online10_85
b3841748 164 8 0 1048575 211992 PO- /dev/online10_88
b38418b8 165 8 0 1048575 626259 PO- /dev/online5_72
b3841a28 166 3 0 1048575 1048572 PO- /dev/online10_89
b3841b98 167 8 0 1048575 384963 PO- /dev/online10_90
b3841d08 168 8 0 1048575 546775 PO- /dev/online5_73
b3841e78 169 8 0 1048575 872998 PO- /dev/online10_91
b3842018 170 2 0 1048575 1048572 PO- /dev/online10_92
b3842188 171 3 0 1048575 1048572 PO- /dev/online10_93
b38422f8 172 8 0 1048575 21790 PO- /dev/online10_94
b3842468 173 8 0 1048575 228969 PO- /dev/online10_95
b38425d8 174 8 0 1048575 1048572 PO- /dev/online5_74
b3842748 175 8 0 1048575 1048572 PO- /dev/online5_75
b38428b8 176 8 0 1048575 1048572 PO- /dev/online5_76
b3842a28 177 8 0 1048575 1048572 PO- /dev/online5_77
b3842b98 178 4 0 1048575 1048572 PO- /dev/online5_78
b3842d08 179 4 0 1048575 1048572 PO- /dev/online5_79
b3842e78 180 4 0 1048575 1048572 PO- /dev/online5_80
180 active, 2047 maximum
sending to informix-list
### Cut here ###
#!/bin/ksh
#
# qio - Query I/O, shows the top queries ordered by the amount of I/O
#
#
exec 2>&1
initdots()
{
for idx in x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
do
print -n "\b \b"
sleep 1
done
}
#
# getonstat - gets the onstat data and loads the 'CSessid', 'CNread' & CNwrite
# arrays
getonstat()
{
Data=""
idx=0
onstat -u | while read Line
do
if [ "$Line" ]
then
set $Line
else
continue
fi
[ "$1" = "address" ] && Data="TRUE" && continue
[ "$4" = "informix" ] && continue
[ "$2" = "active," ] && continue
Primary=${2#???}
Primary=${Primary%???}
[ "${Primary}" != "P" ] && continue
if [ "$Data" ]
then
#print "[$Line]"
#print "$3\t$9\t${10}"
CSessid[idx]=$3
CNread[idx]=$9
CNwrite[idx]=${10}
(( idx = idx + 1 ))
fi
done
(( TotalCurrent = idx -1 ))
}
#
# Loads the previous onstat into PSessid, PNread & PNwrite for comparission
#
assignprevious()
{
# Clean out excess array slots
while [ $TotalPrevious -gt $TotalCurrent ]
do
PSessid[$TotalPrevious]=""
PNread[$TotalPrevious]=""
PNwrite[$TotalPrevious]=""
(( TotalPrevious = TotalPrevious - 1 ))
done
idx=0
while [ $idx -le $TotalCurrent ]
do
PSessid[$idx]=${CSessid[$idx]}
PNread[$idx]=${CNread[$idx]}
PNwrite[$idx]=${CNwrite[$idx]}
(( idx = idx + 1 ))
done
((TotalPrevious=TotalCurrent))
}
#
# printcurrent - prints the values of the CSessid, CNread & CNwrite arrays.
#
printcurrent()
{
idx=0
while [ "${CSessid[idx]}" ]
do
#print "Current (${CSessid[idx]}|${CNread[idx]}|${CNwrite[idx]})"
print "[${CSessid[idx]}|${CNread[idx]}|${CNwrite[idx]}]"
(( idx = idx + 1 ))
done
print "Total: $TotalCurrent"
}
#
# printprevious - prints the values of the CSessid, CNread & CNwrite arrays.
#
printprevious()
{
idx=0
while [ "${PSessid[idx]}" ]
do
#print "Previous [${PSessid[idx]}|${PNread[idx]}|${PNwrite[idx]}]"
print "[${PSessid[idx]}|${PNread[idx]}|${PNwrite[idx]}]"
(( idx = idx + 1 ))
done
print "Total: $TotalPrevious"
}
#
# makedelta - find the entries with a difference and load delta
#
makedelta()
{
Cidx=0
Didx=0
while [ $Cidx -le $TotalCurrent ]
do
# Search the Previous array for a matchin CSessid
Pidx=0
while [ $Pidx -le $TotalPrevious ]
do
if [ ${PSessid[$Pidx]} = ${CSessid[Cidx]} ]
then
# Found a match, now check for a delta
Cio=$(( ${CNread[Cidx]} + ${CNwrite[Cidx]} ))
Pio=$(( ${PNread[Pidx]} + ${PNwrite[Pidx]} ))
if [ $Cio -gt $Pio ]
then
DSessid[$Didx]=${CSessid[$Cidx]}
(( Ddelta[$Didx] = $Cio - $Pio ))
(( Didx = Didx + 1 ))
fi
fi
(( Pidx = Pidx + 1 ))
done
(( Cidx = Cidx + 1 ))
done
(( TotalDelta = Didx - 1 ))
}
#
# print the delta out. This is the main program output
printdelta()
{
print -n "$sgr0$Thome"
print "$(date "+%D %T") Top queries by I/O (Delta > 100), Sample: $Pause seconds"
# Sort by delta descending
Didx=0
Rows=1
while [ $Didx -le $TotalDelta ]
do
print "${Ddelta[$Didx]}\t${DSessid[$Didx]}\t${Ddelta[$Didx]}"
#print "${Didx}: ${DSessid[$Didx]} | ${Ddelta[$Didx]}"
(( Didx = Didx + 1 ))
done | sort -nr | cut -d\ -f2,3 | while read Sessid Delta
do
[ $Delta -le 100 ] && continue
printf "%s\nSessid: %5d\tTotal I/O: %5d%s\n" $Teol $Sessid $Delta $Teol
(( Rows = Rows + 2 ))
priv onstat -g sql $Sessid > $WORK
if [ "$(grep "Current SQL statement" $WORK)" ]
then
sed -e '1,/Current SQL statement/d' \
-e '/Last parsed SQL statement/,$d' $WORK > ${WORK}.2
else
sed '1,/Last parsed SQL statement/d' $WORK > ${WORK}.2
fi
(( Rows = Rows + $(cat ${WORK}.2 | wc -l) ))
if [ $Rows -le $WinHeight ]
then
OIFS=$IFS
IFS="
"
cat ${WORK}.2 | \
while read InLine
do
printf "%s%s\n" $InLine $Teol
done
IFS=$OIFS
rm -f $WORK ${WORK}.2
else
rm -f $WORK ${WORK}.2
break
fi
done
print -n "EOR$Teod"
}
# main
export WORK=/usr/app/tmp/work.$PID
Thome=$(tput home)
sgr0=$(tput sgr0)
Tclear=$(tput clear)
Teol=$(tput el)
Teod=$(tput ed)
Pause=${1:-60}
WinHeight=$(stty -a | grep rows | cut -d\; -f2)
WinHeight=$(print $WinHeight)
WinHeight=${WinHeight% rows}
typeset -i WinHeight
if [ -z "$WinHeight" -o "$WinHeight" = "0" ]
then
WinHeight=20
else
((WinHeight=WinHeight-4))
fi
TotalPrevious=0
TotalCurrent=0
Pass=0
print -n $Tclear
print "$0: initilizing - $Pause seconds, please wait"
print -n "................................................. ..........."
while :
do
getonstat
((Pass = Pass + 1 ))
[ $Pass = 1 ] && assignprevious && initdots && continue
#printcurrent > Curr
#printprevious > Prev
makedelta
printdelta
assignprevious
#break
idx=1
while [ $idx -le $((Pause + 1 )) ]
do
print -n .
(( idx = idx + 1 ))
done
while [ $idx -gt 1 ]
do
print -n "\b \b"
sleep 1
(( idx = idx - 1 ))
done
done
sending to informix-list