Mailing List
Home
Forum Home
Linux - General Red Hat Linux discussion list
Installation - Getting started with Red Hat Linux
Enterprise Linux 3 - Discussion of Red Hat Enterprise Linux 3 (Taroon)
Red Hat Linux 9 - Discussion of Red Hat Linux 9 (Shrike)
Red Hat Linux 7.2 - Discussion of Red Hat Linux 7.2 (Enigma)
Red Hat Linux 7.3 - Discussion of Red Hat Linux 7.3 (Valhalla)
Apache Web Server
Oracle database, Microsoft SQL server ...
Subjects
• application/x mplayer2 plugin
• RPM error: db4 error(16) from dbenv >remove: Device or resource
   busy
• Command stream end of file while reading
• X Windows problem (xauth)
• Upgrading openoffice 1 1 rpm
• FTP: connection refused
• FTP: connection refused
• mount: /dev/cdrom: is not a valid block device
• Dell Precision 650, RedHat 9, no sound
• how to trace the cause resulting in the crash of bind server
• Virus on the list
• UNINSTALL RPM MYSQL
• usb pen drives: mounting as a user
• broadcom network interface
• make mrproper
• sendmail configuration on redhat
• Couldn 't open PID file /var/run/named/named pid Permission denied
• Promise 378 controller
• kernel 2 6 and /dev/sound/mixer not found
• Problem using up2date
• mrtg step by step howto/configuration for a newbie?
• Compiling and Installing Kernel 2 6
• Can 't locate module ppp0, can 't locate module ppp compress 21
• HOW I CAN MAKE BOOTABLE FLOPPY DISKET
• Lotus Notes under Wine
• /etc/security/limits conf question
• Intel E/1000 driver
• Command stream end of file while reading
• rpm database corrupt
• qla2300 modules
 
High IO wait with little reason

High IO wait with little reason

2006-02-19       - By Wayne G

 Back
Reply:     1     2     3     4  

Its not easy to tell what's going on, but something that jumps out at
me, is that the non-powerpath disks (hda) have huge percent used and
service times. I assume the rest of the mess of disks is powerpath
related. You said those do not report correctly with iostat. But I
wonder why any disk activity whatsoever results in these high service
times and percent use for relatively small disk i/o. Your system is
relatively idle. The system to user cpu usage ratio is unacceptable.
Forgive me, I cannot remember the tools to use in an EMC environment
for checking the status of your disks. Start there and work your way
back to powerpath.

-wayne

On 2/10/06, Benji Spencer <ben.spencer@(protected)> wrote:
>  We have several machines which run various oracle databases (using OCFS)
> and have (today) noticed that some of them have an unusually high iowait
> (per top) when the system is mostly quiet.
>
>  The system does have several NFS mounts which connect to an EMC NAS head.
> Most of the disk (all except sda) are also Fiber attached SAN disk (EMC).
> The system also does have EMCs PowerPath installed and running (which iostat
> doesn't seem to report properly on).
>
>  Looking at the network, it generally is < 500 kilobits with spikes up to 1
> megabit (giglink).
>
>  What else can I/should I look at?
>
>  Here are some of the stat output I have:
>
>   12:04:08  up 8 days,  4:46,  1 user,  load average: 1.04, 1.09, 0.82
>  394 processes: 393 sleeping, 1 running, 0 zombie, 0 stopped
>  CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
>             total    1.5%    0.0%    0.4%   0.0%     0.0%   16.5%   81.4%
>             cpu00    1.5%    0.0%    0.1%   0.0%     0.0%   19.3%   78.8%
>             cpu01    1.7%    0.0%    0.1%   0.0%     0.1%   11.3%   86.4%
>             cpu02    0.7%    0.0%    0.9%   0.0%     0.0%   22.1%   76.0%
>             cpu03    1.9%    0.0%    0.3%   0.0%     0.0%   13.3%   84.2%
>  Mem:  7990576k av, 7171340k used,  819236k free,       0k shrd,  232896k
> buff
>                     3557796k actv, 1460396k in_d,  122272k in_c
>  Swap: 8385920k av,       0k used, 8385920k free                 4672284k
> cached
>
>  Looking at a iostat, I don't see anything sticking out really:
>
>  avg-cpu:  %user   %nice    %sys %iowait   %idle
>             0.00    0.00   15.25   18.50   66.25
>
>  Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await  svctm  %util
>  sdag         0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     0.90   50.00 450.00  90.00
>  sdah         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     4.00    0.00 1000.00 100.00
>  sdai         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdaj         0.00   0.00  5.00  2.00   68.00    3.00    34.00     1.50
> 10.14     3.92   11.43 142.86 100.00
>  sdaj1        0.00   0.00  1.00  1.00   32.00    2.00    16.00     1.00
> 17.00    22.04   20.00 500.00 100.00
>  sdaj2        0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00    21.00    0.00 1000.00 100.00
>  sdaj3        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdaj5        0.00   0.00  2.00  1.00   28.00    1.00    14.00     0.50
> 9.67    21.04   13.33 333.33 100.00
>  sdaj6        0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00    20.00    0.00 1000.00 100.00
>  sdak         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdq          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdr          0.00   0.00  3.00  1.00   76.00    1.00    38.00     0.50
> 19.25     3.67   15.00 250.00 100.00
>  sdr1         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00    21.00    0.00 1000.00 100.00
>  sdr2         0.00   0.00  0.00  0.00   32.00    0.00    16.00     0.00
> 0.00    19.27    0.00   0.00 100.00
>  sdr3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdr5         0.00   0.00  1.00  0.00    8.00    0.00     4.00     0.00
> 8.00    21.00    0.00 1000.00 100.00
>  sdr6         0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50    20.06   30.00 500.00 100.00
>  sds          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdt          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdu          0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     2.99    5.00 500.00 100.00
>  sdv          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdw          0.00   0.00  4.00  3.00  100.00   18.00    50.00     9.00
> 16.86     5.00    0.00 142.86 100.00
>  sdw1         0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50    21.00    0.00 500.00 100.00
>  sdw2         0.00   0.00  1.00  1.00   32.00   16.00    16.00     8.00
> 24.00    21.00    0.00 500.00 100.00
>  sdw3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdw5         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00    21.00    0.00 1000.00 100.00
>  sdw6         0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50    21.00    0.00 500.00 100.00
>  sdx          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdy          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdz          0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     1.00    0.00 1000.00 100.00
>  sdaa         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdaa1        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaa2        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaa3        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdaa5        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaa6        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdab         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     0.00    0.00   0.00   0.00
>  sdac         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdad         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdae         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     1.98   20.00 1000.00 100.00
>  sdaf         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdaf1        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaf2        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaf3        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdaf5        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdaf6        0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sda          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sda1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sda2         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sda3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sda4         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sda5         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdb          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdc          0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     8.00    0.00 1000.00 100.00
>  sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sde          0.00   0.00  4.00  3.00   44.00   18.00    22.00     9.00
> 8.86     3.00    0.00 142.86 100.00
>  sde1         0.00   0.00  1.00  1.00    4.00   16.00     2.00     8.00
> 10.00    22.00    0.00 500.00 100.00
>  sde2         0.00   0.00  1.00  1.00    4.00    1.00     2.00     0.50
> 2.50    21.00    0.00 500.00 100.00
>  sde3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sde5         0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50    21.00    0.00 500.00 100.00
>  sde6         0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00    20.00    0.00 1000.00 100.00
>  sdf          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdg          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdh          0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     4.00    0.00 500.00 100.00
>  sdi          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdi1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdi2         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdi3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdi5         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdi6         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdj          0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     0.00    0.00   0.00   0.00
>  sdk          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdl          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdm          0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     0.92   40.00 460.00  92.00
>  sdn          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00   0.00   0.00
>  sdn1         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdn2         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdn3         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    14.00    0.00   0.00 100.00
>  sdn5         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdn6         0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
> 0.00    21.00    0.00   0.00 100.00
>  sdo          0.00   0.00  1.00  0.00    4.00    0.00     2.00     0.00
> 4.00     0.00    0.00   0.00   0.00
>  sdp          0.00   0.00  1.00  1.00   32.00    1.00    16.00     0.50
> 16.50     1.96   20.00 500.00 100.00
>
>
>
>
>
> ---
> Ben Spencer
> ben.spencer@(protected) 312-329-2288
> --
> Taroon-list mailing list
> Taroon-list@(protected)
> https://www.redhat.com/mailman/listinfo/taroon-list
>
>

--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list