 | | | NFS Help! Terrible performance with sync, fast performance with async | NFS Help! Terrible performance with sync, fast performance with async 2006-11-19 - By Chris Wornell
Back
>A couple of things:
>
>1) Is this RHEL-3 (See http://HEL-3.ora-code.com)? The server and client in RHEL-3 (See http://HEL-3.ora-code.com) default to UDP
>packets which are the 'worst' for high bandwidth networks. You can
>find out what the client is mounting things as by looking in
>/proc/mounts. I forget exactly what needs to be done on the server
>side (RHEL-4 (See http://HEL-4.ora-code.com) supposedly has a better TCP server but I am not sure).
>Once you have both server and client on TCP... you should see an
>improvement.
>From my understanding, UDP should be fine in this scenario because there
is only one switch between the devices and no packet loss is occurring.
I ran my tests with tcp though (RHEL3 uses NFSv3 which does support TCP,
it just has to be explicitly enabled) and the results were no better.
>2) What is your switch set up as? A dumb switch should be ok, but some
>switches will try to do things like 'grouping' packets etc which can
>cause bad performance.
It's the cheapest Dell Powerconnect switch that can do jumbo frames
(i.e. sub $100 switch). I've run my tests over a GigE network and a
100Mbps network using a cheap D-Link switch and same results between the
two.
>3) Look at using iozone as a second test. It may be able to show where
>the problems are better.
I've seen iozone but I haven't tried it yet. Its just meta-data changes
seem to take much longer than they need to due to the commit process. I
should also mention that the tests for reading/writing sequential
problems run perfectly using sync. Its only file creation/deletion that
exhibit the extremely slow speeds.
Thanks,
Chris Wornell
Network Administrator, Information Technology
Peerless Systems Corporation
http://www.peerless.com
office: 310.727.5723
fax: 310.727.5715
mailto:cwornell@(protected)
-- --Original Message-- --
From: taroon-list-bounces@(protected)
[mailto:taroon-list-bounces@(protected)] On Behalf Of Stephen John
Smoogen
Sent: Sunday, November 19, 2006 9:01 AM
To: Discussion of Red Hat Enterprise Linux 3 (Taroon)
Subject: Re: NFS Help! Terrible performance with sync,fast performance
with async
On 11/19/06, Chris Wornell <CWornell@(protected)> wrote:
>
> I've got a problem that I've spent quite a bit of time on, though I'm
not an
> expert at NFS. In summary, operations that require meta-data changes
(such
> as file/directory creations/deletions), perform extremely slow over
sync,
> but over 10x faster using async.
>
> I have two systems, connected to a GigE switch using intel pro 1000
NICs
> (jumbo frames is currently not enabled on any of the points).
>
> The NFS server is a dual-core opteron system with 1GB of RAM and
3x300 SAS
> disk RAID-5 (See http://AID-5.ora-code.com) on a Perc5/i controller with 256MB battery backed cache
(write
> cache is enabled). The file system is ext3. I've configured nfsd to
spawn 32
> processes upon startup. I'm using defaults for export the nfs shares,
no
> changes to rsize or wsize.
>
> The NFS client is a dual Xeon with 4GB of RAM and a single 7200rpm
SATA
> disk. Both systems are running RHEL WS 3 Update 8 and kernel
> 2.4.21-47.0.1.ELsmp.
>
> For testing, I'm using bonnie++. The following are some sample test
results
> that sum up the problem:
>
> Test on NFS server directly (not NFS loopback)
> -Sequential File Creation: 2976
> -Sequential File Deletion: N/A
> -Random File Creation: 3077
> -Random File Deletion: 9922
>
> NFS test with sync enabled
> -Sequential File Creation: 39
> -Sequential File Deletion: 79
> -Random File Creation: 39
> -Random File Delection: 65
>
> NFS test with async enabled
> -Sequential File Creation: 575
> -Sequential File Deletion: 1718
> -Random File Creation: 543
> -Random File Deletion: 1228
>
> Based on the local performance of the NFS server, it does not appear
the IO
> setup is the culprit. My understanding of the sync operation is a
commit
> happens which means the NFS server doesn't reply back until the change
has
> actually been committed to stable storage. There is something
happening
> behind the scenes though which is causing a huge delay before the NFS
server
> replies back the commit was complete.
>
> This question is actually work related and I'm planning to put the
NFS
> server into production, but I'd rather not use async, even with a UPS
and
> dual PSU's on the server. With the newer nfs-utils, sync is the
default
> option as well so it seems like sync should perform relatively well.
>
> Another question is I don't quite understand how the data corruption
> happens if a power loss occurs on an NFS server using async. Even with
sync,
> data transferred over the wire maybe loss if the nfs server gets shut
down
> before that data is committed. Can anyone go into more detail on how
the
> data corruption happens?
>
> Thanks a bunch!
>
A couple of things:
1) Is this RHEL-3 (See http://HEL-3.ora-code.com)? The server and client in RHEL-3 (See http://HEL-3.ora-code.com) default to UDP
packets which are the 'worst' for high bandwidth networks. You can
find out what the client is mounting things as by looking in
/proc/mounts. I forget exactly what needs to be done on the server
side (RHEL-4 (See http://HEL-4.ora-code.com) supposedly has a better TCP server but I am not sure).
Once you have both server and client on TCP... you should see an
improvement.
2) What is your switch set up as? A dumb switch should be ok, but some
switches will try to do things like 'grouping' packets etc which can
cause bad performance.
3) Look at using iozone as a second test. It may be able to show where
the problems are better.
--
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"
--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list
--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list
|
|
|