 | | | NFS Help! Terrible performance with sync, fast performance with async | NFS Help! Terrible performance with sync, fast performance with async 2006-11-19 - By Chris Wornell
Back
I've got a problem that I've spent quite a bit of time on, though I'm
not an expert at NFS. In summary, operations that require meta-data
changes (such as file/directory creations/deletions), perform extremely
slow over sync, but over 10x faster using async.
I have two systems, connected to a GigE switch using intel pro 1000 NICs
(jumbo frames is currently not enabled on any of the points).
The NFS server is a dual-core opteron system with 1GB of RAM and 3x300
SAS disk RAID-5 (See http://AID-5.ora-code.com) on a Perc5/i controller with 256MB battery backed cache
(write cache is enabled). The file system is ext3. I've configured nfsd
to spawn 32 processes upon startup. I'm using defaults for export the
nfs shares, no changes to rsize or wsize.
The NFS client is a dual Xeon with 4GB of RAM and a single 7200rpm SATA
disk. Both systems are running RHEL WS 3 Update 8 and kernel
2.4.21-47.0.1.ELsmp.
For testing, I'm using bonnie++. The following are some sample test
results that sum up the problem:
Test on NFS server directly (not NFS loopback)
-Sequential File Creation: 2976
-Sequential File Deletion: N/A
-Random File Creation: 3077
-Random File Deletion: 9922
NFS test with sync enabled
-Sequential File Creation: 39
-Sequential File Deletion: 79
-Random File Creation: 39
-Random File Delection: 65
NFS test with async enabled
-Sequential File Creation: 575
-Sequential File Deletion: 1718
-Random File Creation: 543
-Random File Deletion: 1228
Based on the local performance of the NFS server, it does not appear the
IO setup is the culprit. My understanding of the sync operation is a
commit happens which means the NFS server doesn't reply back until the
change has actually been committed to stable storage. There is something
happening behind the scenes though which is causing a huge delay before
the NFS server replies back the commit was complete.
This question is actually work related and I'm planning to put the NFS
server into production, but I'd rather not use async, even with a UPS
and dual PSU's on the server. With the newer nfs-utils, sync is the
default option as well so it seems like sync should perform relatively
well.
Another question is I don't quite understand how the data corruption
happens if a power loss occurs on an NFS server using async. Even with
sync, data transferred over the wire maybe loss if the nfs server gets
shut down before that data is committed. Can anyone go into more detail
on how the data corruption happens?
Thanks a bunch!
Thanks,
Chris Wornell
Network Administrator, Information Technology
Peerless Systems Corporation
http://www.peerless.com <http://www.peerless.com/>
office: 310.727.5723
fax: 310.727.5715
mailto:cwornell@(protected) <mailto:cwornell@(protected)>
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas
-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@(protected) Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>I've got a problem that I've spent quite a bit of time on, though I'm
not an expert at NFS. In summary, operations that require meta-data changes
(such as file/directory creations/deletions), perform extremely slow over sync,
but over 10x faster using async.<br>
<br>
I have two systems, connected to a GigE switch using intel pro 1000 NICs (jumbo
frames is currently not enabled on any of the points). <br>
<br>
The NFS server is a dual-core opteron system with 1GB of RAM and 3x300 SAS disk
RAID-5 (See http://AID-5.ora-code.com) on a Perc5/i controller with 256MB battery backed cache (write cache is
enabled). The file system is ext3. I've configured nfsd to spawn 32 processes
upon startup. I'm using defaults for export the nfs shares, no changes to rsize
or wsize.<br>
<br>
The NFS client is a dual Xeon with 4GB of RAM and a single 7200rpm SATA disk.
Both systems are running RHEL WS 3 Update 8 and kernel 2.4.21-47.0.1.ELsmp. <br>
<br>
For testing, I'm using bonnie++. The following are some sample test results
that sum up the problem:<br>
<br>
Test on NFS server directly (not NFS loopback)<br>
-Sequential File Creation: 2976<br>
-Sequential File Deletion: N/A<br>
-Random File Creation: 3077<br>
-Random File Deletion: 9922<br>
<br>
NFS test with sync enabled<br>
-Sequential File Creation: 39<br>
-Sequential File Deletion: 79<br>
-Random File Creation: 39<br>
-Random File Delection: 65<br>
<br>
NFS test with async enabled<br>
-Sequential File Creation: 575<br>
-Sequential File Deletion: 1718<br>
-Random File Creation: 543<br>
-Random File Deletion: 1228<br>
<br>
Based on the local performance of the NFS server, it does not appear the IO
setup is the culprit. My understanding of the sync operation is a commit
happens which means the NFS server doesn't reply back until the change has
actually been committed to stable storage. There is something happening behind
the scenes though which is causing a huge delay before the NFS server replies
back the commit was complete.<br>
<br>
This question is actually work related and I'm planning to put the NFS server
into production, but I'd rather not use async, even with a UPS and dual PSU's
on the server. With the newer nfs-utils, sync is the default option as well so
it seems like sync should perform relatively well.<br>
<br>
Another question is I don't quite understand how the data corruption happens if
a power loss occurs on an NFS server using async. Even with sync, data
transferred over the wire maybe loss if the nfs server gets shut down before
that data is committed. Can anyone go into more detail on how the data
corruption happens?<br>
<br>
Thanks a bunch!</span></font><font size=2 face=Arial><span style='font-size:
10.0pt;font-family:Arial'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Times New Roman"><span style='font-size:
10.0pt'>Thanks,<br>
<br>
Chris Wornell<br>
Network Administrator, Information Technology<br>
Peerless Systems Corporation<br>
</span></font><a href="http://www.peerless.com/"
title="http://www.peerless.com/"><font size=2><span style='font-size:10.0pt'
>http://www.peerless.com</span></font></a><br>
<font size=2><span style='font-size:10.0pt'>office: 310.727.5723<br>
fax: 310.727.5715<br>
</span></font><a href="mailto:cwornell@(protected)"
title="mailto:cwornell@(protected)"><font size=2><span style='font-size:10.0pt
'>mailto:cwornell@(protected)</span></font></a>
<o:p></o:p></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>
--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list
|
|
|