Benchmarks of the ClusterNFS

We are reading/writing 4GB of zeroes in 4KB chunks. We use 4GB because it's twice the RAM on the NFS server, a methodology suggested here.

Unless otherwise noted, NFS server is set to default options (wdelay, sync).

These benchmarks were repeated a couple of times to make sure subsequent results were consistent, but only representative data is posted here (that's why the syntax is inconsistent sometimes - you're getting the median data, not every single trial that was run). I tried to make sure the RAID was not in use when the benchmarks were being performed (i.e. no read/write activity, no one logged in, no strenuous processes running, no SGE jobs busying the network).

NB: make sure to umount and remount after each write before reading that file back in. Otherwise caching will skew results!

lorien (the NFS server)

OK... what do we have to work with?

any_machine:~# rpcinfo -p lorien|grep nfs
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs

Lovely... looks like the NFS server provides everything we want.

direct disk access

data path: read/written to RAID directly

The results for reading are a bit wobbly, here are 2 extreme cases:

[avu@lorien tmp]$ time dd if=/dev/zero of=/home/tmp/nfs-benchmark.1 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    5m27.601s
user    0m0.243s
sys     0m18.761s

# OR...

[avu@lorien tmp]$ time dd if=/dev/zero of=/home/tmp/nfs-benchmark.1 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    7m9.049s
user    0m0.214s
sys     0m21.652s

# results for writing are also wobbly, could fragmentation have anythingn to do with it?
# files written over NFS take longer to read than files written to RAID directly...
# here are 2 extreme cases:

[avu@lorien tmp]$ time dd if=/home/tmp/nfs-benchmark.0 of=/dev/null bs=4096 count=1048576
524287+1 records in
524287+1 records out

real    0m58.708s
user    0m0.093s
sys     0m4.642s

# OR...

[avu@lorien tmp]$ time dd if=/home/tmp/nfs-benchmark.1 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    0m32.790s
user    0m0.136s
sys     0m6.231s

direct disk access summary

NFS access via loopback mount

data path: read/written through loopback mount

These results are much more consistent than reads/writes from the RAID directly (less than +/- 30 seconds on write tests, +/- 3 secs on read tests).

Default options (NFS v3):

[root@lorien tmp]# mount -v -t nfs localhost:/home/ /mnt/nfs/
localhost:/home/ on /tmp/nfs-test type nfs (rw,addr=127.0.0.1)

# WRITING:
[root@lorien tmp]# time dd if=/dev/zero of=/nfs/tmp/nfs-benchmark.5 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    11m32.378s
user    0m0.170s
sys     0m9.513s

# unmount and remount...

# READING:
[root@lorien tmp]# time dd if=/nfs/tmp/nfs-benchmark.3 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    1m7.457s
user    0m0.145s
sys     0m7.438s

Now let's switch to NFS v4 (making sure to add fsid=0 to /etc/exports on the server first):

[root@lorien tmp]# mount -v -t nfs4 localhost:/ nfs-test/
localhost:/ on /tmp/nfs-test type nfs4 (rw,addr=127.0.0.1)

# WRITING:
[root@lorien tmp]# time dd if=/dev/zero of=/tmp/nfs-test/tmp/nfs-benchmark.2 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    11m34.297s
user    0m0.177s
sys     0m9.331s

# unmount and remount...

# READING:
[root@lorien tmp]# time dd if=/tmp/nfs-test/tmp/nfs-benchmark.5 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out

real    1m0.614s
user    0m0.154s
sys     0m7.041s

NFS access via loopback mount summary

Given the degree of variation from benchmark to benchmark, I conclude there is no difference in performance between NFS v3 and v4 using default options, but maybe they will begin to differ if tweaking.

some random cluster node (e.g. sinclair)

data path: through the single gigabit switch connecting random cluster node and NFS server

WRITE ME

xander (mouth of the tunnel)

data path: squeezed through an SSH connection that goes from xander through the 802.11g wireless LAN (54Mbps max) through the AirPort Express base station plugged into a gigabit switch going into 100Mbit building network, then through the Aether down to the colo center, then through their gigabit switched network coming out on garibaldi; the other end of the SSH connection releases the packets that go from garibaldi via gigabit switch to lorien (whew!)

tunnel params:

• ssh -C is used for compressing data sent through the VPN tunnel (gzip compression level 6, where 1 is "fastest" and 9 is "best")
  • TODO: compression doesn't seem to improve anything, so maybe we shouldn't use it
• blowfish cypher is used (supposedly faster than AES128... right?)

Clearly this is the most important benchmark as it actually captures the NFS performance reduction that is a result of the NFS server being down the street.

Let's try mounting at the mouth of the tunnel using the most basic options, some of which are defaults on some systems (NFS version 2, UDP, rsize = wsize = 1024, timeo = 7).

xander:~# mount -t nfs -o nfsvers=2,proto=udp lorien:/home /mnt/nfs/

xander:~# time dd if=/dev/zero of=/nfs/tmp/nfs-benchmark.0 bs=4096 count=1048576
File size limit exceeded

real    62m53.680s
user    0m0.280s
sys     0m3.030s

xander:~# ll /mnt/nfs/tmp/nfs-benchmark.0 
-rw-r--r--  1 4294967294 4294967294 2.0G Jan 18 11:17 /mnt/nfs/tmp/nfs-benchmark.0

So, it transferred only halfway (4294967294 is not the file size, it is the "nfsnobody" UID/GID resulting from root_squash).

Let's read a complete file back in.

xander:~# umount /mnt/nfs
xander:~# mount -t nfs -o nfsvers=2,proto=udp lorien:/home /mnt/nfs/

DO THE READING

Now let's kick it up to nfsvers=3:

xander:~# mount -t nfs -o nfsvers=3 lorien:/home /mnt/nfs/

avu@xander:~$ time dd if=/dev/zero of=/nfs/users/avu/nfs-benchmark.0 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes transferred in 1189.181316 seconds (3611701 bytes/sec)

real    19m49.236s
user    0m0.500s
sys     1m7.410s

xander:~# umount /mnt/nfs/
xander:~# mount -t nfs -o nfsvers=3 lorien:/home /mnt/nfs/

xander:~# time dd if=/nfs/users/avu/nfs-benchmark.0 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes transferred in 446.050649 seconds (9628878 bytes/sec)

real    7m26.070s
user    0m0.390s
sys     0m6.670s

Now let's kick it up to proto=tcp:

xander:~# mount -t nfs -o nfsvers=3,proto=tcp lorien:/home /mnt/nfs/ 

xander:~# time dd if=/dev/zero of=/nfs/tmp/nfs-benchmark.0 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes transferred in 1449.513382 seconds (2963041 bytes/sec)

real    24m9.552s
user    0m0.460s
sys     0m13.240s

xander:~# umount /mnt/nfs
xander:~# mount -t nfs -o nfsvers=3,proto=tcp lorien:/home /mnt/nfs/

xander:~# time dd if=/nfs/tmp/nfs-benchmark.0 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes transferred in 694.062235 seconds (6188159 bytes/sec)

real    11m34.073s
user    0m0.590s
sys     0m6.460s

Now let's try NFS v4:

xander:~# mount -t nfs -o nfsvers=4 lorien:/home /mnt/nfs/
NFSv4 not supported!  # whoops! so much for that

xander (mouth of the tunnel) summary

read (MB/s)	write (MB/s)	nfsvers	proto	rsize	wsize	timeo	ssh -C	other options
not supported		4	N/A
9.18	3.44	3	udp	1024	1024	7	yes	none
5.90	2.83	3	tcp	1024	1024	7	yes	none

-- Created by: AndrewUzilov on 16 Jan 2007