How We Test Networked Storage Devices – Revision 2

Background

Updated: December 29, 2008

From the inception of our NAS Performance Charts, Don Capps’ Iozone has been our primary NAS test tool. IOzone is a filesystem benchmark tool that generates and measures a variety of file operations. It’s been ported to many machines and runs under many operating systems, is free and even comes with source code!

For this update of our NAS test process, we investigated other tools. But we decided to continue to use Iozone in order to maintain consistency in testing as we transition to a new hardware platform.

Iozone also is OS and application neutral, i.e. it does not rely on or use OS-level optimizations, such as those implemented by Microsoft Vista for file copy operations. Finally, Iozone’s ability to easily test throughput over a range of file sizes makes it easy to obtain valuable insight into NAS performance.

The Pentium 4-based NAS testbed machine summarized in Table 1 has served us well. But the new series of small-business NASes like QNAP’s 5-bay TS-509 Pro and NETGEAR’s 6-drive ReadyNAS Pro have transfer rates beyond the testbed’s capability to test.

Table 1: Current NAS Test bed iozone computer

The new NAS Testbed machine is summarized in Table 2 and has the following key differences:

• Faster CPU
  The Intel Core 2 Duo E7200 is overkill, but removes any worries about CPU-limitations
• More memory
  The 512 MB of RAM in the old testbed ensured that cache effects were eliminated when the 1 GB file size was tested. But most systems today run with at least 1 GB of RAM and, with Vista, more often 2 GB. So RAM was increased to better reflect today’s systems.
• PCIe gigabit NIC
  The PCI-based NIC in the old testbed limited maximum performance test capability to approximately 67 MB/s. The PCIe NIC in the new testbed provides a network connection speed of 113 MB/s.
• Vista SP1 OS
• Separate OS drive
• Dual-drive RAID 0 data array

Table 2: NAS Test Machine

The last three bullets bear some explanation. The RAID 0 array is not needed by Iozone, which does not access the hard drive on the machine that it runs on when running the actual reads and writes to the target filesystem. It is being added, along with the Vista SP1 OS to support new read and write file copy benchmarks.

We are adding the Vista SP1 filecopy benchmark with mixed feelings. One one hand, Vista will soon join the ranks of Windows Me as an OS that we’d all rather forget. Microsoft has stopped actively marketing Vista and the drumbeat has started on its successor, Windows 7.

But the arguments for Vista (and SP1 specifically) are that it really does deliver higher filecopy performance (see How To Build a Really Fast NAS – Part 6: The Vista (SP1) Difference for the full story). And since it comes on most new PCs, the installed base is significant and growing. The addition of a filecopy benchmark should also help to satisfy those who find it difficult to understand the Iozone throughput vs. filesize plots and just want to know how fast their filecopy will be.

Setup

The Iozone machine and NAS-under-test are connected via a gigabit or 10/100 switch, as the benchmark dictates. The gigabit switch automatically handles jumbo frames up to 9K.

The Tests

The Iozone benchmarks will continue to be run using a 64 KB record size, for consistency and because this record size is still widely used. However, to reflect the increased RAM size on both the NAS testbed and some high-end NASes, file sizes up to 4 GB will now be tested.

The Iozone command line used will be similar to this:

iozone -Rab [results file name] -i 0 -i 1 -+u -f [path to directory on NAS device under test] -y 64k -q 64k -n 64k -g 4G -z

This creates a binary file format file of results in Excel compatible format, with the record and file sizes described above. CPU utilization of the IOzone test bed is also logged and checked to ensure that it is not a factor in the test results.

Results are imported into the NAS Chart Database, where they are available in both average (Figure 1) and throughput vs. filesize (Figure 2) forms.

Figure 1: Average Throughput Benchmark Chart

Figure 2: Throughput vs. File Size Chart

The new Vista SP1 file copy benchmarks are done using the robocopy command built into Vista (and available as a download as part of the free Windows Server 2003 Resource Kit Tools).

The benchmarks consist of Write and Read transfers of a single test folder containing a single non-compressed ripped DVD. The 4.35 GB (4,680,843,264 bytes) folder contains 38 files of various sizes ranging from 1 GB to 10 KB.

For both the Iozone and Vista SP1 file copy benchmarks, tests are run for each LAN connection speed (100 Mbps, 1000 Mbps and 1000 Mbps w/ 4K jumbo frames) and volume format (JBOD / RAID 0, RAID 1, RAID 5 and RAID 10) supported by the product.

NOTES:

– For products supporting RAID 5, RAID 1 tests are not run.

– For multi-drive NASes, the benchmarks without a RAID designation, (100 Mbps Write Performance, for example), are run in RAID 0.

Discuss this in the Forums