How We Test Networked Storage Devices – Revision 3

Introduction

Our NAS Tests have continued to evolve based on industry trends and reader requests. This revision of the NAS "How We Test" adds descriptions of the backup and iSCSI target tests that have been added starting in May 2009. There are also minor revisions to some of the tests.

Hardware Used

The NAS Testbed machine summarized in Table 1 is capable of testing NASes with file transfer performance in excess of 100 MB/s. It has these key differences from the original NAS Test platform:

• Faster CPU
  The Intel Core 2 Duo E7200 is overkill, but removes any worries about CPU-limitations
• More memory
  The 512 MB of RAM in the old testbed ensured that cache effects were eliminated when the 1 GB file size was tested. But most systems today run with at least 1 GB of RAM and, with Vista, more often 2 GB. So RAM was increased to better reflect today’s systems.
• PCIe Gigabit NIC
  The PCI-based NIC in the old testbed limited maximum performance test capability to approximately 67 MB/s. The PCIe NIC in the new testbed provides a network connection speed of 113 MB/s.
• Vista SP1 OS
• Separate OS drive
• Dual-drive RAID 0 data array

Table 1: NAS Test Machine

Write and Read Performance

From the inception of our NAS Performance Charts, Don Capps’ Iozone has been our primary NAS test tool. IOzone is a filesystem benchmark tool that generates and measures a variety of file operations. It’s been ported to many machines and runs under many operating systems, is free and even comes with source code!

Iozone also is OS and application neutral, i.e. it does not rely on or use OS-level optimizations, such as those implemented by Microsoft Vista for file copy operations. Finally, Iozone’s ability to easily test throughput over a range of file sizes makes it easy to obtain valuable insight into NAS performance.

The Iozone machine and NAS-under-test are connected via a Gigabit switch, which automatically handles jumbo frames up to 9K. We no longer test with a 100 Mbps connection, since the performance of all current NASes would be limited by using it.

The Iozone benchmarks are run using a 64 KB record size for consistency with previous data and because this record size is still widely used. However, to reflect the increased RAM size on both the NAS testbed and some high-end NASes, file sizes up to 4 GB are now tested.

The Iozone command line used will be similar to this:

iozone -Rab [results file name] -i 0 -i 1 -+u -f [path to directory on NAS device under test] -y 64k -q 64k -n 64k -g 4G -z

This creates a binary file format file of results in Excel compatible format, with the record and file sizes described above. CPU utilization of the IOzone test bed is also logged and checked to ensure that it is not a factor in the test results.

The same command line is run three times and either the best run or an average of best runs is used.

For products supporting RAID 5, RAID 1 tests are not run.

For multi-drive NASes, the benchmarks without a RAID designation, (1000 Mbps Write Performance, for example), are run in RAID 0.

File Copy Performance

We added a Vista SP1 filecopy benchmark with mixed feelings. One one hand, Vista will soon join the ranks of Windows Me as an OS that we’d all rather forget and be replaced by Windows 7.

But Vista SP1 really does deliver high filecopy performance (see How To Build a Really Fast NAS – Part 6: The Vista (SP1) Difference ). The addition of a filecopy benchmark should also help to satisfy those who find it difficult to understand the Iozone throughput vs. filesize plots and just want to know how fast their filecopy will be.

The tests are done using the robocopy command built into Vista (and available as a download as part of the free Windows Server 2003 Resource Kit Tools).

The benchmarks consist of Write and Read transfers of a single test folder containing a single non-compressed ripped DVD. The 4.35 GB (4,680,843,264 bytes) folder contains 38 files of various sizes ranging from 1 GB to 10 KB.

The separate RAID 0 array in the NAS testbed was specifically added to support this test and is capable of testing file copy speeds over 100 MB/s.

Backup

Starting in May 2009, we added backup performance testing. The test uses an Iomega UltraMax Pro Desktop Hard Drive configured in RAID 0 as the backup device. The drive is attached via USB 2.0 and, if supported, eSATA. Tests are performed with the drive formatted in FAT, EXT3 and NTFS, depending on the formats supported.

The test copies the same 4.35 GB ripped DVD test folder used in the File Copy tests from the NAS to the attached drive. Throughput is calculated using NAS-provided timestamps for the filecopy start and end.

iSCSI Target Performance

To test iSCSI performance, an iSCSI Target is configured on the NAS under test, using RAID 1 for dual-drive NASes or RAID 5 for multi-drive NASes that support it. An iSCSI initiator using an unencrypted connection is then configured on the NAS Testbed machine and connected via Gigabit Ethernet without jumbo frames to the NAS under test. The standard File Copy test is then run for read and write to the iSCSI target.

Using the Charts

Results are imported into the NAS Chart Database, where they are available in both average (Figure 1) and throughput vs. filesize forms.

Figure 1: Average Throughput Benchmark Chart

The Charts have many controls (Figure 2) that can be used to select the data displayed:

Figure 2: NAS Chart Controls

• Select Benchmark – This dropdown is used to choose the benchmark test data
• Filters – The Filters are used to reduce the number of products displayed based on product attributes. The filters are "sticky" and remain applied when you change benchmarks. So you may need to clear them if you get odd results.
• Filesize mode – By default, the charts display data for "large" filesizes between 32 MB and 4 GB. Switching to "small" filesize mode displays data for filesizes between 64 KB and 16 MB. Note that "small" filesize performance is usually higher than supported by the physical LAN connection because it represents cached performance, i.e read / write to memory vs. the actual NAS.
• Cached Performance Filter – A new averaging algorithm was introduced in March 2009 to remove data distortion in the average performance calculation caused by extremely high cached performance shown in some newer NASes. This article explains the filter.
• Features – This button displays detailed product information including noise level, power consumption, processor, memory and more. Data for up to four products can be shown simultaneously in table format
• Throughput vs. File Size – This button plots throughput vs. file size for up to six products (Figure 3). It is useful to see where cache effects stop and for further insight into performance.

Figure 3: Throughput vs. File Size Chart

• Benchmarks – This button lets you generate throughput vs. filesize comparisons for up to six different benchmark tests for a single product. It is useful for comparing performance of different RAID modes or with and without jumbo frames enabled.
• Price vs. Performance – This button generates a plot of product price vs. the selected performance benchmark. Note that it does not take into account NAS capacity so does not represent price / MB or GB.

Figure 4: Price vs. NAS 1000 Mbps Read Performance

Discuss this in the Forums