How We Test Networked Storage Devices – Revision 5

Updated 5/13/15: Error corrections

What’s New

It’s hard to believe that five years (!) have passed since we last changed our NAS test process. But time marches on and so do improvements in NAS performance. The old Intel i3-based "V3/ V4" testbed actually had plenty of years left except for two things: the increasing number of NASes with 10GbE and Windows 8. So it’s time retire ol’ "V3 / V4" and bring on the SmallNetBuilder V5 NAS Testbed.

Although we are changing hardware, we’re not changing benchmarks, except to add 10GbE versions. Intel’s NAS Performance Toolkit (NASPT) will remain our primary benchmarking tool along with Windows robocopy to run Windows FileCopies of our standard ripped-DVD test folder.

Testbed Configuration

The new testbed system is an off-the-shelf Dell OptiPlex 9020 with a quad-core Intel Core i7-4770 3.40 GHz CPU running Windows 8.1 Pro. The Gigabit Ethernet port is a built-in Intel Ethernet Connection I217-LM and an Intel X540-T2 card has been added for dual 10GbE copper ports.

Disk performance doesn’t affect NASPT results, but it does affect robocopy file copies. Since the V5 testbed will be testing 10GbE NASes we don’t want disk performance standing in the way. After considering SSDs, we instead decided to make use of the Dell’s 8 GB of RAM and turn part of it into a ramdisk using SoftPerfect’s RAM Disk. (Thanks to Cinevate’s Dennis Wood for doing the research and suggesting SoftPerfect.)

The V5 NAS Testbed machine summarized in Table 1 is capable of testing NASes with file transfer performance up to the 1.25 GB/s limit of its 10 Gigabit Ethernet port.

Figures 1 and 2 show HD Tune write and read benchmark results for the RAM disk used only for Windows File Copy tests.

HD Tune write benchmark – Testbed RAM disk

Since the 10GbE interface tops out at 1250 MB/s, the RAM disk has more than enough performance for file copy tests.

HD Tune read benchmark – Testbed RAM disk

The Benchmarks

All benchmark tests are run with the NAS under test connected to the V5 Testbed via a TP-LINK TL-SG2216 10/100/1000 switch. A single Gigabit link is used for tests unless otherwise noted.

Each benchmark test is run three times and the average of the three runs is used. Tests are mixed to eliminate "warm cache" effects, i.e. each test is not run three times in a row.

For products supporting RAID 5 or higher, RAID 1 tests are not run.

For multi-drive NASes, the benchmarks without a RAID designation, (Write File Copy, for example), are run in RAID 0.

The benchmark tests include:

• Windows File Copy: This is the only test common to the previous NAS test suite, but is run using Windows 8 instead of Vista SP1.
  The tests are done using the robocopy command built into Windows. The benchmarks consist of Write and Read transfers of a single test folder containing a single non-compressed ripped DVD. The 4.35 GB (4,680,843,264 bytes) folder contains 38 files of various sizes ranging from 1 GB to 10 KB. Transfers are executed to and from the dedicated RAID 0 volume on the NAS Testbed, which is capable of 200+ MB/s transfer speeds.
• Intel NASPT File Copy To NAS: A single 1.15 GB Windows backup file is written to the NAS.
• Intel NASPT File Copy From NAS: A single 1.15 GB Windows backup file is read from the NAS.
• Intel NASPT Directory Copy To NAS: 236 MB consisting of 2,833 files in 44 folders is written to the NAS.
• Intel NASPT Directory Copy From NAS: 236 MB consisting of 2,833 files in 44 folders is read from the NAS.
• Intel NASPT Content Creation:
  Scripted routine simulating a user creating a web advertisement from various content sources including video and 3D modeling software. 95% writes with mix of 1k, 4k & little reads. Wide range of writes up to 64kB, mostly sequential.
• Intel NASPT Office Productivity:
  Common productivity applications: e-mail, spreadsheet, etc. running scripted routines.
  Balance of reads & writes, mostly 1k & 4k reads with a large number of smaller reads.
  Mostly 1kB writes, mostly sequential accesses.
• Intel NASPT HD Playback and Record: Simultaneous "playback" (read from NAS) and "record" (write to NAS) of approximately 500 MB 720p HD files. Both read and write are sequential.
• Intel NASPT 4x HD Playback: Simultanous "playback" (read from NAS) of four approximately 310 MB 720p HD files. Read is sequential.

Attached Backup

We test backup performance to an attached drive using built-in NAS backup features. The test uses a Startech USB 3.0 eSATA to SATA Hard Drive Docking Station (SATDOCKU3SEF) containing a WD Velociraptor WD3000HLFS 300 GB drive as the backup target.

The drive is attached via USB 2.0 and, if supported, eSATA and USB 3.0. Tests are performed with the drive formatted in FAT, EXT3 and NTFS, depending on the formats supported by the NAS under test.

The test copies the same 4.35 GB ripped DVD test folder used in the Windows File Copy tests from the NAS to the attached drive. Throughput is calculated using NAS-provided timestamps for the filecopy start and end.

Network Backup

For network backup, we use the same 4.35 GB ripped DVD test folder used in the Windows File Copy tests, which is first copied to the NAS under test.

The network backup target is a folder on the RAID 0 volume on the NAS test bed. Since Windows does not have a built-in rsync function, we run DeltaCopy server to handle rsync.

A network backup job, with no encryption or compression and a full copy of the folder is then executed. Throughput is calculated using NAS-provided timestamps for the filecopy start and end.

iSCSI Target

To test iSCSI performance, a 10 GB iSCSI Target is configured on the NAS under test, using RAID 1 for dual-drive NASes or RAID 5 for multi-drive NASes that support it. An iSCSI initiator using an unencrypted connection is then configured on the NAS Testbed machine and connected via Gigabit Ethernet without jumbo frames to the NAS under test. Our standard Windows File Copy test is then run for read and write to the iSCSI target.

Using the Charts

Results are imported into the NAS Chart Database. The default view shows the Windows File Copy benchmark, which is common to the current and previous test suites. The color of the bar indicates the version of test process used.

A cropped view of the NAS Chart in its default view is shown below. Other benchmarks are accessed using the Select benchmark dropdown.

NAS Chart default Windows File Copy view

Filters can be used to narrow the products shown. Filters are "sticky" and remain applied when you change benchmarks. So you may need to clear them if you get odd results.

Discontinued products are archived and hidden by default. Clicking on the Archived Products Hidden [Switch to show] link will display them. This control is "sticky" and will remain applied when you change benchmarks.

The Features button displays detailed product information including noise level, power consumption, processor, memory and more. Data for up to four products can be shown simultaneously in table format

The Benchmark Summary button produces a summary of all benchmark results for a single product selected via checkbox.

The Price vs. Performance button generates a plot of product price vs. the selected performance benchmark (figure below). Any filters applied in the Chart view are carried into this view. The screenshot below shows the File Copy Write Performance benchmark with results filtered for SoC-1 class and two bay products.

Note that the plot does not take into account NAS capacity so does not represent price / MB or GB. Mousing over a product icon will display the product name, model number, performance value and price.

Price vs. Performance – File Copy write

Other NAS Chart features are described here (scroll to the NAS Chart Changes section).

Questions and comments about the NAS Charts can be sent to [email protected].

Discuss this in the Forums