Breaking the 750 MB/s barrier
In parts 1 through 4 of this series, we discussed 10GbE networking basics, built up a software toolkit, reviewed 10GbE NAS performance, and took a close look at SMB3 networking advancements. In this penultimate installment, we’ll look over a plug and play 10GbE NAS solution and build up an Adobe CC editing workstation capable of exceeding 750 MB/s over 10GbE networks.
Before we get started, let’s look at how all of this can be integrated into your existing network. There have been several questions related to MacOS workstations and possible network configurations posed over at Cinevate’s blog. Here are two possible network configurations. The first is very similar to what we are using at Cinevate.
Switched 10GbE network example
The direct connect setup below would not require purchasing a 10GbE switch in cases where only two 10GbE connected workstations were required, and your NAS or server had a dual port 10GbE NIC installed.
Unswitched 10GbE network example
Buy a NAS, or build a server?
If you do decide to purchase a 10GbE NAS to share your video projects, you’ll still need a fast 10GbE equipped editing workstation. But you certainly won’t need to build your own server. In Part 3 of this series, I reviewed QNAP’s TS-470 PRO, but recommended an 8-bay (or more) NAS to fully take advantage of 10GbE network speeds.
For those with an existing equipment rack system, I’d suggest starting with the QNAP TS-EC879U-RP or TS-879U-RP, which have room for 2 optional PCIe cards. One card can be added for 10GbE (2 ports) and optionally, another to connect additional enclosures like the REXP-1200U-P. These drive enclosures allow you add much more storage without adding another NAS.
I had a chance to test out QNAP’s TS-870 Pro NAS fully populated with eight Hitachi 4 TB drives, which returned some very impressive performance numbers. This NAS performs extremely well and at a price that should make you stop and question the value in building your own server. If you are looking for warranty and support, QNAP has proven to be excellent in this department. Regardless of what brand you choose, make sure the NAS hardware is sufficient for your bandwidth needs.
QNAP TS-870 front
On the back view of the TS-870 Pro, you can see I’ve connected both 10GbE and 1GbE interfaces as per the wiring diagram above, integrating the NETGEAR 8 port 10GbE switch as well as our older 1GbE equipment. You can connect both ports to your switches and configure them for redundancy, or link aggregation in the NAS network configuration.
QNAP TS-870 rear
Multiple NAS network connections will not magically increase network performance for a single connection to the NAS, but can increase overall bandwidth to clients as network loads increase. This behavior potentially changes if the NAS is running Windows 2012 Storage Server, which can aggregate network ports transparently to increase single connection speeds.
Part 4 (SMB3) of the series covers this behavior in some detail. For a NAS running Linux based software (almost all of them), a single SMB3 connection to Windows 8.1 was limited in my testing to ~ 740 MB/s. You would likely see a lower number with MacOS / SMB2, or Windows 7 or earlier.
In the following tests, the TS-870 Pro NAS shows impressive ATTO results with 8 drives configured as a RAID 5 array and ATTO queue depth set to 4.
QNAP TS-870 Pro SMB3 vs. iSCSI performance
A typical Windows 8.1 large file copy/paste shows writes to the TS-870 PRO at ~ 550 MB/s and reads at ~ 740 MB/s fully loaded with hard drives and configured in RAID 5. Using Intel’s NASPT tool, the results of “real world” application traces are shown.
QNAP TS-870 Pro Intel NASPT results
In the case you are working exclusively on the MacOS platform and looking for a plug and play solution, a 10GbE NAS along with workstation 10GbE network cards from companies like Small Tree (who supports the Intel X540 cards) is all that would be required. If you’re looking to put your own Windows based 10GbE server or workstation together, read on.
What software should I use if I’m building my own workstation/server?
If you’ve read my previous 10GbE posts, you’ll know that for Adobe CC video/photo editing, 10GbE performance is best on either Windows 8.1, or Windows 2012 Server due to SMB3 multichannel features. In many tests between workstations, Windows 8.1 performed just as well as Server 2012 as far as large file network file transfers were concerned. Windows SMB3 multichannel will attempt to establish up to four threads per network interface (NIC) per session by default.
We also learned in previous posts that RSS (found in advanced NIC configuration) will try to spread this workload over available CPU cores. This is likely why a processor like the i7 4770K with four physical cores, showed better performance than the i5 processors (2 physical cores) I tested in 10GbE network tests.
A Windows 8.1 based NAS, with an i7 processor and fast RAM will perform very well as a 10GbE server, or workstation. Just remember that Windows 8.1 is limited to 20 SMB inbound connections if you plan on using it as a “server”.
Here are some pretty impressive numbers generated between two Windows 8.1 workstations, both sharing RAM disks over 10GbE connections. RAM disk software creates a virtual hard disk in workstation RAM, removing the testing bottleneck created by local SSD or hard disks.
Two Windows 8.1 workstations – Intel NASPT
PCIe SSD drives are the only single drives right now of sustaining these speeds, some in excess of 2000MB/s.
Two Windows 8.1 workstations – Windows filecopy
So how did we manage server builds to allow 900MB/s data transfer? After many builds, OS tweaks and tests, here are my official workstation and server build recommendations for the tech savvy DIY builder looking to edit video via a shared storage 10GbE solution.
WorkStation Build
The parts list for the workstation build is shown in Table 1. Note this does not include a Windows 8.1 license. This configuration could also be used as a less powerful server for small workgroups.
| Component | Description | Price |
|---|---|---|
| Case | Rosewill BlackHawk (an older Antec case is pictured below) |
$79 |
| Motherboard | Asus Z87-A or Z87-WS | $149 / $349 |
| CPU | Intel Core i7 4770K @ 3.4GHz | $359 |
| CPU Cooler | Noctua NH-U12S | $77 |
| Memory | GSKILL RipJaws X Series (4 x 8GB -> 32GB) | $520 |
| Video Card | Nvidia GTX 760 or GTX 780 | $250 / $520 |
| SSD | 500 GB Samsung Evo | $330 |
| 10GbE NIC | Intel X540-T1 | $388 |
| Power Supply | Corsair TX750 (750 W) | $89 |
| Total | $2241 – $2711 | |
Table 1: 10GbE Workstation
Being a fan of Asus for several decades now, there are several motherboards I would suggest looking at as the foundation for a performance Adobe CC editing computer. The Asus Z87-A ($150) has sufficient PCIe slots for basic applications. But a previous-generation Z77-V board performed quite well during 10GbE testing both as a server and workstation board.
You can see the Z77-V here configured with 32 GB of RAM, a RocketRaid RAID 2720 SGL card (supports 8 hard drives) and the Intel X540-T1 10GbE NIC.
Asus Z77-V motherboard
My own editing workstation shown below is based on the Asus Z87-WS motherboard, using two Nvidia GTX 650 Ti Boost cards. They are connected with an SLI cable (just in case you’re a gamer too). However, when using Adobe CC, you need to disable SLI using Nvidia’s driver interface. Adobe Premiere CC will find and use all the CUDA cores it can for rendering etc. Combining two cards, Adobe Premiere CC will see 1536 CUDA cores and 4GB video memory.
You can see the Intel X540 NIC nestled between the video cards, nicely cooled by the Nvidia video card’s cooling fan right beside it. If you compare this case arrangement with the Corsair 550D used later in our server build, you can see a large improvement in terms of cable management, and therefore airflow using a newer case like the 550D.
The Noctua NH-U12S CPU cooler is one of the most efficient units out there, and supported overclocking in this system to 4.3 GHz in Turbo mode with no issues. The i7 4770K processor presents 8 cores to the operating system, so even a small overclock can have significant impact on video rendering times. As a video editing machine may operate for hours during a render at 100% CPU loading, an efficient CPU cooler is important.
You can see I’ve spec’d RAM at 32 GB. This is more an advantage if you’re using Adobe CC After Effects, where RAM can be assigned to CPU cores in order to speed up rendering times. If you’re not doing much in After Effects, 8 or 16 GB of RAM might serve your needs just fine. You would see very little difference in the Adobe Premiere CC Benchmark tests.
10GbE editing workstation, Asus Z87-WS, i7 4770K CPU, 32GB RAM, dual Nvidia GTX650 Ti Boost, X540-T1 10GbE
That’s all for now. The next and last installment will provide a few more alternatives for the workstation build and spec out the high performance 10GbE server build.
Dennis Wood is Cinevate’s CEO, CTO, as well as Chief Cook and Bottle Washer. When not designing products, he’s likely napping quietly in the LAN closet.