As noted at the end of How To Find The Best Router For Gigabit Internet, I’ve been working on a new router test process. The new process will refocus on router performance instead of the functional testing using CDRouter in the Version 4 process. The new process was inspired by Jim Salter’s benchmarking using ApacheBench that was part of his build-you-own router series over on Ars.
I’ll go into more detail when I write up the How We Test article for the new router benchmarking process. But in a nutshell, the process uses four different-sized files, downloaded via HTTP at five levels of concurrency. The final process will likely use a subset of the 20 benchmarks. But this preview will provide a chance for you (and product makers) to weigh in on the process before it is locked down.
Setup
The new router test bench is shown below. Two Intel quad-core i5 class Dell small form-factor machines running Ubuntu 16.0.4 LTS are used for the nginx web server and ApacheBench test client. Both are loaded with 16 GB of RAM, so hard drive performance will not put the brakes on throughput. The gigabit switch is there for convenience so the router under test can grab an IP address from the SNB house LAN.
SNB V10 Router Benchmark Setup
I decided to run this process with the router under test in its default settings, with one exception that I’ll describe shortly.
Baseline
I spent a few weeks, on and off, just characterizing the two machines connected to a switch. The results are shown in the first plot below. The plot shows throughput reported by ApacheBench vs. number of concurrent HTTP connections for four image sizes using a 60 second test. I used actual JPG and PNG images. Surprisingly, the number of concurrent connections didn’t seem to matter in most cases. Only the smallest 2 KB image showed any significant throughput variation with increasing concurrent connections.
Throughput baseline
I also plotted number of HTTP requests. I did this to check whether ApacheBench failures significantly affected throughput or number of requests. I found they don’t, but decided to keep the benchmark, for reasons you’ll see shortly. Again, only the 2 KB file shows variation with number of concurrent connections.
Request baseline
Ubiquiti ERL
Ubiquiti’s sub-$100 EdgeRouter Lite-3 gets many recommendations for folks looking for a router capable of handling a gigabit internet connection. It uses a dual-core Cavium CN5020 @ 500 MHz. How To Find The Best Router For Gigabit Internet showed this is good recommendation as long as QoS features are not used. Enabling Qos disables Cut-through forwarding, which is responsible for gigabit wire-speed performance on the ERL as well as most other current-generation consumer routers.
Instead of plotting throughput, I instead plotted % of baseline throughput and requests. This makes it easier to see at a glance how a product compares to baseline through-the-switch performance. The ERL does quite well for the larger file sizes, but not so much for the smaller.
The % of requests plot pretty much tracks the throughput plot. The rise above 100% is due to the difference in relatively small number of requests for the large file sizes and, I think, variation due to the increase in errors that usually come along with the largest (and sometimes smallest) filesize tests.
Ubiquiti EdgeRouter Lite – Download
I wasted spent way too much time trying to get an upload test working. Long story short, ApacheBench isn’t really that great for upload testing and I abandoned the effort. But it turned out I just needed to put the nginx server on the LAN side, open port 80 to it, put the ApacheBench client on the WAN and run the same tests. Voila! Upload (LAN to WAN) tests!
Since the data flow direction is the same (server to AB client) for uplink, the same results are used to calculate the % of baseline results for both download (WAN to LAN) and upload (LAN to WAN). Upload performance is very similar to download for the ERL.
Ubiquiti EdgeRouter Lite – Upload
eero
The new crop of distrbuted Wi-Fi system products generally have basic routing feature sets. But what about performance? I ran the new benchmark tests on eero, a typical three-pack DWS design that uses a dual-core Qualcomm IPQ8062 @ 1 GHz. Download results show eero can move the largest 759 KB PNG file at gigabit wire-speed. But reducing file size (and increasing HTTP request rates) significantly reduces throughput.
eero – Download
The 759 KB line in the upload plots looks pretty wonky. That’s because only one of the five tests actually ran. The other four tests aborted after 10 failures. ApacheBench doesn’t provide diagnostics beyond saying the tests aborted. And checking the server log didn’t turn up anything in its error logs, either. I reran the tests a few more times and in each case got aborted uploads for some or all the tests. Suffice it to say, eero doesn’t like uploading large files concurrently, at least not like ApacheBench wants to see them.
eero – Upload
ASUS RT-AC88U
For the last experiment, I tested a typical, expensive big ol’ router. ASUS’ RT-AC88U uses a dual-core Broadcom BCM4709C0KFEBG @ 1.4 GHz, but doesn’t perform as well as Ubiquiti’s ERL. Where the ERL was able to move both the larger 759 KB and 108 KB files at wire-speed, ASUS’ throughput dropped down for the 108 KB file and also declined with increased concurrent downloads.
On the other hand, 10 KB file transfers were about the same for both products, except for the ASUS’ falloff as concurrent sessions increased. The only advantage the RT-AC88U had over the ERL is for the smallest 2 KB file size and lower number of concurrent connections. But the advantage disappears as concurrent connections increase.
ASUS RT-AC88U – Download
The ASUS’ upload performance is essentially the same as download.
ASUS RT-AC88U – Upload
Closing Thoughts
So it looks like this process, plus checking for throughput reduction using the techniques described in How To Find The Best Router For Gigabit Internet, will result in a more robust test of wired router performance. The new process will still have the weakness of not checking performance with encrypted WAN connection types, unfortunately. But I think it will provide a more effective tool for selecting routers suitable for faster internet connections. Let me know what you think.