This process applies to routers tested after May 15 2017. The previous process can be found here.
Updated 8 June 2017 – Changed Bufferbloat metric.
The V4 Router test process didn’t have quite the impact we hoped. Readers remained more focused on performance testing and showed little interest in the functional testing the CDRouter platform brought to bear. So with our thanks to QA Cafe, we’re retiring CDRouter.
With most of today’s routers using some form of cut-through forwarding (CTF) aka NAT acceleration, simple single stream-based throughput tests tend to show essentially gigabit wire-speed. With TCP/IP overhead, this is around 941 Mbps. Not surprisingly, most of the routers tested with the previous process showed unidirectional up and downlink throughput over 900 Mbps. In real world use, however, many buyers experienced throughput less than half the benchmarked result.
The all new process, which is being dubbed Revision 10 to line up with the new wireless test process, attempts to turn up the heat on routers with three new tests:
- HTTP Score
- Bufferbloat delay
- Cut Through Forwarding (CTF) Score
We are also including simple WAN to LAN and LAN to WAN throughput tests for quick reference.
Each test will be detailed in the next sections.
NOTE: We do not test LAN – LAN, i.e. the router’s switch, performance. The switch devices used in today’s routers all support non-blocking gigabit wire-speed performance.
NOTE: The same tests are used for both wired-only and wireless routers. The wireless portion of wireless routers is tested separately.
Routing Throughput
A 30 second iperf3 test using TCP/IP is run to get a simple measure of router throughput performance. Published benchmarks are:
- WAN to LAN Throughput – TCP
- LAN to WAN Throughput – TCP
Results are expressed in Mbps.
HTTP Score
Our new process was inspired by Jim Salter’s benchmarking using ApacheBench that was part of his build-you-own router series over on Ars. The SNB setup is essentially the same as Jim’s, except our tests use different benchmark metrics more suited to comparison using our charting tools. We also test upload (LAN to WAN) performance in addition to download and have simplified the test to use only 2048 concurrent connections.
The web server machine is a Dell Optiplex 9010 Small Form Factor with quad-core Core i5 3570 CPU @ 3.4 GHz, loaded with 16 GB of RAM. It runs Ubuntu 16.0.4 LTS with nginx 1.10.0.
The ApacheBench machine is a Dell Optiplex 790 Small Form Factor with quad-core Core i5-2400 @ 3.1 GHz, also loaded with 16 GB of RAM. It also runs Ubuntu 16.0.4 LTS and has ApacheBench 2.3 installed
A TP-LINK TG-3468 PCIe Gigabit Ethernet adapter (Realtek RTL8168B based) is installed in both systems and used for test traffic.
V10 Router Test Process configuration
WAN to LAN (download) performance is tested using a shell script calling a series of ApacheBench commands to download graphic image files of sizes 2 KB, 10 KB, 108 KB and 759 KB using 2048 conncurrent connections. Each test is run for 60 seconds.
LAN to WAN (upload) performance is tested by swapping the webserver and ApacheBench client connections between WAN and LAN. Port 80 on the router under test is opened to the IP address of the nginx webserver and the ApacheBench client is pointed at the router under test’s WAN IP address. The same tests are run again in this direction.
The plots below show baseline performance of the test configuration with web server and ApacheBench client directly connected to each other (no switch). The plot is the average of three test runs.
Gigabit Ethernet LAN Reference throughput
Published benchmarks are:
- HTTP Score – WAN to LAN
- HTTP Score – LAN to WAN
Scores are expressed as percent of routed throughput vs. throughput of client and server directly connected. Maximum score of 100% means routed throughput was equal to direct-connect throughput. Each benchmark is the average of the scores with 2 KB, 10 KB, 108 KB and 759 KB file sizes. You can also plot the values for each file size and see maximum, minimum or individual values. When plotting, the key is: [A] 2 KB, [B] 10 KB, [C] 108 KB and [D] 759 KB.
Bufferbloat Score
Updated 8 June 2017
As Bufferbloat.net’s introduction states: "Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data." Bufferbloat is the bane of gamers, but can also affect voice and video calling and any other real-time application.
Our test is run on a private network with the same two machines directly connected to the router WAN and LAN ports. We use the betterspeedtest.sh script to test bufferbloat. It basically saturates router bandwidth using netperf to generate traffic while simultaneously running a ping test. It measures upload and download bloat separately. The test is run for 3 minutes in each direction.
Published benchmarks are:
- Bufferbloat Score – Average Downlink
- Bufferbloat Score – Maximum Downlink
- Bufferbloat Score – Average Uplink
- Bufferbloat Score – Maximum Uplink
The benchmark values published are Bufferbloat Score = (1 / Measured delay from betterspeedtest.sh script output in ms) x 1000, with higher values being better. This makes smaller measured delays into larger numbers so that they can be properly ranked. To convert back milliseconds, calculate 1 / (score / 1000).
Cut Through Forwarding (CTF) Score
This test attempts to measure performance with CTF disabled or bypassed. Throughput without CTF enabled is typically reduced by 50% or more. Not all routers provide CTF controls and instead automatically disable CTF when certain features are used. So this test uses multiple methods to find reduced performance.
The test setup is the same as used for the HTTP score test, except iperf3 is used to generate test traffic. The iperf3 server is run on the router LAN side and the iperf3 client on the WAN side. This configuration requires forwarding TCP port 5201 through the router under test’s firewall. So all tests are run through a forwarded port.
30 second LAN to WAN and WAN to LAN tests are run for each change to the router under test listed below. Note all features listed may not be supported on all routers.
- NAT Boost / Acceleration disabled – If a CTF disable is provided, we try that first.
- QoS enabled – Up and downlink bandwidth are set to 1,000 Mbps. Automatic, adaptive or "smart" QoS is used. If the QoS GUI allows, the test client is set as the prioritized client.
- Traffic metering enabled
- Parental Control enabled – This is generally some form of category or keyword blocking
The benchmark value published is worst case CTF Score (%) = (Throughput with Enabled (Disabled) Feature / Baseline Throughput) x 100. Baseline throughput measurements are the values from the iperf3 Routing Throughput tests.
Scores are expressed as percent of routed throughput vs. throughput of client and server directly connected. Maximum score of 100% means routed throughput was equal to direct-connect throughput.