This process applies to routers tested after 15 May 2016. The previous process can be found here.
Updated 1/29/17 – Removed UDP tests due to process error
It has been just over six years (!) since we last revised our wired router test process. Since then, many routers now routinely max out our simple throughput tests. So it’s time to work these products a lot harder.
The new process does away with the pair of computers and IxChariot, instead using QA Cafe’s NTA1000, and flagship CDRouter router test solution.
QA Cafe NTA1000
CDRouter is a Linux-based router test suite that includes over 4000 functional and performance tests. The video below provides a short overview of CDRouter.
CDRouter Overview
Setup
We test all routers by directly connecting WAN and LAN ports to the NTA1000’s Gigabit Ethernet ports. Note the same test setup is used for both wired-only and wireless routers, i.e. the wireless portion of wireless routers is not involved in this testing.
We do not test LAN – LAN, i.e. the router’s switch, performance. The switch chips used in today’s routers all support non-blocking wire-speed performance.
Router Test Setup
All routers are prepared for test as follows:
- Upgraded to latest firmware
- Reset to factory defaults
- QoS features disabled (if present)
- Remote administration disabled (if enabled)
- WAN type set to DHCP client
- DoS protection enabled (if present)
- LAN DHCP server range set to 192.168.1.10 – 30
- LAN DHCP server lease time set to 120 minutes (if available)
- All ALG passthroughs enabled
- Remote management (HTTP / HTTPS) disabled (if enabled, if available)
- Local HTTPS management enabled (if available)
- TCP port 21 forwarded to 192.168.1.200
- UDP port 988 forwarded to 192.168.1.200
- Triggered port forwarding (if available) set for Dialpad
- Trigger port = 7175
- Trigger type = UDP
- Opened ports = 51200-51201,51210
- Opened port type = TCP, UDP
- Triggered port timeout set to 60 seconds (if available)
Performance
We run these tests, selected from the CDRouter perf.tcl IPV4 performance test module:
- TCP download throughput test WAN to LAN
- TCP upload throughput test LAN to WAN
- TCP bidirectional throughput test
UDP download throughput test WAN to LANUDP upload throughput test LAN to WANUDP bidirectional throughput test
Each test is run with 10 simultaneous connections with each connection presenting a 95 Mbps load. Each test runs for 5 minutes. Note that CDRouter uses iperf to measure throughput.
For TCP throughput tests, we report throughput and retries. For UDP throughput tests, we report throughput and % packet loss. CDRouter reports up and download throughput separately for the bidirectional tests. The Router Charts bidirectional benchmarks present the sum of the separate results.
We have retired the Maximum Simultaneous Connection test. We now test whether 3,000 simultaneous TCP and UDP sessions are supported in separate tests. This new test is actually harder than the previous maximum connection test because it opens the connections over a longer period of time, then goes back and checks they are still open.
Functional Tests
CDRouter’s primary focus is router functional testing. We run 245 tests, checking DNS, DHCP, UPnP, Xbox, NAT, firewall and other functions. We also run nmap scans from LAN and WAN to check for open ports. You can download a PDF of the full test list to see exactly what we run. Note we start the test by throwing a series of DoS attacks at the router, giving it 30 seconds to recover from each one.
In the Router Charts, we report the results in the Router Functional benchmark as a % of tests passing. The list of failing tests with explanation is noted in the product review.