We’ve revised our router test process with a new Simultaneous Connection test capable of measuring as many connections as routers will provide.
Updated 20 October 2010: Clarify wireless router testing
Updated 10 June 2010: Max Connection test rounding
This process applies to routers tested after 15 March 2010. The previous process can be found here.
We test all routers on a private network using dedicated computers connected to the WAN and LAN side of the router as shown in the figure below. Note that the same test setup is used for both wired-only and wireless routers, i.e. the wireless portion of wireless routers is not involved in the testing.
All routers are prepared for test as follows:
- Upgraded to latest firmware
- Reset to factory defaults
- QoS features disabled (if present)
- LAN client put into DMZ
- SPI disabled (optionally, if router allows)
The last two points require some explanation. The LAN client is put into DMZ because it is the fastest and easiest way to allow WAN to LAN traffic and test results from the WAN IxChariot endpoint to traverse the router under test’s NAT firewall. Alternatively, the specific ports used by IxChariot could be opened in the router under test’s firewall. Note that since the router DMZ function is just a special case of port mapping, i.e. all ports and protocols opened to the specified LAN IP address, this does not affect router performance.
SPI is disabled if the router provides this function as a work-around for problems encountered with some routers. Virtually all current routers use NAT with Stateful Packet Inspection (NAT+SPI) and the simple test configuration we use sometimes will not allow test results to be returned from the WAN endpoint.
The problem can be worked around by using a second network adapter on both endpoint machines connecting directly to the IxChariot console. But we have found that disabling SPI is simpler and in most cases does not affect router throughput.
Once the router is prepared, the following tests are performed:
Throughput
Throughput is a measure of how fast data flows through the router. The test sends data from computer to computer, measures how much time it takes, and calculates the result in Mbps (Megabits per second).
The setup shown in the diagram above is used with one computer running an IxChariot endpoint connected to the router WAN port and a second computer running the IxChariot console and endpoint connected to a router LAN port.
Our test uses IxChariot’s Throughput.scr IxChariot script with the defaut 100,000 Byte file size changed to 1,000,000 Bytes and TCP/IP protocol. Three tests are performed, each one minute in length:
- WAN to LAN – Data flows from the WAN-side IxChariot endpoint to the LAN-side endpoint. This is a test of router download speed.
- LAN to WAN – Data flows from the LAN-side IxChariot endpoint to the WAN-side endpoint. This is a test of router upload speed.
- Simultaneous – A combination of the previous two tests. This test is run to see how the router handles simultaneous traffic. Also, many current-generation routers have routing speeds in excess of 100 Mbps, but do not have Gigabit Ethernet ports. This test ensures that we find the limits of the routing engine without being limited by 100Mbps Ethernet.
Maximum Simultaneous Connections
This test measures the maximum number of connections or sessions that a router will concurrently support. A high number of connections is desired for peer-to-peer file sharing (BitTorrent, etc.) and on line gaming.
This test previously used IxChariot and was limited to a maximum of 200 connections due to IxChariot licensing restrictions. While this maximum used to be sufficient, most current generation routers have no problem handling more than 200 simultaneous sessions.
Our new method uses two Windows programs originally attributed to Matrix21. server.exe is installed on a WAN-side computer and client.exe is installed on the LAN-side machine. The router under test is first power cycled to clear any existing sessions.
The server program is started first via command line and listens for traffic from the client program. The client program is then started and sends a UDP packet to the server starting at a port number entered as part of the command line and waits for a response from the server.
If a response is received, the client increments the port number by one and sends a new packet. This process continues until the client does not receive a response from the server. The count of the last successful transfer is taken as the Maximum simultaneous session count.
The test is repeated three times, with the router power-cycled between each run and the highest number used.
Updated 10 June 2010
If it is obvious that the router under test has a limit on a binary multiple boundary, i.e. 2048, 4096, 8192, etc., then results will be rounded to that boundary.
You can download a zip of the programs and instructions from here if you’d like to run your own tests.
Tests Not Performed
Latency / Ping Time: (also known as lag, latency, or ping time) that the router introduces into a data stream, and is essentially what you’d measure by using the ping command.
We do not perform this test because all current products have response times below 1 mS, which is lower than the latency of most (if not all) Internet connections.
UDP Streaming: This test measures how well a router can keep up with a continuous stream of data. In addition to giving an indication of whether you’ll have trouble watching video program streams, it tends to show flaws in the router’s routing “engine”. It uses the connectionless UDP protocol, which has less overhead and fewer error recovery mechanisms than the TCP protocol (picture a fire hose being turned on vs. a water bucket brigade).
We currently do not perform UDP streaming testing on routers. However, we are investigating test scenarios to assess router performance for handling video streaming.
LAN Switch Performance: We do not perform any throughput testing on routers’ LAN switch ports. Performance is determined by the LAN switch chip and is not limited by a product’s routing code or CPU. There is no practical difference in performance among today’s 10/100 or 10/100/1000 switch chips.