Updated June 14, 2006
As of May 2006, we have moved to test routers using IxChariot. For routers tested before May 2006, we used this procedure.
The setup shown in the diagram below is used with one computer running an IxChariot endpoint connected to the router WAN port and a second computer running the IxChariot console and endpoint connected to a router LAN port.
All routers are prepared for test as follows:
- Upgraded to latest firmware
- Reset to factory defaults
- QoS features disabled (if present)
- LAN client put into DMZ
- SPI disabled
The last two points require some explanation. Because an IxChariot endpoint behaves like a server, it is necessary to, at minimum, open the ports that IxChariot uses through the router under test’s firewall. However, we usually just put the IxChariot LAN-side machine in DMZ to save time. Note that since the router DMZ function is just a special case of port mapping, this does not affect the router performance.
Virtually all current routers use NAT with Stateful Packet Inspection (SPI) and IxChariot usually has difficulty returning test results through NAT+SPI firewalls. So we typically also disable SPI if the router firewall settings allow this. Although early NAT+SPI routers showed significant performance differences when SPI was disabled, current designs have addressed this issue and disabling SPI does not significantly affect router throughput.
Once the router is prepared, the following tests are performed:
Throughput – Throughput is a measure of how fast data flows through the router. The test sends data from computer to computer, measures how much time it takes, and calculates the result in Mbps (Megabits per second).
Our test uses IxChariot’s Throughput.scr IxChariot script with the defaut 100,000 Byte file size changed to 1,000,000 Bytes. Three tests are performed, each one minute in length:
•WAN to LAN – Data flows from the WAN-side IxChariot endpoint to the LAN-side endpoint. This is a test of router download speed.
•LAN to WAN – Data flows from the LAN-side IxChariot endpoint to the WAN-side endpoint. This is a test of router upload speed.
•Simultaneous – A combination of the previous two tests. This test is run to see how the router handles simultaneous traffic. Also, many current-generation routers have routing speeds in excess of 100Mbps, but do not have gigabit Ethernet ports. This test ensures that we find the limits of the routing engine without being limited by 100Mbps Ethernet.
Maximum Simultaneous Connections – This test measures the maximum number of connections or sessions that a router will concurrently support. It uses the IxChariot standard throughput.scr script with TCP/IP with the file_size parameter (the size of the transferred file) changed from its default of 100,000 to 1,000,000 Bytes.
Because IxChariot has known problems with NAT + SPI routers, the LAN side machine is put in DMZ and router SPI features are disabled if possible. One minute tests are then run using 2, 32, 64, 96, 128, 160 and 180 connections, evenly divided in each case between up and downlink. Failed tests are retried twice. In some cases, additional tests will be run to more accurately determine the failure point, which will result in values other than those listed above.
Response Time – This test measures the delay (also known as lag, latency, or ping time) that the router introduces into a data stream, and is essentially what you’d measure by using the ping command.
We no longer perform this test because all current products have response times below 1mS, the limit that can be measured by IxChariot or other common tools.
UDP Stream – This test measures how well a router can keep up with a continuous stream of data. In addition to giving an indication of whether you’ll have trouble listening to Internet audio or watching video program streams, it tends to show flaws in the router’s routing “engine”. It uses the connectionless UDP protocol, which has less overhead and error recovery mechanisms than the TCP protocol (picture a fire hose being turned on vs. a water bucket brigade).
We are currently in the process of re-evaluating our streaming test methods to more accurately reflect the capabilities of current designs and the requirements of video streaming and IPTV.