Introduction
The recent introduction of our Hardware Router Charts has generally met with reader approval. However, not everyone was happy with the performance of at least one the top-rated routers. The short story is that the D-Link DGL-4300 seemed to severly limit upload throughput when used with the eMule P2P file sharing application.
The jury is still out as to whether the problem is in the router itself, or due to ISP bandwidth throttling. But the discussion raised a valid question about our current router test methodology. The current method uses a combination of Ixia’s Qcheck and IxChariot applications, and tests upload (LAN-WAN) and download (WAN-LAN) throughput, response time and UDP streaming performance separately using two computers on a private LAN.
The P2P Contenders
While this test accurately tests the best-case performance of a router, it has its weaknesses. The most basic weakness is that it doesn’t reflect how a router responds to the simultaneous up and download traffic that routers see in real life. And the 500 kbps rate used in our UDP Streaming test doesn’t reflect the performance requirements of IPTV and other video streaming applications that are becoming increasingly important consumer broadband applications.
But more to the point of this article is that our current methods don’t test whether the products can handle the dozens of simultaneous up and download connections used by gaming and peer-to-peer file sharing applications. So I set out to both devise a test to gauge P2P performance and see how a collection of routers from the top of our chart would hold up under it.
The Products
I selected seven routers from among the top 10 fastest routers on the latest updated router chart (April 2006). The routers chosen all happened to be wireless, mainly because that’s what has been coming in for test most recently.
Belkin’s Wireless Pre-N (F5D8230-4) router was one of the first MIMO routers and the original model I had for test is based on a Broadcom BCM4704 processor. It’s essentially the same as the Linksys WRT54GX, which was also among the routers tested. The 10/100 switching capability is provided via a Broadcom BCM5325 and the WAN port Ethernet is handled by an Altima (Broadcom) AC101L 10/100 Ethernet Tranceiver.
Belkin F5D8230-4 Wireless Pre-N Router
D-Link is represented by two models: the DI-634M and DGL-4300. Both are based on Ubicom’s IP3023 Wireless Network processor, which as you’ll see shortly is pretty powerful. The DI-634M’s 10/100 switch chip is a Realtek RTL8305SC, while the DGL-4300’s gigabit switching features come courtesy of a Broadcom BCM5385.
D-Link DI-634M Wireless 108G MIMO Router
D-Link DGL-4300 Wireless 108G MIMO Router
The Products, Continued
As I mentioned earlier, the Linksys WRT54GX is an internal physical identical twin to the Belkin Pre-N.
Linksys WRT54GX Wireless-G / SRX Broadband Router
The two NETGEAR RangeMax products in the test are very different internally. The original WPN824 RangeMax is based on an Atheros AR2313 Wireless System on a Chip that includes a 32-bit MIPS R4000-class processor and multi-protocol MAC/baseband processing engine. 10/100 switching is handled via a Marvell Link Street 88E6060/88E6060-I 6-Port Fast Ethernet switch.
The newer WPNT834 RangeMax 240 is a different beast, using a Realtek RTL8651B Advanced Home Gateway Controller, which also contains a 6 port 10/100 switch.
NETGEAR WPN824 RangeMax Wireless Router
NETGEAR WPNT834 RangeMax 240 Wireless Router
Like the D-Links above, the ZyXEL X-550 is also based on Ubicom’s IP3023. The design closely resembles that of the DI-634M since it also uses a Realtek RTL8305SC for the 10/100 switch.
ZyXEL X-550 XtremeMIMO Broadband Router
Internal Construction Summary & Photos
Table 1 summarizes the information from the previous pages and adds Flash and RAM capacities. The OpenWrt Wiki’s TableOfHardware was a helpful resource in compiling this information.
| Product | Firmware Version Tested |
Processor | Flash | RAM | Switch |
|---|---|---|---|---|---|
| Belkin’s Wireless Pre-N | 1.01.03 | Broadcom BCM4704 |
4 MB | 16 MB | Broadcom BCM5325 |
| D-Link DI-634M | 2.01 | Ubicom IP3023 |
4 MB | 16 MB | Realtek RTL8305SC |
| D-Link DGL-4300 | 1.6 | Ubicom IP3023 |
4 MB | 16 MB | Broadcom BCM5385 |
| Linksys WRT54GX | 1.02.10 | Broadcom BCM4704 |
4 MB | 16 MB | Broadcom BCM5325 |
| Netgear WPN824 | V2.0.15_1.0.11 | Atheros AR2313 |
N/A | N/A | Marvell 88E6060 |
| Netgear WPNT834 | V1.0_28NA | Realtek RTL8651B |
4 MB | 32 MB | In processor |
| ZyXEL X-550 | X-550_V1.2 | Ubicom IP3023 |
4 MB | 16 MB | Realtek RTL8305SC |
Table 1: The Products and Technologies
For curious readers, photos of each of the router boards are found below. You can click on any of them for a larger view.
Figure 1: Airgo reference / Belkin Pre-N, Linksys WRT54GX main board
Figure 2: D-Link DI-634M main board
Figure 3: D-Link DGL-4300 main board
Figure 4: Netgear WPN824 main board
Figure 5: Netgear WPNT834 main board
(more photos can be found here)
Figure 6: ZyXEL X-550 main board
The Test
I had two requirements for the P2P stress test. The first was that the test had to be done without requiring connection to the Internet. While this might seem strange for applications that depend on interaction with numerous partners scattered across the world, it’s the only way that I can ensure that I’m testing the router and not dozens of Internet connections and their associated ISP’s.
The second requirement for the P2P test was that it had to force the router to establish and maintain more than 100 simultaneous connections. Both P2P and gaming applications open multiple simultaneous connections through a router’s firewall, with each session requiring CPU cycles and, more importantly, a chunk of not-so-plentiful memory.
For example, a search for available gaming servers might involve launching a few hundred server queries. Although the queries themselves are pretty small and don’t require a lot of router throughput, the sheer number of them can cause routers with buggy code to lock up and require rebooting. On the other hand, the bandwidth of most broadband connections limits the number of useful simultaneous BitTorrent or eDonkey sessions to a few dozen (or even fewer). But each connection is a steady stream taking as much bandwidth as the router (and its Internet connection) can provide.
Given these requirements and the tools available to me, I chose Ixia’s IxChariot as my instrument of router torture because it could handle up to 180 simultaneous connections and is a breeze to set up and modify. And its plots of throughput vs. time are handy to monitor both the in-process and finished test runs. I used the standard throughput.scr script with TCP/IP with the only modification to change the file_size parameter (the size of the transferred file) from its default of 100,000 to 1,000,000 Bytes.
Two computers were involved in the test, which were connected into my lab’s switch. The LAN-side machine also ran the IxChariot console and was a Dell Inspiron 4100 notebook with a 1 GHz Intel Celeron CPU, 576 MB of RAM and integrated 3Com 3C920 Ethernet adapter, running WinXP SP2 Home. The WAN-side machine was an HP Pavilion 716n with a 2.4 GHz Intel Pentium 4 CPU, 504 MB of RAM and Intel PRO / 1000 MT desktop Ethernet adapter, also running WinXP SP2 Home.
The WAN-side machine and WAN port on the router under test were assigned static IP addresses so that the test scripts could easily be re-used. The LAN-side machine was allowed to pick up whatever IP address was handed out by the router under test. Because IxChariot has known problems with NAT + SPI routers, I found that I had to both put the LAN side machine in DMZ in order to get the tests to run correctly. I also had to disable the SPI features of each router if it allowed me to do so.
For each router, I upgraded to the latest firmware and reset the router to factory defaults. Next, I opened the ports that IxChariot needed, put the LAN machine in DMZ and disabled the firewall SPI features if I could. I then attempted to run five tests with 2, 64, 128, 160 and 180 simultaneous connections.
In each case, I evenly split the connections between upload and download and ran the tests for one minute. While an even split between upload and download connections isn’t exactly the scenario that a router would encounter in a P2P application, I figured it would suit my purposes.
Test Results
Figure 7 shows the results plotted so that it’s easy to see the differences in the connections vs. throughput characteristics of the routers. The plot also contains a baseline trace showing test results with a straight 10/100 Ethernet connection. This was run both to ensure that my test setup could handle the test connection load and also to establish a baseline for connection vs. throughput. The graph shows that the Ethernet connection throughput does drop off with the number of connections; most likely due to IxChariot overhead.
Figure 7: Router Throughput vs. Simultaneous Connections
(click image to enlarge)
While none of the products tested locked up due to the heavy connection load, most were unable to successfully complete all five runs. I judged a run to be a failure when IxChariot either threw an error telling me that a test pair (a connection) failed to be initialized, ended abruptly, or failed to return a result after 2-3 minutes.
The Netgear RangeMax 240 tops the chart with 2, 64 and 128 connection performance virtually equal to the reference Ethernet connection. The RangeMax 240 isn’t perfect, however, since its throughput starts to degrade more than the other products above 128 connections. But on the plus side, the 240 survived all five tests, which is more than can be said for every other router except for the D-Link DGL-4300.
Speaking of the D-Links, Figure 7 shows that the DI-634M and DGL-4300 have nearly identical throughput for the runs that they both completed. But I’m not sure why the DGL-4300 was able to complete all five tests while the DI-634M was reliable only up to the 128 connection run, since both use the Ubicom processor and have 16 MB of RAM.
The poor showing of the Zyxel X-550 is also a puzzle, since its hardware design is virtually identical to that of the DI-634M. The Zyxel only completed the 2 and 64 connection runs and also had lower throughput than the D-Links.
The next throughput tier was occupied by the Belkin Pre-N and Linksys WRT54GX twins. Throughput was virtually identical, but the Belkin managed to complete the 160 connection test, while the Linksys was unable to go beyond 128.
The Netgear WPN824 RangeMax had the poorest showing, managing to complete only the two connection test. While the single-chip Atheros processor design may be good for product margins, it apparently can’t hold up to the rigors of P2P.
Closing Thoughts
Peer-to-Peer may be the domain of bootleggers, slackers and cheapskates today, but it probably will play an important part in the legal distribution of video in the not too distant future. Real-time video streaming, particularly of HD content that doesn’t have the “H” compressed out of it, simply won’t scale to the level needed for the content providers to make a decent buck.
So somewhere, sometime, the download experiments that movie providers like Movielink and CinemaNow are running will probably become a part of the normal video content distribution / acquisition system. And when that happens, it will be important for routers to be able to handle more than a handful of simultaneous connections. (See this article by Robert X. Cringely for his take on who the “legit” P2P players are likely to be.)
So assuming you buy the premise that your router needs to handle P2P, should you buy only the top performing RangeMax 240? In reality, most of the products tested will probably work fine on the broadband connections that most of you appear to have, judging from our recent poll on Internet download speed. Given download speeds between 5 and 10 Mbps and more limited upload speeds, you’ll probably only be able to open at most less than a dozen up and download streams. The possible exception is the original Netgear WPN824 RangeMax, which was the only product to fail the 64 connection test.
However, if you’ve got a nice, speedy 30 Mbps fiber connection, then you probably would want to aim for the Netgear RangeMax 240, or the D-Link DI-634M, DGL-4300, or its wired only twin the DGL-4100. Any of these should serve you nicely given that they support up to 128 simultaneous connections.
Finally, be aware that most ISPs monitor for P2P traffic and rather than threaten you with service termination, simply dial down your bandwidth. So while most of these routers may up to the challenge of handling high-bandwidth P2P traffic, you may find that when you go to fire them up, your big ol’ pipe gets pinched down into a little bitty straw, which is no fun at all.