Introduction
In Part 1 of this NTK, I gave an overview of the state of 802.11g development, described the key performance related issues, and offered some advice for prospective purchasers of these not-quite-ready-for-prime-time products.
In this second, and final part of the NTK, I’ll go into more detail on the performance differences that you’ll find between current Linksys and Buffalo Tech products. I’ll also present the results of my first detailed performance tests for products based on Intersil’s PRISM GT chipset.
A Closer Look at “Interoperability”
Much has been made in the press in the past month or so about “interoperability problems” among draft-11g products. From what I’ve seen, however, this may be partly a “whisper campaign” to spread FUD to slow down the competition, and partly incorrect conclusions due to the use of inadequate test methodology.
Although you may be inclined to think “functional issues” when you hear “interoperability problems”, my testing with draft-11g devices has shown throughput variation to be the dominant interoperability issue. In some cases, however, the interoperability effect on throughput is so severe that it can easily be mistaken for functional problems if simple file transfer or other techniques that look only at average or total throughput are used.
To really understand what these products are doing, you need to look at throughput performance over time, with the data preferably in graphic form. A simple example that reinforces this point is shown in Figure 1.
Figure 1: Throughput comparison – Linksys vs. Buffalo Tech – original
(click on the image for a full-sized view)
This Chariot test result, which you’ve seen before in Part 1of this NTK, shows the throughput performance of the first shipping Buffalo Tech and Linksys draft-11g products. Although the numerical average 21.8Mbps result for the Linksys is as indicative of the product’s performance as the throughput plot, that’s certainly not the case for the Buffalo Tech test pair.
You can see by looking at the lower trace in the plot that the time and duration of a measurement will make a significant difference in the results obtained. And once you start doing tests of draft-11g and 11b products running simultaneously in a mixed environment, looking at throughput vs. time is essential to understanding how the products under test are interacting.
I’ve also found it helpful to separate interoperability into three categories:
-
draft-11g and “Legacy” 802.11b
-
draft-11g and draft-11g (Vendor Interoperability)
- draft-11g and 802.11b (Mixed WLAN Interoperability)
I’ll look at each of these categories in the next sections.
Draft-11g and “Legacy” 802.11b
Revised March 10, 2003
This class of interoperability is due to problems caused by some older 802.11b products. It includes the Short Preamble related issues that I covered in Part 1, but also a misaligned data rate issue. This second problem comes from the fact that some 802.11b devices don’t know what to do when confronted with draft-802.11g stations advertising data rates other than the 1,2, 5.5 and 11Mbps supported by 802.11b devices. This problem initially prevented Cisco 340 and 350 cards from associating, i.e. connecting, with draft-11g APs. But most vendors have released firmware updates that fix the problem, so you probably won’t see it unless you have early-vintage product (Dec 2002 / Jan 2003) with original firmware.
I’m told that the IEEE Task Group g is aware of both issues and will probably be addressing them in future drafts or the final spec. In the meantime, product vendors have either already patched their firmware and drivers to fix these issues, or will do so shortly.
Note that with all the flavors and vintages of 802.11b equipment out there, I’d be hard pressed to say with absolute certainty that real, functional-failure-type problems in this category don’t exist. But I can say that in my testing using 802.11b products based on Intersil PRISM II, TI ACX-100, Agere Systems, and Atheros 5100X chipsets, I’ve yet to run across a situation where a draft-11g product, based on either the Broadcom or Intersil chipsets, would not associate and successfully transfer data.
Draft-11g Vendor Interoperability
This category includes interoperability issues that come from mixing products from different networking product vendors, with the primary effect being throughput variation. In Part 1 of this NTK, I had only products based on Broadcom’s draft-11g chipset to play with, and was still able to show a significant difference in performance between the Buffalo Technology and Linksys products tested. This time, I was able to add Intersil-based products to the mix, and found yet another variation in throughput behavior.
NOTE: Since I was unable to obtain review product from D-Link after repeated attempts, I worked directly with Intersil – the maker of the wireless chipset that D-Link uses in their “Xtreme G” draft-11g products. Although I can’t say which specific products I used for my testing, let’s just say that the results should be representative of Intersil-based products that you can buy today.
Figure 2: Intersil, first version
(click on the image for a full-sized view)
Figure 2 shows the performance with the first available firmware for the Intersil devices. As with my other tests, I used both a client and AP based on the same (in this case Intersil) chipset, and had no 802.11b clients active for this test. You can see a relatively steady throughput characteristic, but slower than the throughput obtained with Broadcom-based products.
After reporting my results to Intersil, they sent new client and AP firmware (which has since become publicly available), which I installed and used to generate the results shown in Figure 3.
Figure 3: Intersil, first vs. new versions
(click on the image for a full-sized view)
The resulting 35% throughput improvement was not bad, but as Figure 4 shows, still not equal to the latest available Broadcom-based products.
Figure 4: Intersil new vs. Broadcom new
(click on the image for a full-sized view)
Figure 4, which compares Intersil based product with the Linksys WRT54G router and WPC54G CardBus card, shows an approximately 18% throughput advantage for the Broadcom-based Linksys pair.
Draft-11g – WEP effects
Unfortunately, for Intersil at least, this isn’t the only bad throughput news. My testing also showed an approximately 10% WEP-enabled throughput hit, which is shown in Figure 5.
Figure 5: Intersil – with and without 128bit WEP enabled
(click on the image for a full-sized view)
These results caused me to go back and rerun the same test on the Linksys pair, with the results shown in Figure 6.
Figure 6: Broadcom – with and without 128bit WEP enabled
(click on the image for a full-sized view)
Although there’s a slight difference of about 1%, I’d say the difference is well within the uncertainty of the measurement and therefore not significant.
The result of all this is that with 128bit WEP enabled, Broadcom’s average throughput advantage over Intersil – with 11g stations only and no 11b clients in range – increases to about 35%. When I asked Intersil whether they would be able to make the WEP-enabled throughput hit go away, I was assured that the slowdown was due to early firmware, and would be “ironed out” in a future release.
But before you start taking Intersil-based draft-11g products off your shopping list, you’d better take a look at what happens when 802.11b stations are added to the picture.
Mixed WLAN – Station association
The test that really differentiates draft-11g products is how they function in “mixed” WLANs, i.e. those containing both draft-11g and 802.11b stations. If you remember from Part 1, the mechanism that dictates how 11g and 11b clients play together is called “Protection”, and is a major influence on throughput when 11b clients are around. It’s also an area that continues to evolve, with a major change in behavior dictated in the 6.1 version of the draft standard at the end of January.
Although the 11g draft standard does not require the use of protection, from what I can see from my latest round of testing, the trend seems to be toward automatic adaptive 11b protection. The main effect of this trend is it produces three distinct throughput levels for draft-802.11g devices:
-
Level 1 – No 11b clients sensed (Highest throughput)
-
Level 2 – 11b clients associated, but idle
-
Level 3 – 11b clients active (Lowest throughput)
Although exact throughput values and the percent change from level to level depend on implementation, virtually all devices I’ve tested so far exhibit this tri-level behavior. The only exception is Intersil’s first available firmware, which has only two Levels, as illustrated below.
• The basic test setup and methodology used for testing is described here.
• All the following tests were taken under best-case signal conditions, with AP / router and client about five feet apart. WEP was not enabled unless otherwise noted.
Figure 7 shows a comparison plot of two Chariot runs. The bottom trace shows the results with the first-available Intersil client driver and AP firmware, and the top trace shows newer firmware released a few weeks later. During both runs, I plugged a NETGEAR WAB501 Dual-Band CardBus card (Atheros 5100X-based) into a second laptop sitting close by at about the 20 second point in the test run. I then shut down and removed the WAB501 at about the 50 second mark.
Figure 7: Intersil First vs. New – Temporary WAB501 Association
(click on the image for a full-sized view)
Two things are clear from the test runs:
-
The first-available firmware showed no change in throughput with or without an 11b client associated. I attribute this to a non-adaptive implementation of 11b protection that effectively limits maximum throughput to a lower, “Level 2” value.
- The newer firmware implements an adaptive protection mechanism. This version of protection yields a higher “Level 1” throughput in what appears to be a protection “standby” or “sniffing” mode, but then drops to the same lower “Level 2” throughput once an 11b client is detected and associates with the AP.
By the way, don’t associate any significance to that fact that the lower trace stops after one minute. I just was in a hurry and terminated the test when it was clear that things weren’t going to change.
It turns out that I’d seen a similar behavior with the Broadcom-based products, which is easier to see with the Linksys implementation because it doesn’t have the Buffalo Tech’s “throughput hopping” behavior. Figure 8 shows you what I mean.
Figure 8: Intersil New vs. Linksys New – Temporary WAB501 Association
(click on the image for a full-sized view)
This plot shows the same experiment that I ran for Figure 7, but comparing the newer Intersil firmware with the newer Linksys-Broadcom firmware contained in the WRT54G router. I was a little sloppy with my plug and unplug times for the WAB501 11b client, so don’t attach any significance to the difference in times of the throughput changes in Figure 8.
The main things to note here are that both Intersil and Broadcom seem to be pursuing similar “adaptive protection” strategies and that Broadcom presently seems to have a more efficient implementation, as shown by their higher throughput for both “Level 1” and “Level 2” states.
Mixed WLAN – More Station Association
Further experiments showed that different 802.11b clients had varying effects on the draft-11g test pair’s throughput when they temporarily associated with the draft-11g AP.
Figure 9: Intersil New – WAB501 vs. ORiNOCO Temporary Association
(click on the image for a full-sized view)
Figure 9 uses the Intersil-based draft-11g test pair and once again shows the throughput effect from the temporary association of a WAB501 client. But this time it also overlays the effect from the temporary association of an ORiNOCO Gold client. Although both cause a throughput drop at about the same time, the effect from the WAB501’s association goes away seconds after I shut down and remove the card. But throughput doesn’t recover from the ORiNOCO’s association until about a minute after I remove the card. I found a similar effect from the temporary association of Intersil PRISM II and TI ACX-100 based clients.
This delayed-recovery effect isn’t limited to Intersil, either. Figure 10 shows the results from the same pair of tests as shown in Figure 9, but run with the Linksys WRT54G / WPC54G pair instead.
Figure 10: Broadcom New – WAB501 vs. ORiNOCO Temporary Association
(click on the image for a full-sized view)
The plot again shows a quick recovery from the temporary association of the NETGEAR WAB501 client, but that it takes about three minutes for the Linksys pair to switch back to its maximum throughput “standby” protection mode.
I ran similar tests with both Intersil and Linksys/Broadcom test pairs and other 11b clients, and found that the Atheros-based NETGEAR WAB501 was the only 11b client that consistently allowed a fast recovery to the “Level 1” throughput mode.
The other observation worth noting is that I could clearly see that the drop to the Level 2 throughput didn’t take place until the 11b client was actually associated with the draft-11g AP under test. I could see quick throughput “dropouts” during the association process, but only a successful association caused the switch to the lower “Level 2” state.
I’ll next look at what happens when you try to ignore 11b stations!
Mixed WLAN – No Protection
Although the “adaptive protection” that it looks like both Intersil and Broadcom are adopting would seem to eliminate the need for a “G-Only” mode, Broadcom appears to still be allowing that alternative. The feature goes by different names (Buffalo Tech calls it “Turbo” and Linksys uses a more descriptive “G-Only” moniker), but the action is the same, i.e. disabling the 11b protection mechanism.
Intersil, however, is a firm believer in protection and doesn’t provide a way to disable it. While this might seem to put buyers of Interil-based products at a disadvantage, my testing shows that there’s little to be gained by disabling protection, and that turning it off can cause more harm than good.
Figure 11 shows two Chariot runs for the Linksys WRT54G / WPC54G test pair, which uses the latest available Broadcom firmware. I again used the Linksys products because their absence of “throughput hopping” makes it easier to detect the changes that are of interest.
Figure 11: Broadcom New – Mixed vs. G-Only modes, No 11b Stations
(click on the image for a full-sized view)
The screenshot shows two Chariot runs, one in “Mixed Mode” and the other in “G-Only” mode. Comparing the average throughput values for the two runs shows a mere 3% throughput gain in “G-Only” mode, hardly anything to write home about. But assuming you’re the type who has to squeeze every last Mbps from their network connection, what’s the harm in throwing the “G-Only” switch?
If you think back to Part 1, you’ll remember that disabling protection doesn’t prevent draft-11g stations and APs from hearing 11b data transfers, it just prevents understanding them. With protection disabled, 802.11b clients become just like any other 2.4GHz noise source and can interfere with your 802.11g WLAN’s transmissions. The following tests once again illustrate this point.
Figure 12: Broadcom New – G-Only Mode. NETGEAR MA401 in range
(click on the image for a full-sized view)
Figure 12 takes the same Linksys test pair used in Figure 11, sets the AP in G-Only mode, then brings a NETGEAR MA401 station into range. (The other trace in the plot is a reference run in G-Only mode, but without any active 11b clients in-range.) Although the “throughput dropouts” are visually disturbing, a glance at the average throughput numbers for the two runs shows little real impact.
The effect, however, can differ from client to client, as the next test shows.
Figure 13: Broadcom New – G-Only Mode. NETGEAR WAB501 in range
(click on the image for a full-sized view)
Figure 13 shows the same test runs as Figure 12, but this time using a NETGEAR WAB501 Dual-Band client, forced to 11b-only mode. I’m not sure what the WAB501 is doing, but this time it has a significant effect, knocking the average throughput down about 18% from test run to test run.
You also need to remember that the inability to understand (and potential for interference) works in the other direction too. Although I haven’t run any tests to quantify the effect, Protection-disabled draft-11g stations will act as interference sources for 11b-based WLANs. Although you might have nice, new draft-11g stuff and think you’re getting an extra boost from running “G-Only”, you could be interfering with your next-door neighbor, who hasn’t yet decided to upgrade his 11b WLAN!
Once again, the lesson here is that you run draft-11g gear with Protection off at your own risk. Although it may come across as somewhat paternalistic, maybe that’s why Intersil doesn’t give you this switch to play with.
Mixed WLAN – Broadcom implementations
The real test of draft-11g implementations is how they handle WLANs with both draft-11g and 11b active stations. Part 1 gave you a taste of how Broadcom handles this complex juggling act. I’ll now show a few more examples, including how Intersil handles mixed networks.
Figure 14 shows a test done with original firmware using the Broadcom-based Buffalo Tech WBR-G54 router and Buffalo WLI-CB-G54 draft-11g and ORiNOCO Gold 11b clients. The Buffalo Tech card starts first, and is joined by the ORiNOCO card at the 20-second mark. Both cards run until the Buffalo Tech card completes its programmed number of transfers and stops, letting the ORiNOCO card finish by itself.
Figure 14: Buffalo Tech Original – WLI-CB-G54 & ORiNOCO Stations
(click on the image for a full-sized view)
The results show that the draft-11g client throughput drops to a lower “Level 3” value once the 11b station becomes active, but that the 11g client and 11b client appear to share their time together relatively well. Note that the Buffalo Tech client’s throughput is slightly greater than the ORiNOCO’s.
Figure 15 shows the same test, but this time run with the latest available Buffalo Tech AP firmware.
Figure 15: Buffalo Tech New – WLI-CB-G54 & ORiNOCO Stations
(click on the image for a full-sized view)
Once again, the Buffalo Tech and ORiNOCO clients work well together and appear to have almost equal throughput while both are active. But although you might think that all Broadcom implementations are equal, the next tests using Linksys gear will show that you’d think wrong!
Figure 16 shows a similar test run as before, but this time using a Linksys WAP54G AP and WPC54G draft-11g and NETGEAR MA401 11b clients. The WAP54G is running its original firmware.
Figure 16: Linksys Original – Linksys WPC54G & NETGEAR MA401 Stations
(click on the image for a full-sized view)
This time, you can see that the Linksys client gets the raw end of the throughput deal, dropping below the MA401. The effect on the Linksys client was so severe, that if I hadn’t been running Chariot and been able to see that it was still running, I probably would have thought that the transfer had hung.
I next switched in the Linksys WRT54G router that had the latest available Broadcom firmware and reran the same test. (I set up the router as an access point so that the router’s throughput wouldn’t get in the way of the measurement.)
Figure 17: Linksys New – Linksys WPC54G & NETGEAR MA401 Stations
(click on the image for a full-sized view)
Figure 17 shows that performance has changed, but not really in the direction it’s supposed to! With the new firmware, the 11b NETGEAR card looks like it’s not only still getting better throughput than the Linksys card, but also higher throughput than before!
This indicates to me that the Broadcom firmware hasn’t yet implemented the changes mandated in the 6.1 version of the 802.11g draft spec, which dictate that 11g stations get more time on the air than 11b stations. A recent conversation with Broadcom confirmed that draft 6.1 changes weren’t incorporated into the firmware versions that I tested, since Broadcom said they had released their “6.1” firmware to their customers only within the last week or so.
So how about Intersil? Do they handle active draft-11g and 11b clients any better?
Mixed WLAN – Intersil implementation
Although Intersil may be slightly behind Broadcom in the draft-11g-only throughput race, right now, they’re ahead in their handling of mixed 11b / draft-11g stations. A look at Figure 18 will show you why I say this.
Figure 18: Intersil Original – Intersil & NETGEAR WAB501 Stations
(click on the image for a full-sized view)
Once again, I set up my mixed draft-11g / 11b test, but this time used the Intersil AP and client along with a NETGEAR WAB501 Dual Band card (forced to 11b mode). The test result clearly shows that the draft-11g client has much better throughput than any mixed-mode case that I tested with Broadcom-based gear. Sure looks to me like Intersil got some of the 6.1 draft changes included into their first release!
Figure 19: Intersil New – Intersil & NETGEAR WAB501 Stations
(click on the image for a full-sized view)
I also reran the test using the newer firmware that Intersil provided, but Figure 19 doesn’t show much difference in performance. I also ran the same test using the other 11b clients that I test with and the results are shown in Figures 20 – 22.
Figure 20: Intersil New – Intersil & ORiNOCO Gold Stations
(click on the image for a full-sized view)
Figure 21: Intersil New – Intersil & NETGEAR MA401 Stations
(click on the image for a full-sized view)
Figure 22: Intersil New – Intersil & D-Link DWL-650+ Stations
(click on the image for a full-sized view)
The Atheros 5100X-based NETGEAR WAB501 and TI ACX-100-based D-Link DWL-650+ seem to have the best mixed mode operation with the present firmware. Although these results are encouraging, they show Intersil also still has some work to do to achieve the desired mixed-mode performance.
Conclusions
So there you have it, probably more than you ever wanted to know about draft-11g products and what makes them tick. Even with the first Intersil-based products now in the market, I still haven’t seen anything that would cause me to change most of my summary from Part 1.
These are products based on a draft standard that’s still undergoing significant changes, and the general smart shopper’s motto of “Caveat Emptor” (Buyer Beware) definitely should be your guide. Remember, the folks trying to sell you this stuff understand what “draft standard” means. It’s up to you to make sure that you do!