Introduction
In Part 1, I provided details on how my OFDMA benchmark test was developed. In this second and final part, I’ll show the results for two Wi-Fi 5 (802.11ac) and six Wi-Fi 6 (802.11ax) routers that were run through the process.
I used the last variation of the test described in Part 1. In sum:
- iperf3 TCP/IP traffic run simultaneously to four STAs (Intel AX200, Win 10, 21.80.2.1 driver)
- bitrate (-b): 50Mbps
- length (-l): 256 Bytes
- DSCP values (–dscp), one per STA: 0 (CS0), 96 (CS3), 160 (CS5), 192 (CS6)
- 200ms interval ping run concurrently from AP to STA on each pair. Ping is always DUT to STA for both uplink and downlink traffic.
- congestion measured using an octoScope Pal6 (Qualcomm AX based) associated to the DUT, running 1bps of traffic so that stats could be recorded.
Table 1 shows the routers tested, with key attributes.
| Product | Streams | Platform | OFDMA | Firmware | Notes | |||
|---|---|---|---|---|---|---|---|---|
| 2.4 GHz DL |
2.4 GHz UL |
5 GHz DL |
5 GHz UL |
|||||
| ASUS RT-AX58U / RT-AX3000 [$180] |
2 (4, 5 GHz Rx only) |
Broadcom | Yes | Yes | Yes | Yes | 3.0.0.4.384_8601 | BCM43684: 5 GHz BCM6750: 2.4 GHz |
| ASUS RT-AX88U [$350] |
4 | Broadcom | Yes | Yes | Yes | Yes | 3.0.0.4.384_9107 | BCM43684: both radios |
| ASUS GT-AX11000 [$450] |
4 | Broadcom | Yes | Yes | Yes | Yes | 3.0.0.4.384_9165 | BCM43684: all radios |
| Evenroute IQrouter V3 [$139] |
2 | Mediatek | N/A | N/A | N/A | N/A | v3.3.2 | MT7610E: 5 GHz MT7603E: 2.4 GHz Wi-Fi 5 router. No OFDMA |
| NETGEAR R7800 [$170] |
4 | Qualcomm | N/A | N/A | N/A | N/A | V1.0.2.68 | QCA9984: both radios Wi-Fi 5 router. No OFDMA |
| NETGEAR RAX15/20 [$100] |
2 | Broadcom | Yes | Yes | Yes | Yes | V1.0.1.56 | BCM6755: both radios |
| NETGEAR RAX45/50 [$330] |
2 -2.4 GHz 4 – 5 GHz |
Broadcom | Yes | Yes | Yes | Yes | V1.0.2.20 | BCM43684: 5 GHz BCM6750: 2.4 GHz |
| NETGEAR RAX120 [$400] |
4 | Qualcomm | Yes | No | Yes | No | V1.0.1.122 | Qualcomm QCN5054 (5 GHz x2) Qualcomm QCN5024 (2.4 GHz) Chipset not capable of UL OFDMA |
Table 1: Routers tested
Since there are so many routers, including all products produces pretty unreadable plots. So I’ll be showing separate plots for the ASUS and NETGEAR routers. Both sets of plots will include two Wi-Fi 5 routers and octoScope’s Pal6 configured as a four-stream AX AP for reference.
The NETGEAR R7800 is a four-stream Wi-Fi 5 (802.11ac) router that is my "go-to" AC reference router. Whenever I have a product that is behaving oddly, I put the R7800 back in the chamber to ensure that it’s the product, not the testbed that is misbehaving.
The OpenWRT-based Evenroute IQrouter V3 was included because it has the codel and airtime fairness work of the MakeWiFiFast group baked into its wireless drivers for its Mediatek chipsets. IQrouter’s feature set mainly focuses on optimizing wired routing performance and, specifically, reducing the effects of bufferbloat on the router’s ISP connection. But since it also has some Wi-Fi latency reduction, it’s included to see how it stacks up against the R7800 and the AX products.
(Not so) funny story; I actually wrote this article twice. The first version, which posted for about 15 minutes before I took it down, was based on bad data. Shortly after the article posted, I went back to the testbed to start to reconfigure it and found one of the channels for one of the STApals was not connected. Yup, just hanging in the breeze. After some verbal self-abuse and a few more dents in the lab wall, I started out to retake the data.
But while I was at it, I figured I might as well make the test configuration better support MU-MIMO. After its years of not adding value to Wi-Fi performance, I’m told that MU-MIMO actually works now, at least the 802.11ac downlink form of it. The 802.11ax standard has added an uplink form of MU-MIMO, but it wasn’t supported in the first AX chipsets.
MU-MIMO uses beamforming, which requires spatial diversity, i.e. devices must be physically separated. The testbed didn’t support this because all four STApals used the same two antennas.
OFDMA testbed – before
The figure below shows the testbed’s new configuration. Pairs of STApals each have their own attenuator and are connected to their own pair of antennas, situated at corners of the 38" octoBox. The unused ports of the 1:4 RF splitter/combiner all have 50 ohm terminators. Ideally, each STApal would have its own antenna pair sitting in a box corner. But even this non-ideal configuration revealed performance differences when MU-MIMO was enabled and disabled in test runs.
OFDMA testbed – after
On to the results.
Latency – ASUS
We’ll start with a summary of 80% probability downlink latencies with OFDMA off and on in Table 2.
| Product | 80% latency – OFDMA off | 80% latency – OFDMA on | % Change |
|---|---|---|---|
| Evenroute IQrouter V3 | 10.1 | 10.1 | – |
| NETGEAR R7800 | 8.9 | 8.9 | – |
| octoScope Pal6 | 24.5 | 7.0 | -71 |
| ASUS GT-AX11000 | 5.7 | 4.2 | -26 |
| ASUS RT-AX58U / RT-AX3000 | 4.6 | 6.2 | +35 |
| ASUS RT-AX88U | 4.9 | 3.4 | -31 |
Table 2: ASUS router OFDMA latency summary – downlink
The first plot shows downlink latency with OFDMA disabled for the ASUS routers and three "reference" products, using an 80% probability benchmark. While all the ASUS AX products are clustered fairly closely together, the reference octoScope Pal6 is not. Note the two AC reference products, the IQrouter v3 and NETGEAR R7800, have similar behavior.
ASUS Latency CDF – OFDMA off – downlink
I rechecked the Pal6 results at least half a dozen times. Some of the results are shown below, with the value used above in bold black.
Pal6 Latency CDF – OFDMA off – downlink – retests
The downlink latency with OFDMA enabled plot confirms the results in the table above. The Pal6 is most improved with latency decreasing 71%. The RT-AX58U is the only router to get worse, increasing 35%. Still, all the ASUS routers still have latencies better than the AC reference routers.
ASUS Latency CDF – OFDMA on – downlink
Turning to uplink latencies, the 80% probability latencies are summarized with OFDMA off and on in Table 3. Once again, the Pal6’s latency is most improved, decreasing 57%. And, once again, the RT-AC58U is the only one of the three ASUS routers where latency gets worse.
| Product | 80% latency – OFDMA off | 80% latency – OFDMA on | % Change |
|---|---|---|---|
| Evenroute IQrouter V3 | 22.6 | 22.6 | – |
| NETGEAR R7800 | 24.9 | 24.9 | – |
| octoScope Pal6 | 25.3 | 11.0 | -57 |
| ASUS GT-AX11000 | 17.7 | 15.9 | -10 |
| ASUS RT-AX58U / RT-AX3000 | 15.1 | 18.1 | +20 |
| ASUS RT-AX88U | 17.8 | 14.9 | -16 |
Table 3: ASUS router OFDMA latency summary – uplink
The uplink latency with OFDMA disabled plot shows all the ASUS routers clearly differentiated from the "reference" products.
ASUS Latency CDF – OFDMA off – uplink
The uplink latency with OFDMA enabled plot shows the big change in Pal6 latency. Once again, the AX products generally have lower latencies than the AC.
ASUS Latency CDF – OFDMA on – uplink
Latency -NETGEAR
Table 4 has a summary of 80% probability downlink latencies with OFDMA off and on for the three AX NETGEAR routers tested. Like the three ASUS routers, all three NETGEAR OFDMA off latencies are lower than the two AC and Pal6 reference products. Latencies for the Broadcom-based RAX15 and RAX45 are about half the Qualcomm-based RAX120, the only Qualcomm-based router in this test.
| Product | 80% latency – OFDMA off | 80% latency – OFDMA on | % Change |
|---|---|---|---|
| Evenroute IQrouter V3 | 10.1 | 10.1 | – |
| NETGEAR R7800 | 8.9 | 8.9 | – |
| octoScope Pal6 | 24.5 | 7.0 | -71 |
| NETGEAR RAX15 | 4.2 | 3.3 | -21 |
| NETGEAR RAX45 | 4.9 | 4.9 | 0 |
| NETGEAR RAX120 | 7.9 | 7.7 | -3 |
Table 4: NETGEAR router OFDMA latency summary – downlink
Here’s the downlink latency with OFDMA disabled plot…
NETGEAR Latency CDF – OFDMA off – downlink
…and downlink latency with OFDMA enabled, below. There’s not much else to say.
NETGEAR Latency CDF – OFDMA on – downlink
Turning to uplink latencies, the 80% probability latencies are summarized with OFDMA off and on in Table 5. Latency improvement is much better than downlink for the RAX120. But enabling OFDMA seems to seriously break the RAX15. To check, I repeated the test, with similar results.
| Product | 80% latency – OFDMA off | 80% latency – OFDMA on | % Change |
|---|---|---|---|
| Evenroute IQrouter V3 | 22.6 | 22.6 | – |
| NETGEAR R7800 | 24.9 | 24.9 | – |
| octoScope Pal6 | 25.3 | 11.0 | -57 |
| NETGEAR RAX15 | 13.9 | 145.5 | +947 |
| NETGEAR RAX45 | 15.9 | 13 | -18 |
| NETGEAR RAX120 | 29.4 | 22 | -25 |
Table 5: NETGEAR router OFDMA latency summary – uplink
The uplink latency with OFDMA disabled plot shows a definite shift to the right (higher latencies) vs. the downlink plot, just as we saw with the ASUS routers. Note the RAX120 with higher latency than the R7800.
NETGEAR Latency CDF – OFDMA off – uplink
The uplink latency with OFDMA enabled plot shows the RAX120 now better than the R7800 and about even with the IQrouter v3. The RAX15 is, as they say, off the chart, but not in a good way.
NETGEAR Latency CDF – OFDMA on – uplink
Conclusion: My takeaway for latency is that it appears OFDMA can provide some reduction in latency. But I suspect the effect would not be noticeable in the real world. It’s not like switching OFDMA on is taking 100’s of milliseconds of delay and cutting it to 1 ms.
Efficiency
AX’s prime directive is to increase airtime efficiency, i.e. use less airtime to move a given number of packets. Fortunately, octoScope’s Pals measure airtime congestion. So by looking at the difference in congestion with OFDMA on and off, we can see if AX is following its directive. I’ll be using bar plots of Airtime congestion measured by the Pal6 associated to the DUT to show what OFDMA brings to the party.
The downlink congestion plot shows the three ASUS routers, with the first bar in each pair the average OFDMA off congestion and the second the on. Although congestion is relatively low, hovering around 25% for all products, you see a slight increase in congestion when OFDMA is enabled for each product. Much like we saw with latency, however, you’d never notice this in real-world use.
ASUS airtime congestion – OFDMA effect – downlink
I didn’t include the Pal6 and two AC reference products on the plot above, because the Pal6 would have compressed the scale, making it difficult to see the subtle change in airtime congestion. So here are the products plotted separately. Looks like high latency and high congestion go together for the Pal6.
Reference product airtime congestion – OFDMA effect – downlink
Here’s the uplink congestion plot for the ASUS trio. Airtime use is more than double that of downlink. And, this time, enabling OFDMA ever-so-slightly lowers congestion. But you wouldn’t notice it and it’s just as likely just measurement variation.
ASUS airtime congestion – OFDMA effect – uplink
Here’s uplink congestion for the reference products. The IQrouter v3 and R7800 both have lower congestion in this direction. But look at the Pal6! Its almost 20% reduction in congestion is the most significant change I’ve seen in all these tests.
Reference product airtime congestion – OFDMA effect – downlink
Now for the NETGEARs. Here’s downlink…
NETGEAR airtime congestion – OFDMA effect – downlink
…and uplink, below. We finally see a product stand out from the pack, the two-stream RAX15. It has both lower congestion than all the other products tested and the congestion is significantly reduced when OFDMA is enabled. In fact, all three NETGEAR routers show lower congestion when OFDMA is enabled. But the reduction for the RAX45 is pretty slight.
As we saw with the ASUS routers, the three NETGEARs’ uplink congestion is more than twice their downlink. The RAX15 congestion increase with OFDMA is large enough to be real. But the changes in RAX120 and RAX45 congestion are likely within measurement accuracy.
NETGEAR airtime congestion – OFDMA effect – downlink
Conclusion: While there is some evidence that OFDMA affects airtime efficiency, the effect, when seen, is very slight.
Throughput
The story for total throughput improvement with OFDMA enabled isn’t any rosier than for latency or congestion. OFDMA’s desired effect here is higher aggregate throughput. But if there is any improvement seen in these tests, it’s very slight; well within the +/- 5% that I use as a rule of thumb when dealing with any Wi-Fi measurements. I’m going to use bar charts of average throughput, since time plots aren’t that helpful.
I’ll use the same plotting method used for airtime congestion, with OFDMA off, then on, for each product. Here’s ASUS downlink. Not much to see here.
ASUS average aggregate throughput – OFDMA effect – downlink
And average dowlink throughput for the reference products. The Pal6’s lower throughput with OFDMA off tracks with its higher latency we saw earlier.
Reference product average aggregate throughput – OFDMA effect – downlink
And here’s uplink. Aside from around 20 Mbps lower average throughput, there isn’t much to see here, either.
Average aggregate throughput – OFDMA effect – uplink
Here’s uplink for the references. No big OFDMA on/off difference here for the Pal6.
Reference product average aggregate throughput – OFDMA effect – uplink
Moving on to the NETGEARs. Downlink…
NETGEAR average aggregate throughput – OFDMA effect – downlink
…and uplink. The only significant change is the RAX15, going in the opposite of the desired direction. This appears to track with the big change in congestion observed earlier
NETGEAR average aggregate throughput – OFDMA effect – uplink
Conclusion: OFDMA does nothing to improve aggregate throughput.
One last observation about throughput. Note that this benchmark uses four, 50 Mbps throughput streams, but none of the products achieve 200 Mbps total throughput. Experiments I’ve done show some of this is due to the mix of traffic priorities and some is due to limiting buffer length.
Closing Thoughts
It’s been a long wait to have consumer router manufacturers enable OFDMA in their Wi-Fi 6 products. Along the way, I’ve tried various OFDMA test methods, some suggested by Broadcom, others by router manufacturers. My conclusion at this point is that OFDMA produces no discernable benefit for the average consumer.
It’s possible that more devices and different traffic mixes will show latency, throughput and/or airtime congestion improvements. But the use cases are very specific and the improvements are not consistent enough to make OFDMA a reason to buy a Wi-Fi 6 router.
At this point, I’m going to set aside my OFDMA testing efforts and turn back to putting together a new Wi-Fi test suite so that I can get back to product testing. Sadly, OFDMA is yet another technology borrowed from the mobile/cellular world that doesn’t deliver the goods in the Wi-Fi world, where the device, not the network, is the boss.