MU-MIMO Retest: Six Routers Compared

Introduction

With our updated VeriWave test set in place and a new standardized MU-MIMO test process, I went back and retested all the MU-MIMO enabled routers that have crossed the SmallNetBuilder test bench to date. The only exception is Amped Wireless’ Athena. I didn’t have one on hand and decided to not request one. Amped announced its redesigned replacement, the ATHENA-R2 at CES 2016, which is scheduled to start shipping any day now. So we’ll just test the new one when Amped sends it for review.

I checked for new firmware before each router was tested. The products and firmware versions in this test are compiled in Table 1.

Keep in mind all products tested are based on Qualcomm Atheros MU-MIMO radios. Table 1 shows three different QCA MU-MIMO radios used: the original 4×4 QCA9980; latest generation 4×4 QCA9984 and 3×3 QCA9982.

Products using Broadcom MU-MIMO capable radios have only "beta" level firmware released at this time. I’m waiting for released firmware before adding Broadcom-based MU-MIMO routers to the Router Charts. But I did test an ASUS RT-AC88U with beta MU-MIMO firmware that I’ll show the results for later on.

MU-MIMO

The primary benefit of MU-MIMO is higher total throughput when multiple MU-MIMO devices are in use. The first chart shows a 2-to-1 difference between best and worst performing products for maximum MU-MIMO throughput gain.

MU-MIMO throughput – maximum

The 802.11ac standard supports up to four clients in a single AP-to-STA transmission. In practice, today’s 4×4 AP and router designs support only three STAs per transmission. The diagram taken from Is MU-MIMO Ready For Prime Time? shows what happens with four MU-MIMO STAs. The second transmit frame swaps out MU STA #1 and swaps in MU-STA #4. STA #1 then gets serviced in another transmission as a normal (SU) STA.

Four STAs – MU

As more MU STAs are added, each must be swapped in and out. Ideally, the maximum throughput reached with three MU STAs would continue indefinitely. But in reality, the overhead of efficiently scheduling MU and SU airtime takes its toll and throughput decreases.

The plot below compares the NETGEAR R7800, which had the highest total throughput gain with the TP-LINK Archer C2600, which had the lowest. Also included is the Linksys EA7500, the first 3×3 router supporting MU-MIMO. The NETGEAR yielded peak throughput of 807 Mbps with three MU-MIMO STAs, as expected. But the TP-LINK didn’t produce its lower peak throughput (390 Mbps) until six MU STAs were in use.

MU-MIMO Throughput vs. STA

The 3×3 Linksys EA7500 is interesting. Total throughput gain peaks with two devices at 462 Mbps, slightly better than half the NETGEAR R7800’s total MU-MIMO gain. Four devices drop throughput back down below the single device starting point, which then jumps back up with five devices. Past that, results were obtained only with 6, 8 and 14 devices. In the missing tests, all MU-MIMO STAs were able to connect, but would not pass the test TCP/IP traffic.

SU-MIMO

Without the benefit of MU-MIMO, routers should share available throughput evenly among all connected clients, assuming ideal conditions. Our test makes conditions as ideal as possible, using a cabled environment and identical simulated STAs. So the SU (single user) MIMO tests are a good indication of how efficent routers are at serving multiple STAs.

The plot below shows average SU-MIMO throughput for the six tested routers. Hmmm, what’s up with the TP-LINK?

SU-MIMO throughput – average

The next plot again shows the TP-LINK Archer C2600, NETGEAR R7800 and Linksys EA7500. Of these three, the NETGEAR is the only one showing proper SU behavior. The Linksys would not pass traffic beyond 6 STAs and the TP-LINK not only starts out with less than half the proper throughput, but also starts significantly declining after 5 STAs. Throughput then falls to single digits beyond 12 STAs.

SU-MIMO Throughput vs. STA

Pulling the TP-LINK plot from Is MU-MIMO Ready For Prime Time? shows SU performance was also problematic then.

MU vs. SU Throughput – TP-LINK Archer C2600

MU vs. SU

In Is MU-MIMO Ready For Prime Time?, I used composite MU and SU plots to compare MU to SU throughput as the number of MU STAs increased. For the Router Charts, this is simplified with the MU, SU Throughput Difference plot. The average view shown below lets you quickly zero in on the better performing products.

MU, SU Throughput difference – average

You can then compare up to four products to see how MU throughput gain varies with the number of MU STAs. The results are very different for our three example products. Only the NETGEAR R7800 has the expected performance. It even does a great job of maintaining total throughput gain as the number of MU STAs increases.

Most products behave more like the TP-LINK and fall to near zero throughput gain by the time our 16 STA test limit is reached. The Linksys EA7500’s plot is misleading; throughput for 8 and 14 STAs is only as high as shown because there are no SU results for those tests.

MU, SU Throughput difference vs. STA

We’ll now look at the MU, SU Difference results for each product. The NETGEAR R7800 is the best example of what you should expect from a MU-MIMO router. Total throughput gain peaks at three devices, then remains as close to that peak as possible as more MU devices are added. Some throughput decline is expected, but the slope should be as gentle as possible.

MU, SU Throughput difference vs. STA – NETGEAR R7800

The NETGEAR R7500v2 would be our next pick for a well-behaved MU-MIMO router. Its curve is nowhere as good as the R7800’s. But the router does a decent job of keeping throughput gain high out to 11 MU STAs.

MU, SU Throughput difference vs. STA – NETGEAR R7500v2

The Linksys EA8500 begins the transition into products with less desirable MU-MIMO behavior. Its throughput doesn’t peak until six STAs, but at least it’s a decent gain when it does. Throughput decline is fairly steep, but there is still 100 Mbps gain over SU with 14 STAs.

MU, SU Throughput difference vs. STA – Linksys EA8500

TRENDnet’s TEW-827DRU starts out well, but only produces 250 Mbps of additional throughput when it properly peaks with three devices. From there, decline is swift with many point of net throughput loss after 9 STAs.

MU, SU Throughput difference vs. STA – TRENDnet TEW-827DRU

TP-LINK’s Archer C2600 is another router you should let someone else buy if you have a lot of MU-MIMO devices. Like the Linksys EA8500, it produces peak throughput gain with six STAs, then quickly loses the gain as more STAs are added. The more troubling behavior is the net throughput loss with up to three STAs in use.

MU, SU Throughput difference vs. STA – TP-LINK Archer C2600

QCA and Linksys have some work to do to improve the 3×3 Linksys EA7500‘s MU behavior. As noted previously, behavior seems to fall apart after six MU devices are connected. And MU throughput gain is not very impressive up to that point.

MU, SU Throughput difference vs. STA – Linksys EA7500

Broadcom MU-MIMO

Curiousity got the better of me and I decided to see if Broadcom’s beta MU-MIMO firmware produced anywhere near reasonable performance. It didn’t. I loaded up an ASUS RT-AC88U with 9.0.0.4.380_2697 beta firmware and ran the MU-MIMO test suite. The plot below shows both test runs and the average of both MU and SU results.

MU vs. SU – ASUS RT-AC88U

Broadcom MU-MIMO is clearly not ready for use judging by these results. Not only is there a fairly rapid decline as SU STAs are added, there is modest MU throughput gain only in one of the two MU test runs. There may be other reasons to load up this beta release, but MU-MIMO performance isn’t one of them.

2×2 MU-MIMO

It looks like 2×2 MU radios are going to be common in high-end smartphones. So I ran the MU-MIMO test suite and simply switched the Veriwave to use 2×2 SU and MU STAs instead of 1×1. The results of two test runs on a QCA-based NETGEAR R7500v2 are not encouraging.

The 2×2 STAs yield 250 Mbps higher MU and SU starting throughput than 1×1 STAs. But there is again only one MU test with net throughput gain over SU. All the other tests produced lower MU throughput than SU. The only good news here is SU throughput is pretty well-behaved as STAs are added.

MU vs. SU – ASUS RT-AC88U

Closing Thoughts

Even though we’re in the second generation of MU-MIMO routers, the technology still has much maturing to do. Folks who have been developing MU-MIMO in mobile technologies know this is hellishly complex stuff; far beyond my simple mind’s ability to grasp much beyond its basic principles.

I have to say I’m not a believer that MU-MIMO will end up as a significant contributor to improving the lot of folks trying to tame the Wi-Fi beast. But for consumer router makers, it’s one more thing to slap on the box front in hope of luring desparate buyers.

Just like the non-MU beamforming that is part of 802.11ac, MU-MIMO will probably contribute something to better Wi-Fi performance down the road. But it will be years before it’s a significant contributor. 802.11ax will be here by then, giving Wi-Fi marketeers more buzzwords to use and driving poor Wi-Fi consumers even further into madness.

That said, if you simply must have a MU-MIMO router now, your best best for sure is one based on QCA’s MU-MIMO technology. And of those, NETGEAR’s R7800 Nighthawk X4S looks like it deserves its #1 spot in our AC2600 Router Ranking for MU-MIMO performance, too.

NETGEAR R7800 Nighthawk X4S

Discuss this in the Forums