Is MU-MIMO Ready For Prime Time?

Introduction

It’s still early days for MU-MIMO routers. Buyers of first-generation AC2350 / AC2400 class 4×4 "MU-MIMO ready" routers using Quantenna 5 GHz radios are still waiting for firmware to enable MU-MIMO. But most consumer router makers have moved on to produce a second round of 4×4 routers, using chipsets from Qualcomm Atheros (QCA) and more recently, Broadcom.

QCA-based AC2600 class routers have all shipped with working MU-MIMO from the get-go. Broadcom’s 4×4 routers are shipping as "MU-MIMO capable", with new firmware to turn on MU-MIMO promised by the end of this year. (Déjà vu, anyone?)

Not that all this really matters, because MU-MIMO devices are scarce as hen’s teeth. There are no MU-enabled tablets and only Acer has built it into some of its notebooks you can actually buy in the U.S. MU-enabled smartphones, which hold the most potential to drive MU-MIMO adoption, are China only for now and require a new phone in any event.

Despite limited availability and other gotchas and caveats listed in the sidebar above, eager-beaver early adopters are snapping up MU-MIMO routers of all flavors anyway. (It’s a tribute to consumer networking companies’ marketing troops that they’ve been so effective in getting people to buy (much) more expensive routers that provide little to no benefit to most users. Such is the power of those bigger numbers on the box!)

The Method

Anyway, since router makers are going to make ’em and gullible consumers are gonna buy ’em, SmallNetBuilder is on the job to test them. Our test process hasn’t been as good as we’d like it to be, since it has depended on manufacturers to loan MU-MIMO devices to test with.

Our first MU-MIMO router review of Linksys’ EA8500 used a trio of Dell Inspiron 13 7000 series laptops specially equipped with QCA reference design MU-MIMO adapters.

Since then, we’ve received three China-only Xiaomi Mi Note Pro smartphones for testing the TP-LINK Archer 2600, Amped Wireless RTA2600 and NETGEAR R7500v2 (review coming soon). But the Xiaomi phones are long overdue to be returned and we need a more permanent solution with better test capability.

This test marks the first appearance of VeriWave as a SmallNetBuilder test partner. VeriWave is a division of Ixia, the company that has long provided its IxChariot network performance software test tool to us. IxChariot forms the basis of virtually all our router and wireless performance testing, acting as our primary network traffic generation and measurement tool.

Ixia VeriWave is a precision hardware / software tool for Wi-Fi testing, fundamentally different from software only test tools like IxChariot. VeriWave uses hardware-based traffic generation and analysis, with the ability to track every packet end-to-end and provide fine control of transmission parameters. Specifically, Ixia has provided a WaveTest 20 three slot chassis loaded with RF46014 WaveBlade Wi-Fi, WBE1601 WaveBlade Ethernet and Management modules.

The Wi-Fi blade acts as the Wi-Fi device (STA), supporting 802.11a/b/g/n/ac. It has four RF ports, so can emulate any STA from an ancient 802.11b 1 Mbps slowpoke to a full 4×4 MU-MIMO 1733 Mbps speedster. Actually, the Wi-Fi blade can create up to 500 (!) fully independent stateful clients.

The Management blade is the controller for the two instrumentation blades, allowing the chassis to sit on a LAN and be accessed via a web interface for simple management chores. The Ethernet blade is a single-port Gigabit wire-speed traffic generator with full packet-capture capability.

The Wi-Fi blade is a complete VeriWave custom design; no store-bought radios from any of the usual suspects are used. The block diagram below from the WaveTest data sheet, shows the client stack architecture that provides the load with multiple independent stateful clients. Download the data sheet to dig into the blade specs.

VeriWave WaveTest Client Stack

The heavy lifting for testing is done by a suite of Windows applications. Each application takes the raw capability of the blades and configures it into easier-to-configure test scenarios.

The Test

With all this power at hand, I wanted to go beyond what I’ve been doing with three MU-MIMO devices. So, working with VeriWave, we came up with a MU-MIMO scaling test. Since up to 500 MU-MIMO STAs (or mix of SU, MU and any other type) can be created, we thought it would be interesting to see what happens when today’s MU-MIMO routers are loaded with four or more devices.

The custom test script VeriWave developed starts with a single MU 1×1 STA. The STA is configured to connect at the maximum MCS 9 index rate, 80 MHz bandwidth and short guard interval to achieve the maximum 433 Mbps link rate. Although the Wi-Fi WaveBlade can be configured to use any of the MIMO channel models (A-F), we kept things simple and used bypass mode.

The test connects the STA, then runs a TCP Goodput test, with MSS, Window and Frame size set so the link runs as fast as the device will allow. Each test runs for 30 seconds and throughput is recorded at the end of the test. We used a single TCP/IP session (connection) for each test.

NOTE: The TCP Goodput script is capable of much more. But we configured it so that it hits the device with traffic similar to how we use IxChariot’s throughput script for our other network throughput testing.

The test then adds another MU STA, verifies connection, repeats the 30 second test and records throughput for each STA. Tests were run by VeriWave with up to 100 STAs during development. But to keep test time manageable, I tested only up to 16 STAs. The results were still plenty interesting!

The test is repeated with the STA mode set to SU, to see how total throughput behaves with non MU devices.

The Wi-Fi WaveBlade normally can be either directly cabled to the device under test or use an antenna connection in an RF chamber. For MU-MIMO testing, however, the Wi-Fi WaveBlade and router must be cabled. This is needed so the WaveBlade can precisely control beamforming to ensure test consistency. Because the device was cabled and I have a completely quiet RF environment, I was able to simply connect the VeriWave chassis and router under test as shown in the photo below.

That’s a NETGEAR R7500v2 connected and ready to rock. 50 dB attenuators were needed between the router and Wi-Fi WaveBlade because it has a limited input range and would have been overloaded otherwise. Attenuators are usually required with direct-cabled testing because signal levels are abnormally high.

VeriWave chassis and router under test

I tested all four routers you can buy today with working MU-MIMO, shown in Table 1. All use QCA radios. I would have loved to include the Broadcom-based NETGEAR R8500 and ASUS RT-AC88U and RT-AC5300 in this test. Don’t think they dodged a bullet; when they ship working MU-MIMO, they’ll be tested!

The Results

Before we get to the results, let’s review what is supposed to happen. The diagram below shows three SU (non MU-MIMO) STAs take three transmit airtime periods.

Three STAs – SU

For three MU STAs, only one time period is required, ideally tripling effective bandwidth use.

Three STAs – MU

Switching back to SU, increasing the STAs to four, just takes up more airtime. Each must wait its turn.

Four STAs – SU

For four MU STAs, only two time periods are needed. Since two other STAs get time in the second slot, the AP is challenged to provide even throughput distribution for all clients. This challenge resulted in much lower overall throughput than the 3 STA case as we’ll see.

Four STAs – MU

Linksys EA8500

I’ll be showing an MU vs. SU throughput plot for each product tested. First up is the Linksys EA8500. It does pretty well, peaking at 730 Mbps for three devices. The fourth STA causes throughput to drop below the SU throughput line, which steadily declines as each STA is added. MU throughput then heads up and hits a lower peak at 7 STAs, then declines again.

Linksys EA8500 – MU vs. SU total throughput vs. STAs

NETGEAR R7500v2

The NETGEAR R7500v2 plot’s overall shape is similar to the Linksys’. But its maximum three-client peak is only 653 Mbps. It also hits throughput equal to SU mode with 14 STAs and stays with the SU line. In contrast, the Linksys looks like it still hasn’t settled down at the maximum 16 STAs tested.

NETGEAR R7500v2 – MU vs. SU total throughput vs. STAs

TP-LINK Archer C2600

Now we get to the interesting stuff. TP-LINK’s MU-MIMO router shows the usual three client peak at 688 Mbps. But what’s up with the SU line? Instead of starting around 340 Mbps like the Linksys and NETGEAR and slowly ramping down, it starts at 156 Mbps and ramps up. The more interesting result is at 12 STAs, where things basically stop working. The STAs still connect, they just aren’t passing much traffic at all.

TP-LINK Archer C2600 – MU vs. SU total throughput vs. STAs

Amped Wireless RTA2600

Bringing up the rear is Amped’s RTA2600. With SU STAs, it behaved like the Linksys and NETGEAR. But it obviously didn’t handle the MU clients well at all, producing total throughput worse than SU for two to 14 STAs.

Amped Wireless RTA2600 – MU vs. SU total throughput vs. STAs

All Products

So let’s put this all together. The composite plot below enables easier comparison of the four tested products, along with a handy average SU throughput line.

The Linksys EA8500, the first QCA-based MU router released in May and the most recently-released NETGEAR R7500v2 are the two best-behaved products in this test. Both produce similar maximum throughput with three STAs and manage to stay above the SU line (except for the four STA test) up to around 10 STAs.

The TP-LINK Archer C2600 starts out pretty good, but basically loses it at after 8 STAs and really loses it at 12 and above. Finally, the Amped Wireless RTA2600, is just a puzzle. Although it uses the same basic QCA design as the other three products, it obviously doesn’t react very well to our test conditions.

All products – MU total throughput vs. STAs

The composite SU plot isn’t as interesting, although it shows how closely three of the four routers behave.

All products – SU total throughput vs. STAs

To make sure a bad test run wasn’t at fault for the Amped’s poor results, two more full test passes were run for both MU and SU to produce the composite plot below. While there is run-to-run variation, the trend is pretty consistent.

Amped Wireless RTA2600 MU vs. SU repeatibility

There’s one other bit of insight the VeriWave system can provide into what might be going on. By using its packet capture ability, wireless frames can be examined and data extracted to see how many STAs are actually using MU frames. Ideally, MU devices should be using MU frames all the time, especially under the ideal conditions of this test.

STAs with MU Frames

The plot above shows this was not the case. The Linksys EA8500 and NETGEAR R7500v2 were able to work with the highest number of MU STAs – 11. They also happened to be the most well-behaved. The fact that the TP-LINK Archer C2600 stops using MU frames after 8 STAs could be releated to its drop to essentially no throughput with 12 and higher STAs connected.

Closing Thoughts

For most buyers of MU-MIMO routers, these results are academic. They have no MU devices, so aren’t really affected one way or another. Since the hardware is essentially the same for all these products, the difference is in firmware, which can be fixed well before MU devices ever hit their antennas.

But the exercise shows how far some manufacturers have to go in understanding this very complex new technology. And the work they have ahead to design products using it that provide user experiences that justify the high prices being asked.

For me, this has been more of a tryout of the VeriWave system, which looks like it will be able to significantly expand SmallNetBuilder’s wireless test capabilities. Look for lots more testing fun to come!

Discuss this in the Forums