How We Test Wi-Fi Systems – Revision 1

Introduction

Updated 8/6/17: Many changes to reflect process changes.

Since mesh / distributed Wi-Fi systems are here to stay, it’s time to get a better test process in place for them. So we’re replacing the open air method used in the Mesh Mashup and other previous reviews of Wi-Fi systems. The new process can handle products containing up to three nodes and can also test router / extender designs like NETGEAR’s Orbi.

The process consists of five sets of benchmarks:

• Wired routing
• Wi-Fi Throughput vs. attenuation
• Wi-Fi System
• Wi-Fi System Backhaul
• Wi-Fi System Capacity

Routing Performance

We run two benchmarks from the Version 10 router process. Please refer to that document for details. Note that since many Wi-Fi system products require a constant WAN connection, the wired routing tests use a switch between the product under test WAN port and the WAN-side testbed machines.

• 30 second Routing Throughput test (using iperf3)
• HTTP Score

Wi-Fi Throughput vs. Attenuation (RvR)

This benchmark tests how wireless throughput changes as path loss (signal level) changes between the product under test and a standard test client device.

Since most Wi-Fi System testing takes place in the small octoScope Box-18 test chambers described below, we perform throughput vs. attenuation testing there, too. The smaller chambers don’t have a turntable, so the device under test is not rotated for this test. The simplified setup also does not use the MPE-2 multipath emulator and uses unity-gain dipole antennas instead of the high-gain antennas used in our Revision 10 Wireless test process.

Wi-Fi Systems consist of multiple copies of essentially the same hardware that act as routers or access points. So we test only one module in these types of systems. For router / extender type systems, we run this test on the router module only using the dual-band Pal-245 as the station device.

The Pal is allowed to adjust all parameters automatically except for number of MIMO streams, which is set to two. This means this test uses a 2×2 802.11ac client that supports up to 256 QAM on both bands. Maximum 2.4 GHz link rate is 400 Mbps; maximum 5 GHz link rate is 867 Mbps.

Separate tests are run for downlink throughput and uplink throughput on the 2.4 and 5 GHz bands. Each test starts with 0 dB attenuation between the device under test and Pal client device and is increased in 3 dB steps to a maximum of 60 dB for 2.4 GHz or 48 dB for 5 GHz, until the connection is broken or 0 Mbps throughput is measured. Results are recorded as four benchmarks:

• 2.4 GHz Downlink
• 2.4 GHz Uplink
• 5 GHz Downlink
• 5 GHz Uplink

Wi-Fi System Test

These tests assess the performance of Wi-Fi Systems as systems. They require the new test configuration shown below. Each mesh node (AP) is placed in an RF-tight octoScope Box-18. Three Box-18’s are connected by five octoScope quadAttenuators and 4×4 MIMO 1:2 or 1:4 RF splitters.

Three of the quadAttens connect the Box-18’s to an octoScope Pal-24, Pal-5 and Pal-245 via an octoScope 1:4 splitter. Note all RF connections contain four separate signal paths. Although only two of those paths are used for the test STA, Wi-Fi Systems, NETGEAR’s Orbi specifically, may use up to four streams, which the testbed accommodates.

The Pals are octoScope’s partner devices that can function in AP, STA or Monitor mode and support up to 32 virtual STAs. Each is based on a Qualcomm Atheros chipset, has an onboard processor and runs both an iperf3 server and IxChariot endpoint.

The Pal-245 is a selectable dual-band 3×3 device that supports up to three transmit / receive chains, 20, 40 and 80 MHz bandwidth and 256 QAM on both bands. It does not support MU-MIMO or 80+80 or 160 MHz bandwidth.

The Pal-24 and Pal-5 are 2.4 and 5 GHz 4×4 devices, respectively, that support up to four transmit / receive chains, 20, 40, 80, 80+80 and 160 MHz channel bandwidth and 256 QAM. Pal-5 also supports 1×1 and 2×2 MU-MIMO.

Wi-Fi system test configuration

The Pal-245 is our test client (STA) for most tests. The quadAttens set path loss to simulate distance between the STA and each AP. The other two quadAttens set the box-to-box path loss to simulate distance between APs.

Box-18 Root has two gigabit Ethernet connections. One is used to connect the root mesh node to the internet, which most mesh products require for connection to their companion cloud service for proper operation. The second Ethernet connection goes to the root mesh node’s LAN connection, which is where the test traffic generator is connected.

The dashed lines going to the other mesh nodes indicate optional connections for their LAN ports, for backhaul / wireless bridge throughput tests.

The smaller Box-18’s make it possible to fit a lot of test flexibility into the same space occupied by our previous testbed. The new stack, shown below, handles throughput vs. attenuation (rate vs. range) testing in the top half and Wi-Fi System testing in the bottom.

SmallNetBuilder Wireless Testbed

The octoScope TS-MESH module sits with the Box-18’s and holds the quadAttens, Pal and PoE gigabit switches used to provide control and power to them. It also contains the splitters that provide RF connections.

The inside of the Box-18 is pretty tight, so unity-gain dual-band dipole antennas are used, mounted as shown below. For low-profile products, the antennas are angled at 45 degrees as shown; for taller products, they are set straight up. This setup provides typical beacon RSSIs reported by the Pal devices of -30 dBm for 2.4 GHz and -45 dBm for 5 GHz with attenuators set to 0 dB.

Box 18 inside

Wi-Fi System Performance

Note: "Downlink" means data flows from mesh node (AP) to client device. "Uplink" means data flows from client device to mesh node.

Multi-node Wi-Fi systems are generally designed to be in control and provide few user-settable controls. These products use a single SSID and usually don’t allow channels, bandwidth or any other Wi-Fi parameters to be set, other than the network name (SSID) and encryption passphrase. So our process lets each system run as it normally would, including having a live internet connection during testing. All test traffic is run on the LAN side, however, so router throughput does not affect results.

Products are installed using their companion app or web interface. Attenuators are first set as shown in the diagram below. The root node is then installed in the Root Box-18, with the door open so that it can communicate with the smartphone running its app via Bluetooth. Once the root node is connected, the chamber door is closed. The other nodes are then added, one at a time, following the same process. If the app prompts for a firmware update, it is run until the app says all nodes are up to date. No other product settings are changed for testing.

Wi-Fi System test diagram

After all nodes are connected and chamber doors are closed, all nodes are power cycled, with attenuators set as shown above. This lets the system come up and establish operating parameters in the exact test environment.

iperf3 is used for each throughput test with TCP/IP and default parameters, except window size is set to 2048 kb and two test streams are used. Each test is run for 90 seconds, with the first 10 seconds treated as "training" period. The "training" period data is not included in the calculated average for the rest of the test run.

The goal of the process is to measure best-case average up and downlink throughput for both bands on each node. The general test process is:

1. Associate Pal client device to the root node 2.4 GHz radio, run down and uplink throughput tests
2. Associate Pal client device to the root node 5 GHz radio, run down and uplink throughput tests
3. Repeat for the one-hop and two-hop test nodes.

Attenuators 1 and 2 provide the RF paths so the Root, Hop 1 and Hop 2 nodes can form their backhaul links. The attenuation values shown are relatively low, because the path loss provided by air gap between the node and chamber antennas provides most of the loss. Path loss from the Hop 1 node to Root and Hop 2 nodes has been measured to be typically 40 dB in 2.4 GHz and 65 dB in 5 GHz using the 3 dB values shown for Attenuators 1 and 2. The goal here is to allow the branch nodes to have a signal level high enough to not cause the product to indicate that a node needs to be moved.

Testing is controlled by a script that executes the following test plan (reference above diagram):

1. Set the attenuators as shown in diagram above
2. Associate Pal to Root node device on 2.4 GHz. Run downlink throughput test. Run uplink throughput test
3. Associate Pal to Root node device on 5 GHz. Run downlink throughput test. Run uplink throughput test
4. Disassociate Pal from Root node (set Atten 5 to 63 dB). Move Pal to Hop 1 device (set Atten 4 to 0 dB).
5. Repeat Steps 2 and 3
6. Disassociate Pal from Hop 1 node (set Atten 4 to 63 dB). Move Pal to Hop 2 device (set Atten 3 to 0 dB).
7. Repeat Steps 2 and 3

Results are recorded as four benchmarks. Each benchmark contains the values for Root, Hop 1 and Hop 2 nodes.

• 2.4 GHz System Downlink throughput
• 2.4 GHz System Uplink throughput
• 5 GHz System Downlink throughput
• 5 GHz System Uplink throughput.

Wi-Fi System Performance – Extender

Some Wi-Fi systems, NETGEAR’s Orbi most notably, use router / extender architectures. These systems require each extender to connect only to a base router, i.e. extenders cannot be daisy-chained. These two-module systems require a slightly modified test plan.

Instead of testing a second Hop, a second set of tests is run on the Hop 1 module with the Pal-connect attenuator changed from 0 dB to 21 dB. This simulates the test client moved farther away from the extender / satellite.

The router module is installed in the Root Box-18; the Extender / Satellite is installed in the Hop 1 Box-18.

1. Set the attenuators as shown in diagram above
2. Associate Pal to Root node device on 2.4 GHz. Run downlink throughput test. Run uplink throughput test
3. Associate Pal to Root node device on 5 GHz. Run downlink throughput test. Run uplink throughput test
4. Disassociate Pal from Root node (set Atten 5 to 63 dB). Move Pal to Hop 1 device (set Atten 4 to 0 dB).
5. Repeat Steps 2 and 3
6. Change Hop 1 Pal-connect attenuator from 0 to 21 dB.
7. Repeat Steps 2 and 3

Wi-Fi System Performance – Backhaul

This test measures the bandwidth from the Root node Ethernet-connected iperf3 endpoint to an Ethernet-connected iperf3 endpoint connected to Hop 1, then Hop 2. This measures the bandwidth available at each node for client and backhaul connections.

This test uses a second computer running an iperf3 server, connected via Ethernet to each node, one at a time.

1. Confirm that all nodes are operating normally.
2. Connect iperf3 server to Ethernet port of Hop 1 node device. Run downlink throughput test. Run uplink throughput test.
3. Move iperf3 server from Hop 1 node device to Ethernet port of Hop 2 node device. Run downlink throughput test. Run uplink throughput test.

Results are recorded as four benchmarks:

• Hop 1 Backhaul down
• Hop 1 Backhaul up
• Hop 2 Backhaul down
• Hop 2 Backhaul up

Wi-Fi System Performance – Capacity

The test connects a single STA to each mesh node and runs simultaneous throughput tests to all three STAs. It measures how well overall system throughput is managed in a simple loading scenario. Average throughput is recorded for each STA. Only client throughput is measured, but this test also assesses how well or whether backhaul throughput is managed during the test. All test clients are 2×2.

1. Connect Pal-24 to Root Node, set to 2.4 GHz
2. Connect Pal-5 to Hop 1 node (5 GHz)
3. Connect Pal-245 to Hop 2 node (5 GHz). In router / extender systems, this client will also be connected to Hop 1, i.e. the extender.
4. Configure iperf3 for TCP/IP, two streams, 2048 kb window size,
5. Run 30 minute test with the first 30 seconds not included in the averages reported

The test setup looks like this. Splitters have been removed for clarity. Remember, as described above, box-to-box path loss is typically 40 dB in 2.4 GHz and 65 dB in 5 GHz, which includes the 3 dB box-to-box attenuators shown.

Mesh capacity test setup

Results are recorded in two benchmarks:

• Wi-Fi System Capacity – Downlink
• Wi-Fi System Capacity – Uplink

Each benchmark is the total average of the throughput of the three test clients. Per-client throughput is also available by plotting the benchmarks.

Discuss this in the Forums