Wireless devices are becoming increasingly common in the networking landscape. Yet unlike most networking devices, wireless products offer no accurate method for users and implementers to gauge their performance. To complicate this matter, a device's environment can have a huge impact on device performance. Unfortunately, existing network benchmarks are not sufficient to test wireless devices. They presume that any performance deficiencies are the result of the device under test (DUT). With wireless networks, however, this is clearly not the case.

This article is the second of a three-part series that focuses on WLAN test methodologies. The first article addressed the value of WLAN performance testing. The last article will examine the testing of IEEE 802.11g as provided by the CENTAUR Lab at the University of Georgia. It is the job of this article to delve into benchmarking.

The goal of any good network benchmark is to provide a repeatable, fair, and quantitative comparison between devices. If vendors, magazines, and customers all used different tests with varying results, then the whole point of testing—which is to make comparisons—is missed.

The most widely used metric for measuring network-device performance is frames or packets per second. Others, such as latency and jitter, also are used. The performance metric in which people seem to be the most interested, however, is the actual forwarding limit of the device before data is dropped.

Additionally, every network technology has an absolute forwarding limit based on signaling rate, packet size, and technology overhead. For Ethernet, these values are well known and understood. Most reputable vendors produce devices that are capable of performing at 100% of these maximum rates (see table).

Although the forwarding limit is an important performance metric, there is much more to benchmarking a network device. The Internet Engineering Task Force (IETF) has developed and documented basic procedures for characterizing network performance in RFC 2544: Benchmarking Methodology for Network Interconnection Devices. This document contains setup and configuration information as well as six tests for benchmarking network devices. In addition, the document recommends frame sizes, test duration times, and traffic burst patterns to be used in the benchmark tests. Of the six tests described, four are relevant to performance benchmarking: throughput, latency, frame loss rates, and back to back. They are described in more detail below:

• Throughput: Throughput is defined as the maximum rate at which a device can forward traffic without losing data. Throughput tests are normally conducted at a fixed frame size for a specified amount of time. If the device fails to pass all of the transmitted frames in the allocated time, the test is repeated at a lower frame rate. If the device succeeds, the test is repeated with a higher rate.
• Latency: The latency of a device is defined as the amount of time that it takes a device to forward a frame from the input port to the output port. For store and forward devices, which account for almost all modern bridges, this is the difference in time between a device receiving and transmitting the final bit of a frame. This value can vary greatly depending on the load and architecture of the device being tested. The variance is known as jitter.
• Frame loss rate: The frame loss rate is defined as the rate at which packets are dropped for a given frame size and rate. The frame loss rate of a device should be 0 frames/s up to and including the measured throughput value.
• Back to back: A back-to-back test measures the DUT's ability to forward packets with a minimum interpacket gap. The minimum interpacket gap is the smallest amount of time that a device must wait before it can transmit a subsequent data frame. The result of this test is the maximum number of minimum interpacket gap frames that the device can forward without loss. Line-rate devices have a back-to-back value of infinity.

Although this subset of tests does not characterize every conceivable aspect of a device, the results of these tests will give a good indication of a device's basic forwarding abilities. In addition, many more tests can build on the foundation provided by these simple tests.

Wireless 802.11 networks and Ethernet share many similarities. Both have similar frame formats, common 48-b media-access-control (MAC) addresses, and the ability to transmit a range of frame sizes. Plus, devices on both networks use the Carrier Sense Multiple Access/Collision Avoidance (CSMA/CA) scheme to determine when other devices are using the transmission medium. The existing network-device benchmark methodologies could be used for the foundation of WLAN testing. The WLAN-network-layer performance-affecting features, however, will clearly impact performance. Specifically, these features include acknowledgements, retransmissions, and rate shifting. IEEE 802.11-specific benchmark methodologies must take these factors into account.

In a wired environment, the impact of electrical signals or light pulses is not an issue for the transmitted signal across the physical link. The wired-network medium that conducts the signals is mostly constant. But the environment of a wireless network is much more variable. Thus, the 802.11 specification requires that any wireless station that receives a frame must acknowledge that frame by sending a 14-B frame back to the transmitter. This acknowledgement, or ACK, can only acknowledge one frame. It contains no information about which frame it is acknowledging. As a result, a wireless station must wait for an acknowledgement after every transmitted frame before resuming transmission.