January 2006 doc IEEE 802 11 -05 1259 r

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Over the Air Field Testing of 802. 11 Systems Date: 2005 -12 -15 Authors: Pertti Visuri Oleg Abramov Airgain, Inc 5355 Ave Encinas, Carlsbad, CA 92008 760 597 0200 pvisuri@airgain. com 760 597 0200 oabramov@airgain. com Notice: This document has been prepared to assist IEEE 802. 11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802. 11. Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard. " Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802. 11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at . Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Abstract This paper is a presentation of research on a test methodology for over the air testing that will be presented in the January 2006 meeting of the 802. 11 task group T. The formal proposal and methodology submission will follow more closely the guidelines of doc. : IEEE 802. 11 -04/1553 r 3. The purpose of this paper is to introduce the issues to let participants prepare for a discussion. The effect of local signal strength variations on over the air performance measurements of 802. 11 systems is discussed and a solution for obtaining reliable results is presented. Local variations are caused by multipath fading and can result in 15 d. B signal strength variations across a few centimeters displacement of the antenna in either end of a wireless link. These variations can not be eliminated by maintaining fixed locations if the systems in the test use different types of antennas. Statistical methods can be used to obtain accurate results in field tests and to calculate confidence limits for the results. The nature of variations in over the air testing is examined and examples of statistically correct tests are presented. An automated method using a turn table for collecting data to obtain statistically significant results is presented for both signal strength and for throughput tests. The bias effect of continuous motion on throughput tests is explained and results from an automated stop-motion turntable test system that eliminates the effect are presented. A practical methodology for evaluating over the air performance and obtaining reliable and repeatable results is presented Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Tests Are Usually Done in Controlled Environments • Testing of wireless components is usually done in controlled environments where the effect of a single component can be seen. – Wireless chipsets and system functions are tested with the signal contained in a coaxial cable and in the test system using sophisticated simulations of real world environment – Antennas are tested in anechoic chambers that eliminate the effects of reflections and multipath variations. • • These tests can provide precise results of component or subsystem performance directly since changes in environmental factors are controlled and do not affect the test. However, they can not be used to establish performance of complete systems, especially when antennas and the wireless over the air propagation are key elements of the system To include the effect of antennas field tests in real conditions are needed In field tests there are variations that can not be controlled Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Natural Signal Variations in Field Tests • All signals received by a wireless unit have two kinds of variations: – They fluctuate with time – They vary with small changes of location or environment • The time fluctuations can be compensated for relatively easily by performing the test for a sufficiently long period with repeated sampling and averaging the results. – The issues of how long time of averaging is necessary to achieve desired accuracy will be addressed later in this presentation. • However, the situation is more complex regarding local variations. Two examples of wireless 802. 11 g signal strength variations over a 30 second time interval 9 d. B 6 d. B 30 s Submission 30 s Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Signal Variation with Antenna Location • To measure the effect of local signal variations an access point with a standard dipole antenna was moved over a grid of 100 locations and the signal strength was measured in each location • The client station connected to the access point was about 40 m (120 feet) away in a non-line of sight location • The client station was not moved at all during the test • The signal strength was measured using the RSSI reporting feature of an 802. 11 radio card and averaged over hundreds of samples during a few minutes to even out the effect of time fluctuations Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Signal Variation with Antenna Location 15 d. B • Major differences in signal strength of about 15 d. B were observed within 5 cm (2 inches) of one another • The variation pattern illustrated here was measured using a regular dipole antenna Dipole Signal Strength s) 50 Submission cm (20 he nc i Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Signal Variation with Antenna Location • The access point dipole was replaced by a smart antenna that provides a benefit of 3 to 4 d. B in an anechoic chamber test. Nothing else was changed. • The local signal strength variation pattern is different: 12 d. B – The level is about 4 d. B higher – The peaks and valleys are in different locations – The range of variation is 3 d. B smaller Smart antenna Signal Strength Expected Smart Antenna benefit s) 50 Submission cm (20 he nc i Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Signal variation with Access point location • Since the local variation patterns are different for different antenna designs, it is not possible to eliminate the effect of local variations by placing the antenna in exactly same location for comparison tests. • In fact, there is no such concept as the “exactly same location” when the antenna designs are fundamentally different Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Moving Client Station Instead of Access Point • Similar differences in local signal strength variation patterns are observed when the access point is kept stationary and the client station is moved over a 50 by 50 cm (20 by 20 inch) grid • Clearly the signal strength depends on the exact location of both ends of a wireless link AP Client Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Comparing Antenna Solutions • Comparing the signal strength of one antenna to another in each location in the test shown on the previous slide gives values for the difference ranging from 0 d. B to 20 d. B • When performance of one antenna is compared to a different kind antenna the result will have great variations if the test is done only in one Signal Strength location of the antenna-under-test and the Difference between a Smart unit at the other end of the link antenna and a dipole antenna • These variations need to be understood and compensated for to get meaningful results (Smart Antenna signal ) Submission minus (Dipole signal ) equals Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Nature of the local variations • Testing in several different kinds of environments reveals that the local signal strength variations are smaller outdoors and seem to be larger and more closely spaced in highly reflective indoor settings. • The full scale of variation seems to take place within about 50 cm (20 inches) for the 2. 4 GHz frequency, which corresponds to about 4 wavelengths Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Handling Variations to Obtain Accurate Results • • Obtaining accurate and meaningful test results when there are variations caused by uncontrolled variables is a well established discipline Many industries, for example the entire pharmaceutical industry, depend on decisions based on tests where conclusions are based on calculated confidence limits. The same tools are available for testing of over the air wireless communications The distribution of variations in any set of measurements that are affected by a number of independent, uncontrolled factors is likely to be close to a normal distribution. The probability of occurrence of a certain result can be estimated for normally distributed values using their average and the standard deviation estimated from the set of values The standard deviation can be estimated for any set of values by calculating the root mean In a normal distribution the percentage of square deviation of the actual values from samples that fall within one standard their average deviation of the average value is 68%. Over 95% of samples fall between two Based on this it is possible to calculate standard deviations from the average confidence limits for any set of measurements Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Confidence Limits for Field Testing • • • Confidence limits establish the boundaries within which an average calculated from a set of measurements is from the actual average (of an infinite set of tests). The values of the confidence limits for a set of tests depend on – The standard deviation of the results, – Number of measurements taken and Distribution of the d. B difference of signal strength between a smart antenna and a dipole antenna in – The selected level of confidence (the probability that the actual average 500 comparison tests compared to the shape of a normal distribution curve is between the calculated limits) The formula for calculating the limits is: = +/-t (M, C)*SQRT( 2/M) , where is the standard deviation, M is the number of measurements, C is the desired confidence level and t (M, C) is the so called “Student’s coefficient”. – There is a table of values for Student’s coefficient in Appendix 1 – For more than 15 measurements and for the confidence limit of 95% the value of t (M>15, @95%) is approximately 2 Submission Shape of normal distribution Average 4. 0 d. B Measured distribution of signal strength difference Standard Deviation 4. 6 d. B Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Confidence Limits for Field Testing • • As pointed out earlier, the nature of the local signal strength variation depends on the physical environment and on the antenna system. Typical values for standard deviation in indoor tests range from 3 d. B to 5. 5 d. B for a dipole antenna and from 2 d. B to 3. 2 d. B for a smart antenna that was tested in the same environments Assuming a standard deviation of 3. 6 d. B allows an illustration of how many measurements are necessary for various levels of accuracy and percentage confidence. For example: – – – • to obtain a 95% confidence that the actual average value is within 1. 2 d. B of the measured average it will be necessary to make 36 measurements. However, if a 80% confidence of the same 1. 2 d. B limits is sufficient then only 15 measurements are needed. To obtain an accuracy of 0. 6 d. B at 90% certainty about 100 measurement would be needed. Confidence limits for different % confidence levels and numbers of measurements taken in an indoor test of signal strength (standard deviation 3. 6 d. B) These levels of accuracy are similar to the levels generally quoted for standard anechoic chamber testing Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Confidence Limits for Only a Few Measurements • The same calculation of confidence limits can be applied to examine the level of confidence in conclusions drawn from tests that only include measurements at only very few locations of the antenna-under-test. First let us assume that we know the standard deviation of the test results in the environment based on other test results – – – • If only one measurement is taken, the level of accuracy at 95% confidence level is only + 7 d. B Increasing the number of locations in which the measurements are taken to five improves the accuracy to + 3. 2 d. B However, even these confidence limits are still of the same order as the effects that often are being evaluated in field tests. Confidence limits for different % confidence levels and numbers of measurements taken in an indoor test of signal strength (standard deviation 3. 6 d. B) St. Dev. Not known In a new environment, where the standard deviation is not known, the confidence limits for few measurements are even wider since the standard deviation needs to be estimated from the same test results. – In these cases the actual value of the Student’s coefficient needs to be applied to estimate the confidence limits. For example, for two measurements the limits are +30 d. B, for five measurements they would be +5. 5 + 2. 6 and for ten measurements - Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Example of Testing in Different Environments • • • Comparison tests of signal strength were performed in five different indoor environments. Two residential, two different office buildings and one laboratory setting. 100 measurements in a 50 x 50 cm (20 x 20 inch) grid were taken at each location for each of the antennas. The test arrangement is illustrated on slide number 6. This graph shows the average signal strengths at each location and the associated 95% confidence limits for each result calculated according to the formula on slide 14. Since the standard deviation is different at each location the confidence limits are also different for each data point It is appears that the signal strength difference between the Smart Antenna two antennas depends on the Dipole local environment as well as the specific test location. Therefore for drawing general conclusions about over the air performance of wireless systems it is necessary to perform several measurements in many different environments Residential 1 Submission Small office Lab space Residential 2 Large office Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Data Distributions and Confidence Limits Residential 1 5. 8 + 0. 5 d. B - - 1. 1 d. B This slide displays the actual distribution of measurement results+around the average of the d Small office 5. 5 + 0. 6 d. B + 0. 7 d. B - Residential 2 4. 3 + 0. 4 d. B - 2. 3 + 0. 6 d. B Lab space + 0. 7 d. B - + 0. 6 d. B - Large office 2. 1 +0. 5 d. B - + 0. 6 d. B - Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Variation between Local Environments • It appears from results on slides 17 and 18 that, in addition to the local variations within a few wavelengths, there is second factor that causes variation in the performance of different wireless systems: – There seem to be differences in performance in different building environments. These could be caused for example by the nature of multipath effects in different environments and the responses of the wireless systems to these effects Comparison tests of two antennas in five locations indicate a second level effect: – It is appears that a second variation effect is present. In this case the residential and small office environments have a higher difference between the antennas than the large office and lab space. Both of the latter have several metal partitions and furniture whereas the residential and small office have plasterboard walls and ordinary furniture. – The 95% confidence limits based on 100 measurements in each environment are shown. – However, tests in five different environments are not sufficient to prove the effect of the environment. The standard deviation for variation between different environments can be estimated from the five values. It is 1. 7 d. B. – Based on this the 95% confidence limits for the overall performance in different environments are + 2. 2 d. B – More tests would be needed to prove the effect Submission Difference in signal strength between a smart antenna and a dipole in different environments Signal strength difference d. B • Residential 1 Small office Residential 2 Large office Lab space Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Combining Results from Different Environments • • If the various test environments used in evaluating the performance represent the intended environments for using the systems, and the number of tests in each correspond to the pattern of usage, it is meaningful to combine the results to draw general conclusions One way is to combine is to look at the distribution around the average signal strength at each location and graph the distribution normalized to the average of one of the antennas Another way is to look at the distribution of the difference as compared at each measurement location for each of the test points. Both methods are shown below. The confidence limits used for this combined average should be based on the limit calculated using the five measurement results from the five different environments Distributions of normalized local signal strengths number of measurement results at each interval 3. 97 d. B+ 2. 2 d. B - 0 d. B +2. 2 d. B - Smart Antenna Dipole Normalized signal strength d. B Submission Distribution of local signal strength differences 3. 97 d. B + 2. 2 d. B - Smart Antenna signal strength improvement over Dipole Average Signal strength difference d. B Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Confidence Limits for Time Fluctuations • • • As pointed out earlier, the signal strength in 802. 11 systems also fluctuates with time. These variations can be handled the same way as the variations over locations Time variations are smaller than the local variations. Their standard deviation is typically 1. 2 d. B. Therefore their contribution to test result uncertainty is also smaller With one hundred test points the 95%confidence limits for time fluctuations are at about 0. 25 d. B. With 200 ms sampling interval this means a 20 second test at each location. It is simple to increase the test time and include more samples to the averaging Averaging the signal strength further across a hundred locations increases the total number of samples to 10, 000 and reduces the time-related confidence limits to about 0. 02 d. B An example of wireless 802. 11 g signal strength variations over a 30 second time interval Confidence limits for different % confidence levels and numbers of samples (standard deviation 1. 2 d. B) 6 d. B 30 s Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Using a Turn-table to Average Over Local variations • • • To develop an automated method for collecting signal strength measurement results in different locations tests were performed using a rotating turn-table at one end of the link and averaging the measured values The turn table motion converts the local variation to time domain and makes it more convenient to calculate the average of the signal strength The turntable also changes the orientation of the antenna, but its main benefit is that it helps to average over local signal strength variations The observed strength variations reflect the 15 to 20 d. B local variations that were observed in the earlier tests The expectation is that averaging over local variations at one end of the link will reduce the local variations at the other end One full rotation Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 The Effect of Turn Table in One End of the Link • • • Placing the client station on a turn table and averaging over at least one rotation reduced the range of variation caused by small changes in access point location by 50% (from 15 d. B to about 8 d. B) The standard deviation of signal strength across the different locations was reduced from 3. 15 d. B to 1. 50 d. B. This corresponds to a similar reduction of confidence limits However, it is important to note that having a turntable in one end of a test is not enough to eliminate the variation. Several test locations for the other end are needed. – Even with the turn table averaging the results at one end of the link the result for only one location of the antenna-under-test will still have confidence limits of about +3 d. B – Ten different locations of the antennaunder-test need to be measured with the other end of the link on a turn table to reduce the 95% confidence limits to about +1 d. B – In addition, the variation across different environments, discussed on slides 17 to 19 needs to be considered. – Assuming that the building environment has an impact on performance, it would be necessary to perform tests at ten different locations with a turntable at one end of the link in each of the building + environments where the performance is to be characterized. Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Effect of Motion in Signal Strength Tests When one or both ends of the wireless link are physically in motion that would not occur in normal use it is important to consider the effect of the motion to the test result – Each signal strength test that uses the built in function of a wireless LAN card takes place during the packet preamble and takes less than a millisecond. – If the turn table turns at 2 rpm the client moves less than 0. 05 mm during the test. This would not have any effect on the value and the measurements can easily follow the variations in signal strength as can be seen from the test graphs below – However, if there are other dynamic control functions involved it is necessary to consider their time constants as well. For example, a smart antenna system may be affected by the rapid, exaggerated signal strength changes caused by the turn table motion Client stationary Client rotating 2 rpm 45 45 Signal Strength d. B • 35 30 Submission 30 25 25 0 35 10 Time (sec) 20 30 0 10 Time (sec) 20 30 Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Over the Air Throughput Tests • • Since the 802. 11 systems all adjust the data encoding complexity automatically to adapt to different link quality levels, the throughput of a particular link has a strong correlation of the signal quality of the link Consequently the same local variation issues that were discussed in previous slides regarding signal strength will be present in throughput tests, too. Therefore the general conclusions about confidence levels and the need for a number of different test locations to achieve reliable results are valid for throughput tests also Because of the effects of different encoding mechanisms in 802. 11 the real life throughput is an Scurve as a function of link quality Submission Correlation of throughput to signal strength Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Local Variations in Throughput Tests • • • Since the 802. 11 systems all adjust data rates automatically to adapt to different link quality levels, the same local variation issues that were discussed in previous slides regarding signal strength will be present in throughput tests Generally the correlation of throughput with signal level is quite high and the shape and nature of the local throughput variations is very similar to the variations in signal strength Therefore the general conclusions about confidence levels and the need for a number of different test locations to achieve reliable results are valid for throughput tests also Correlation of throughput to signal strength measured by the radio card Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Effect of Client Motion in Throughput Tests • Even though motion may not affect signal strength tests, it can have a significant effect in throughput testing depending on the chipset implementation and the speed of the table – The transmission data rate algorithms in many WLAN chipsets use packet error rate (PER) as the main input parameter for rate setting. The PER is measured over a period of time – The rate setting may be affected by the variations caused by the motion on a rotating turn table which would not be representative of real use situations. – At least some chipsets are greatly affected if the speed of a turn table is higher than 0. 5 turns/minute. It is therefore important to verify that the speed does not have an effect on the tests if a continuous motion turntable is used in the tests. – An ideal way of collecting throughput data for different locations is to use a stop-motion turn table that moves the unit through several locations, but keeps it stationary during the actual tests Effect of Client rotating 1 to 2 rpm Client rotating 2 rpm Signal Strength d. B 45 35 30 25 0 Submission 10 20 30 Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Using a Stop Motion Turn Table • • To study the distribution of throughput performance across small location variations a test was performed using an automated turn table that turned 10 degrees, stopped, ran a standard 30 second Chariot throughput test, and then proceeded the next 10 degrees and stopped for the next test, etc. In these tests the standard deviation Distributions of test results in a stop-motion for the dipole antenna was 2. 6 Mbits/s turntable throughput test for two antennas and for the smart antenna 1. 5 Mbits/s This test was performed at close to maximum throughput. However, even Smart Antenna at this level the better signal strength results in higher throughput. 26. 4 d. B +0. 35 number of test results at each throughput level • - 24. 9 d. B + 0. 6 d. B - Dipole Throughput Mbits/s Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Throughput tests on Stop motion turntable • • To develop a method for throughput testing a set of measurements was taken for nine different wireless access point and client locations in an office building. The stop motion turntable with 18 stops/measurements was used in each location. The results appear meaningful, but can not be used for quantitative analysis as each test point represents only one antenna location only and the variations between locations are large Dipole – The average throughput and the 95% confidence limits are shown for each antenna – As was demonstrated on slide 23, it is necessary to average test results from more than one gateway location to obtain representative results and more definitive conclusions Smart Antenna 3 8 6 6 4 Displayed floor plan is similar but not the actual site (for security reasons) Submission 5 5 2 9 7 1 Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Averaging across more than one location • • Grouping the results in three zones based on throughput level allows meaningful averaging This way at least two different gateway locations are used for each average. This is less than recommended, but can be used here to illustrate the basic method – The results were combined into three groups based on the throughput level of the dipole – Average of all measured throughput differences between the two antennas was calculated for each of the three zones of throughput level Smart Antenna High Dipole Medium Low 3 8 6 5 4 5 6 2 9 7 1 Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Definitive throughput results • Actual difference in throughput Smart Antenna over dipole 6. 7 + 2. 1 Mbits/s - 2. 3 +1. 2 Mbits/s 1. 1 + 0. 7 Mbits/s - 20 to 30 10 to 20 0 to 10 High Low Medium Throughput range (Mbits/s) Submission Throughput difference (%) • By combining the throughput comparison results into three groups enough measurements are available in each group to achieve definitive conclusions at 95% confidence level The results are as expected based on the results of signal strength differences ( slides 17 to 19) and the known relationship between signal strength and throughput (shown on slide 25) Increasing the number of test locations in each zone would narrow the confidence limits further. Ten locations in each zone should be included. This would narrow the limits to half of the ones shown here Throughput difference (Mbits/s) • Percentage difference in throughput Smart Antenna over dipole 92 +46% - 35 +11% - 4. 5 +2. 6% 20 to 30 10 to 20 0 to 10 High Medium Low Throughput range (Mbits/s) Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Conclusions Local signal strength variations have a major effect on over the air performance measurements of 802. 11 systems. They cause single tests to provide results that can vary arbitrarily up to 15 d. B in signal strength or over 10 Mbits/s in throughput in apparently identical tests. Statistical methods, that are very well known and used in several other industries, can be used to obtain accurate results in field tests and to calculate confidence limits for the results. An automated method can be used to conveniently obtain enough data for accurate and reliable results from over the air performance testing. Both ends of a wireless link need to be placed in several different locations and the results averaged in suitable groups to get representative results. Testing can be partially automated by using a turn table. However, in certain situations continuous motion may introduce a bias effect. This can be eliminated by a stop-motion test system that is stationary during the actual test. Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 1 – Student’s Coefficients 0 Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Introduction to the Test Method • • • As has been demonstrated in the main presentation, local signal strength variations caused by multipath fading cause random variations in any over the air testing of wireless systems. The only way to overcome the uncontrolled variations is to perform several measurements at different locations of both ends of the wireless link. Even keeping the test environment completely unchanged will not help if the systems to be tested have different kinds of antennas. There is no such concept as “the same location” for devices with different antennas. This appendix presents a recommended practical method for obtaining reliable and repeatable test results. The recommendations include numbers of measurements to take and methods to average the results and to calculate confidence limits for them Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Taking Measurements at Several Locations • • • The key to all testing is to take measurements at several locations. Both ends of the wireless link must be tested in many locations. It is sufficient to move the devices only a few centimeters to average over the multipath fading variation. However, to obtain results that are representative of the overall performance in a particular building environment, it is better to include measurements made at different parts of the building. The measurements should be grouped so that meaningful averages can be calculated. – For example, to evaluate signal strength differences all results in a particular building environment can be averaged together, since the signal strength difference is not expected to depend on the overall level of the signal – For evaluating throughput differences caused by better signal the results need to be grouped by the level of the 4 throughput, since the expected improvement depends on the 2 5 6 overall throughput level 3 8 6 5 9 7 1 Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Rule of Thumb for How Many Measurements are Necessary • • The range of multipath variations in signal strength of 802. 11 b/g systems tend to be about 15 d. B. Typical values for standard deviation appear to be between 2 and 6 d. B. The corresponding numbers for throughput variation depend on the actual level of throughput, but are similar to these numbers in the mid throughput range. Based on this general information we can roughly estimate the number of tests needed for various levels of accuracy and confidence. The sample calculation from slide 16 is below – For example, to evaluate signal strength differences with better than +1 d. B accuracy at 95% confidence, 50 to 150 individual measurements will be needed for each device to be compared – These measurements should include at least ten different locations for both ends of the wireless link that is to be tested – A practical way to obtain such data is to set one end of the link on a stop-motion turn table and program it to collect the data at 10 to 20 locations along the full turn. This automated test setup should then be operated in ten locations in the building and the results averaged together Submission Confidence limits for different % confidence levels and numbers of measurements taken in an indoor test of signal strength (standard deviation 3. 6 d. B) Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Using Zone-Averaging for Throughput Tests • • If the nature of the difference in performance that is to be evaluated is dependent of another known quantity, the measurements should be grouped so that only test results that are similar in nature averaged together For example, throughput improvements from better signal strength are dependent of the original throughout. Hence the test results for throughput improvement need to be averaged together in at least three different zones of original throughput. – In a practical test plan, selecting the respective locations for the access point and for the stop-motion turn table in the various tests should be done so that all different throughput levels are represented. – As an example, ten locations and 18 stops for measurements at each location will achieve an approximate accuracy of + 20% on the improvement at 95% confidence level. – Only four locations in each zone would provide an approximate accuracy level of + 50% of the resulting average values Submission High Medium Low Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Calculating Confidence Limits for Tests • • The confidence limits on slides 36 and 37 are just examples to indicate typical ranges. The actual confidence limits for the average of any set of measurement results can be easily calculated from the results using the two formulas given below For practical purposes Microsoft Excel spreadsheet has convenient pre-defined functions for calculating both standard deviations and confidence limits for sets that include at least 10 results or where the standard deviation is known. The names and formats for these functions are =STDEV() and =CONFIDENCE(<1 -confidence level>, , ) = +/-t (M, C)*SQRT( 2/M) , where is the confidence limit, is the standard deviation, M is the number of measurements, C is the desired confidence level and t (M, C) is the so called “Student’s coefficient”. – For more than 15 measurements and for the confidence limit of 95% the value of t (M>15, @95%) is approximately 2 Submission Pertti Visuri, Airgain, Inc.

January/2006 doc. : IEEE 802. 11 -05/1259 r 0 Appendix 2 – Recommended Testing Procedure Calculating Confidence Limits when Standard Deviation is not Known • • In cases where the number of measurements is smaller than 15 and the standard deviation is not known from other sources (for example from other relevant tests in the same building environment) it will be necessary to apply the Student’s coefficient from the table in Appendix 1 and the actual formula for the confidence limits As can be seen from the graph, the confidence Confidence limits for various sample sizes limits for only a few measurements are quite wide when the standard deviation (3. 6) is estimated from the measurements in the sample = +/-t (M, C)*SQRT( 2/M) , where is the confidence limit, is the standard deviation, M is the number of measurements, C is the desired confidence level and t (M, C) is the so called “Student’s coefficient” (given in Appendix 1). Submission Pertti Visuri, Airgain, Inc.