Future Directions For The Windows CE Test Kit

Future Directions For The Windows CE Test Kit James Zwygart Lead Program Manager Mobile and Embedded Microsoft Corporation

Agenda Testing Windows Devices Overview of Microsoft Testing Process Tools and Resources

Who Should Test What? Added Applications In. ROM ISVs, OEM/ODM Customization Windows Mobile Microsoft Windows CE Drivers and Radio OEM/ODM

Testing Windows Embedded Devices Fact Quality is more than just Software and Hardware requirements. There must be integration testing Testing is an on-going process that starts from day one, and evolves throughout the product cycle Testing is a mix of different methods, tools, people, and approaches Your In. ROM ISVs should be using all the tools you have available to you

Testing Windows Embedded Devices Fiction Passing the test in the CE Test Kit and requirements for the Mobile Logo Test Kit = High Quality Device. There must be integration testing The tools and resources Microsoft provides are all you need

Agenda Testing Windows Mobile Devices Overview of Microsoft Testing Process Tools and Resources

Definitions Bug – a defect in the product that is tracked in a database Triage – process in which decisions are made whether to fix, postpone, or dismiss a bug MTTF – mean time to failure. Our primary metric for Stress and Stability testing Exit Criteria – set of criteria that must be met to exit a phase of the product Dogfood – using your own product while it is in development

Overview Of Microsoft Testing Process

Agenda Testing Windows Mobile Devices Overview of Microsoft Testing Process Tools and Resources

Tools And Resources Platform Builder for Windows Mobile 5. 0 (Magneto) Remote tools Debugger OEM Documentation Windows Mobile CETK (WMCETK) Application Verifier, TUX, DATK, Radio, Hopper, and more Windows Media Testing Plays. For. Sure Test Kit or the Basic Media Test Kit (b. MTK) Community/Support TAMs Windows Mobile Device Maker Portal General CETK Newsgroups (great to reference)

WCETK Communications How the WCETK communicates with Host and client

Connecting The CETK Connect the CETK to a target device by using Platform Manager Connect the CETK to a target device by using Sockets Command-line from PB or the device Clientside /n=“Server. Name” /p=5555 /a /s=“james cetk. txt file This can be on the device in the same directory as clientside. exe or in the release directory Registry of the device Create a HKEY_LOCAL_MACHINESOFTWAREMicrosoftCETT registry key on the target device

MTTF And Stress (Test Lab Setup) Stress manages test modules that perform the actual testing Stress runs a specified number of test modules at all times Stress tracks memory and object store usage over time, records whether each test module passes or fails, and detects when a test hangs The harness provides the ability to set the level of logging verbosity for the test modules The harness can be configured from the Windows Mobile Stress dialog box

Lab Diagram

Understanding Failures There are 2 types of failures Expected failures Know the driver functionality. So you will understand if a test is failing because the driver does not support some functionality At this point edit the test command line to eliminate this specific case Unexpected failures Use the source code to the tests to help debug why a test is failing

Using Test Sources You can install source code for the tests that ship with the CETK Microsoft provides source code to help you debug your device drivers and create custom tests for the CETK However, you cannot build some of the source code, and source code for some tests is not provided

Advanced Features Auto-run using the /a option Using a storage device with low memory devices Running tests that need to reboot or run “Off-line”

List Of Included Tests CD/DVD-ROM File System Driver Test Winsock 2. 0 Test (v 4/v 6) Keyboard Mapping Test Keyboard Test Mouse Test Touch Panel Test Storage Device Block Driver Read / Write Test Storage Device Block Driver Benchmark Test Smart Card Resource Manager Test Serial Communications Test Serial Port Driver Test Parallel Port Test IR Port Test (Winsock 1. 1) IR Port Test (Winsock 2. 0) Telephony API Server Test Telephony API Client Test Two-Card Network Card Miniport Driver Test One-Card Network Card Miniport Driver Test Bluetooth Test GDI Display Test Wave Audio Sound Card Test Direct. Draw Display Test list Storage Device Block Driver API Test CD/DVD-ROM Block Driver Test Partition Driver Test RDP User Scenario Performance Test

New Tests To CE 5. 0 OAL Cache Test OAL Timer Test OAL IOCTL Test Kernel Scheduler Test User Interface Test Graphics Device Interface Printer Test PC Card 32 -Bit Card. Bus Host Controller Driver Test PC Card 16 -Bit PCMCIA Host Controller Driver Test PC Card Legacy PCMCIA Driver Test USB Host Controller Driver Test Flash Memory Read/Write and Performance Test NDIS Performance Test Audio CD Driver Test Imaging Codec Test Direct 3 D Mobile Driver Comparison Test Direct 3 D Mobile Driver Verification Test Direct 3 D Mobile Interface Test Winsock Performance Test Wireless Network Card Test Notification Test MSMQ Test ICMP API Test Credential Manager Test Local Authentication Plug-in Test RAS Server IOCTL Test RAS Authentication Test RTC Provisioning Test RTC Client Test RTC Presence Test RTC PC to PC Session Test RTC PC to Phone Session Test RTC Instant Messaging Session Test RTC Group and Buddy Test (Part 1) RTC Group and Buddy Test (Part 2) RTC ACL Test Vo. IP Test

New Tests For Later Versions Of Windows CE New tests will be announced at a later date

Test Kits WMCETK There is some overlap between the three test kits CETK (CE Test Kit) LTK (Logo Test Kit) WMCETK (Windows Mobile CETK) CETK General guidelines CETK Suite of tests to validate Windows CE drivers Includes certain tests that are irrelevant to Windows Mobile WMCETK Exposes CETK tests that are relevant to Windows Mobile (omits others) Also adds some additional Windows Mobile-specific tests; e. g. radio, battery LTK A database harness that exposes tests required to validate the requirements in the ‘Hardware and Software Requirements Document’ Includes manual, automated, and partial-automated tests Utilizes a subset of the tests from the WMCETK Adds some additional tests; e. g. a subset of the Plays. For. Sure test kit LTK

When To Use These Tests? Repeat throughout Development of Mobile Device Checkpoint occasionally during development and before final release Primarily needed for BSP bring up WMCETK LTK

The Windows CE Test Kit James Zwygart

Test Suite Drill-Down OAL Tests Timers Confirm that drift is minimal Check for valid resolution Caches Cache Parameters Correctness IOCTL_HAL_GET_DEVICEID IOCTL_HAL_GET_UUID WMCETK LTK

Test Suite Drill-Down Graphics Device Interface Test WMCETK Verifies drawing of shapes Lines, Polygons, Ellipses, CETK Verifies surface manipulation Raster operations Compares ‘known-good’ output to driver under test Test Case 200 Examples of test scenarios Test Case 210 Draw bitmap, line, shape, or text Perform a raster operation (e. g. merge two bitmaps) LTK

Test Suite Drill-Down Radio. Metrics Test Scenarios Voice Call 1000 Incoming and Outgoing Calls General Functionality Mixture of Voice, Data, RIL, and SIM tests Call Hold and Conference Testing Call Waiting, Call Hold, and Call Conferencing SMS 100 outgoing and incoming SMS Messages of varying length Supplementary Services RIL Tests GPRS Performance Test 20 MB of Data received in 4 x 256 KB sections x 20 sections GPRS Stability / GPRS Sync Data testing mimicking Active sync like scenario for 24 hrs Transfer Data, Sleep, Transfer Data, repeat. General Stress Random testing for 24 hrs. to make sure that no crashes occur RIL Test Kit (RILTK) in the making ETA EOY 2005 WMCETK LTK

Key Resources Windows Mobile Device Maker Portal http: //msftmobile. com Access to the latest Adaptation Kits, Documentation, Logo Test Program information Helpdesk for Logo Test Program issues OEM Documentation Design Requirements (Software, Hardware, and Branding) Everything Windows Mobile 5. 0 Plays. For. Sure http: //www. microsoft. com/getplaysforsure

Call To Action Start testing early Share what you have available with your In. ROM ISVs Extend what we provide and add your own tests

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.