Program Review Software Diagnostics And Conformance Testing Mark

Program Review Software Diagnostics And Conformance Testing Mark W. Skall, Division Chief Candy Leatherman, Secretary email: skall@nist. gov Tel. No. : 301 -975 -3262 Div. Web Site: http: //www. itl. nist. gov/div 897/

Division 897 Funding - FY 00 OA - $1, 615 K Projected Balance: $257, 062 (+ $637, 570 transferred to 2001 OA Reserve) ITS - $585 K STRS - $5, 207 K Employees: 35 Full Time 3 Part Time 10 Intermittent 5 Guest Researchers

SDCT’s GOAL To improve quality of software in industry through the development of: § Technology § Measurements § Standards Philosophy: § Concentrate on key areas at forefront of technology § Get involved early and partner with industry § Fill industry void § Transfer technology and move on Specific Strategy: § Conformance Tests § Reference Implementations § Diagnostic Tests § Research to reduce costs of testing § Reference Data § Standards developed jointly with industry

Excerpts from Assessment Panel Report – 6/26/00 Division Review “The planning and documentation methods used by SDCT could serve as a model for other ITL divisions. The process has used clearly-defined criteria to select state-of-the-art programs with clearly defined priorities and goals, wellidentified roles, and measurable contributions to national and international standards organizations. Examples include the work on conformance tests for XML, Distributed Interactive Learning Systems, and for DASE. . . Directions can change quickly within the software industry. The division carefully monitors these changes and acts appropriately. . Industry has provided many testimonials concerning the great value of the division’s standards work, including: ‘The OASIS-NIST XML Conformance test suite is critical for our industry’, Norbert Mikula, CTO, OASIS; ‘NIST made strong contributions (X 3 D) and resolved our knottiest problems’, Don Brutzman, Board of Directors, Web 3 D consortium, and ‘The high quality test suite and certification program is an invaluable resource for ATA’, Robert Peel, Director of Airworthiness and Standards”

Excerpts from Assessment Panel Report – 6/26/00 Laboratory Level Review “New projects are started based on the importance of the work to U. S. industry, and the work viewed by the panel generally had specific, focused goals. An example of this approach is the work on XML done in SDCT. This project is notable because it addresses a major standard on which industry was making little progress, and it also effectively leveraged skills and approaches developed in a now terminated project on VRML. . . Another example of the improved planning process is the existence of clear termination criteria, which have been useful in sunsetting projects. For example, the Real-Time Java project and the work on RBAC satisfied their completion criteria and therefore ended this year. ”

Software Diagnostics and Conformance Testing § Standards and Conformance Testing Group § Software Quality Group § Interoperability Group

Standards and Conformance Testing Group • Development of conformance tests • Development of reference implementations • Research into better ways to do conformance testing • Develop standards jointly with industry Group Leader: Lynne Rosenthal Supy Proj. Leader: Lisa Carnahan Clare Lucey, Secretary

Standards and Conformance Testing Group Mary Brady Lisa Carnahan Laurent Ciarletta (GR) Anthony Cincotta (S) Chris Dabrowski Alden Dima Leonard Gallagher Leonard Gebase Neil Gima (S) Alan Goldfine Martha Gray Michael Kass Clare Lucey Thomas Logue Carmelo Montanez-Rivera Thomas Rhodes Richard Rivello Jacqeline Schneider John Tebbutt Marie-Noelle Terasse (GR) Mark Zimmerman

Division 897 Standards and Conformance Testing Group

Software Quality Group • Develop methods to automate software testing • Develop software diagnostic tools • Develop reference data • Formal methods Group Leader: John Barkley Debbie Blackstone, Secretary

Software Quality Group Tamer Ahmed (GR) Paul Ammann (F) Debbie Blackstone David Brinkley (F) Paul Black Neva Carlson John Cherniavsky Anthony Cincotta Keith Gallagher (F) Roger Gima (S) Michael Koo Mary Laamanen James Lyle William Majurski Douglas White

Division 897 Software Quality Group

Interoperability Group • Ensure Federal agency requirements are input into voluntary standards committees - work through Federal CIO Council • Help Federal agencies and industry achieve interoperability through application of Division products • Provide technical support to voluntary standards committees • Support NIST paperless office effort utilizing digital signatures Acting Group Leader: Lisa Carnahan Vacant, Secretary

Interoperability Group Daniel Allen Bruce Bargmeyer (RA) Frederick Boland Gary Fisher Larry Fitzwater (GR) Elizabeth Fong Kathryn Harvill Roy Morgan Judith Newton Gertrude Sherwood

Division 897 Interoperability Group

Strategy • Conformance Tests Reference Implementations Standards developed jointly with industry Diagnostic Tests Research to reduce costs of testing Reference Data

Development of Conformance Tests • Emphasis on newer technologies • Tests developed in parallel with standards • Tests developed before implementations • Need buy-in from industry • Need high impact • Need technology transfer

Development of Conformance Tests (cont’d) • Legacy activities CGM SQL POSIX Ada and other programming language VRML • Current activities XML and companion standards (DOM, XSL, etc. ) Digital TV X 3 D Learning Technologies (IMS)

XML Conformance Testing Project Team: Mary Brady Carmelo Montanez Ricky Rivello Mark Zimmerman

Electronic Commerce B 2 C B 2 B (business-to-consumer) (business-to-business) Interactive Television (DASE, SMPTE) Distributed Learning (IMS) Manufacturing (ICM) XML Technologies Internet / World Wide Web

Why NIST? n Electronic Commerce Growth – Growth is exponential, already outpacing last year’s predictions – Changing the way businesses do business – Revolutionizing the way we consume and play n Advances are necessary in: – Tools for describing / sharing information n XML technologies – Applying IT to solve vertical market demands n NIST provides: – XML Technologies n Conformance Testing – NIST developed tests, available on-line, for use in testing applications – NIST technical leadership, neutral third-party – Unbiased Feedback of Specification Errors – Vertical Markets n Applied IT research

HTML or XML? Purchase Order

• Mary Brady
• NN Rm 572, Gaithersburg, MD, 20899
• 1234
• Trees
• 50. 00
• 20
• 1000. 00
• Be sure to water the root ball day of delivery
• Susan Carscadden
• NIST, Bldg 301, Gaithersburg, MD, 20899 Figure 1: HTML Purchase Order

XML Technologies Electronic Commerce, Distance Learning, Health Care Manufacturing, Interactive Televison Programming Languages Transformations Formatting Objects Stylesheets DOM Auto Parts Boards Medical Education Domain Specific Procurement Signatures Query Language Namespaces E-Business Learning Objects XML/EDI Ratings . . . Libraries . . . Control Information Description XML. ORG BIZTalk Registries Repositories XML Syntax, Information Set, Schema, Linking, Fragment

XML Conformance Testing XML Test Suite OASIS XML Conformance n • 1000 XML Tests • DTD + 4000 lines of XML • 400 lines of XSL Stylesheet Sun, IBM, Fuji Xerox, Data. Channel, Micro. Star, W 3 C members Why NIST? n 18 months n Technical Leadership Coalesced industry partners First Deliverable n 6 months n XML Test Suite XML files & XSL Stylesheet Future Work n Continued Support n XML Style. Sheets, Schemas XML NS, XLink, XPtr

DOM Conformance Testing W 3 C, OASIS n DOM Test Suite Sun, IBM, Microsoft, Netscape, Oracle • 300 ECMAScript Tests • 14, 500 lines of code XML ‘ 98 n n Met with W 3 C WG Chair NIST asked to develop tests XML ‘ 98 NIST Released n June ‘ 99 n Fundamental, Extended (ECMAScript) Interactive test harness Future Work n Continued Support n n Fundamental, Extended (Java) HTML (ECMAScript & Java) DOM Level 2

FY 00 Progress -Tests Completed XML Test Suite n (release 1) (release 2) – Incorporated new XML tests and fixes n Completed DOM-Ecmascript Test Suite (release 1) Completed DOM-Java Test Suite n Develop XSLT Test Suite n (release 1) – partnership with Lotus Corp. and OASIS

FY 00 Progress -Committee Leadership n n n Chair OASIS Conformance and XML Testing Committees Provide leadership for XSLT and Schema Testing committees Provide leadership for UN/CEFACT eb. XML initiative (conformance and reg/rep) – Provide conformance guidance – Develop conformance clause – Harmonize OASIS and eb. XML efforts

FY 01 Plans Complete XSLT test suite n Develop test suites for XML Schema and other XML Technologies n Develop XML-based automated test tools to improve test development n Continue to chair OASIS testing committees n

FY 00 Resources n n 3. 5 FTE $700 K (STRS) FY 01 Resources n n n 5 FTE Travel required to collaborate in standards and test development efforts Equipment: 10 K

XML Conformance Testing Customers: – IT industry: e. g. , IBM, Sun, Microsoft, Oracle, Data. Channel, Documentum – Vertical industries: e. g. , Education (IMS), Financial (e. g. , Dun and Bradstreet), Travel (e. g. , Sabre) Impact: – Conformance test suites have been used to improve the quality of XML/DOM processors (many of which are embedded in other XML software solutions). These are used in many vertical markets by millions of customers. – Feedback to standards developers improve the specifications

Interactive TV Project Team: Alan Goldfine John Barkley Doug White Len Gebase

Interactive TV • TV and the Internet are each recognized as “a technology …that seems to change everything” (BW, Oct 4 ‘ 99) • Interactive TV is the convergence of TV and the Internet • Profound impact on Electronic Commerce expected • Web. TV: 1 million subscribers, 350 hrs/wk of programming • AOLTV: projected 2 million subscribers in 2 years • ITL partnering to develop the standards and tests required for the success of Interactive TV

Interactive TV Traditionally separate media are converging: • Wireless phones do email • Internet does broadcast radio and TV • Broadcast TV references links to Web pages

Interactive TV TV: (non-interactive) OR Web: Currently, for the viewer: } accessed separately (interactive) With DASE/DDE, TV and Web are converging into a single medium: DASE/DDE

Interactive TV Demonstration Current production: Golf matches on TV DASE/DDE: single medium Web site with associated products DASE/DDE

Interactive TV Demonstration

Interactive TV Advanced Television Systems Committee (ATSC) • Members include: ABC, CBS, IBM, Intel, Lucent, Microsoft, NBC, Warner Brothers • DASE (DTV Application Programming Environment): Standard for a platform independent, high-level abstraction for integrating DTV and the Internet • DASE specifies programming environment for DTV receivers • Downloadable DASE applications portable across different TV set tops

Interactive TV NIST’s Role in ATSC DASE: • Joint project with the High Performance Systems and Services Division (895) • Chair DASE Conformance Working Group where key players include: ABC, Gateway, Microsoft, Sun • Partner with Unisoft to develop conformance tests • Reference implementation for programming environment

Interactive TV Society of Motion Picture and Television Engineers (SMPTE): • Members include: ABC, CBS, IBM, Intel, Microsoft, NBC, Warner Brothers • DDE (Declarative Data Essence): Standard for Internet content and bindings to analog and digital streams (formerly ATVEF) • Declarative Content: HTML, CSS, ECMAScript, DOM, Triggers (for syncing the two mediums)

Interactive TV NIST’s Role in SMPTE DDE: • Contribute text to specification • Partner with Unisoft to develop conformance tests • Develop conformance tests for triggers and stream binding to integrate with existing Unisoft DDE element tests • Develop prototype receiver testbed • Participate in Applications Data Essence (ADE) Study Group

FY 00 Progress n n n n Chair DASE conformance task group Conformance section of the DASE Standard completed Review of DASE conformance test assertions from Unisoft completed DDE Standard completed, adoption underway DDE Bindings Standard underway Test assertions for DDE triggers underway Testbed for DDE receivers underway ADE Study draft completed

FY 01 Plans n n n n Continue to chair DASE conformance group DASE Standard adopted Continue DASE conformance test development with Unisoft DDE and DDE bindings Standards adopted Develop DDE conformance tests for triggers, integrate with Unisoft tests Develop test assertions for DDE bindings Complete DDE Receiver testbed Begin DDE-2 Standard development

FY 00 Resources n n 2 FTE $400 K (STRS, ATP-$100 K) FY 01 Resources n n n 4. 5 FTE Equipment: $20 K Travel required for standards meetings and testing collaborations

Interactive TV n Customers – – n IT, e. g. , IBM, Microsoft broadcast TV, e. g. , NBC, Direc. TV entertainment, e. g. , Disney, Universal consumer electronics, e. g. , RCA, Intel Impact – New Medium: TV/Internet convergence – Interactive applications portable across receivers from different manufacturers – ATSC and SMPTE invited NIST participation – NIST chairs DASE Conformance Group

X 3 D Conformance Testing Project Leader: Mike Kass

Objectives § Continue the NIST/Web 3 D partnership begun in 1996 – ITL created the VRML Test Suite (VTS) and Viper parser § Create an X 3 D Test Suite ( XTS ) – – Modify VRML test requirements and test descriptions Translate existing VRML tests to X 3 D format § Chair the Interoperability/Conformance WG – – Interface with developers and specification writers to resolve ambiguities in specification Provide feedback to developers on browser conformance

FY 00 Progress n n n Completed translation of VTS to X 3 D Test Suite (XTS) Developed test harness to enable developers to contribute tests Interest from Dra. W and Sun for using the test suite and making the results public

FY 01 Plans Low level support - assisting community in using the VRML and X 3 D Test Suites n No new development planned. Maintenance as needed n

FY 00 Resources n n 0. 75 FTE $150 K (STRS) FY 01 Resources 0. 2 FTE (minimal) n No Travel is expected n

X 3 D Conformance Testing Customers: – X 3 D Consortium including Sun, Sony Dra. W Computing, Shout, Blaxxun Impact: – Conformance test suites are being used to improve the quality of X 3 D browsers – Dra. W Computing has said that they would make their test results public

Learning Technologies Project team: Martha Gray Tom Rhodes Jacki Schneider Lisa Carnahan

Learning Technologies Learning technology is: n Learning anytime, anyplace n Interactive, distributive learning systems In collaboration with industry, NIST/ITL is: n defining requirements and specifications n developing prototypes and reference implementations n developing testing techniques and tools

Why Now? n “The Gartner Group estimates that companies in the United States spent $58 billion on training in 1998, and of that, $9. 4 billion went to IT-related training. They expect this figure to grow to $10. 5 billion this year. ” Gilbane Report, July/Aug 1999 n “Just as electronic commerce exploded over the past two years, the stage is now set for ‘e-learning’ to become the next big wave in Internet-based applications” Cisco Systems CEO and President John Chambers n “Worldwide, the education and training market is over a trillion (US) dollars. ” IEEE

Projects 1. IMS Global Learning Consortium, Inc. § Chair conformance team – – – § 2. Develop Meta-data repository reference implementation IEEE and ISO § § 3. advise members on writing IMS specifications ensure testability of specifications advise on how to test and set up testing programs Provide conformance expertise Harmonization of efforts among specifications Do. D’s Advanced Distributive Learning (ADL) § Executive Order 13111 on Federal Training - NIST/ITL coleaders with ADL and DOL.

FY 00 Progress n n n Continue collaboration with IMS, IEEE LT committee and NIST/ATP Chair IMS conformance testing activities Developed IMS conformance test policies and procedures Served as Technical Lead of SBIR solicitation, assessed and recommend proposals Develop repository of learning objects (this work coordinates with Reg/rep subproject)

FY 01 Plans Continue to chair IMS conformance committee providing expertise to emerging IMS specifications n Develop a conformance test plan for applicable IMS specifications n

FY 00 Resources 1. 5 FTE n $300 K (STRS, ATP) n FY 01 Resources 1 FTE n Minimal travel required to attend standards meetings and interoperability demos n

Learning Technologies Customers: – IMS, Do. L, Do. D, Simon&Schuster, Sun, IBM, Apple, Microsoft, UNC, Calif State U Impact: – Use of standard metadata for global learning will improve discovery, evaluation, and use/reuse of learning objects – NIST’s conformance expertise contributed to clearer, less ambiguous, testable IMS specifications – Conformance test plans and procedures will enable IMS to determine compliant products

ATA CGM Testing Program

FY 00 Progress n n Completed final release of the ATA CGM Test Suite Provided guidance to ATA Recognized Test Laboratory on using the test suite and assessing clients Chair Control Board - resolve questions of interpretation of the test suite and standard. Initiated discussions on ATA's e. Commerce activities - explore a possible XML testing program

FY 01 Plans n Continue to support ATA's CGM Test Program – No new CGM development planned – Maintenance as needed n Provide guidance on the use of NIST's XML test suite and its applicability and use in ATA

FY 00 Resources n n 0. 25 FTE $75 K (STRS) FY 01 Resources 0. 25 FTE n Travel required to support ATA test program and potential for an ATA XML test program n

CGM Customers: – Air Transport Association – Airlines and their suppliers (e. g. , United, Northwest, Airbus, Fed. X, Boeing, Rolls-Royce) – CGM Developers (e. g. , Autotrol, Intergraphics) Impact: – NIST Conformance test suite used by ATA CGM Test Program to validate CGM interpreters – Quality, reliable CGM products used for representing graphics in airframe maintenance manuals

Operational Testing Issues Test suites should be available on the Internet Test suites should be free of charge Operational testing should be done by private sector Industry associations may issue certificates

Technology Transfer from NIST to ATA Developed CGM Interpreter Test Suite Developed Test Program Framework Developed test procedures Formed and chair Control Board Developed Test Lab criteria and evaluation criteria Assessed Test Lab for ATA recognition Serve as technical adviser Completed transfer

Testing Services Status of Software Operational Testing Services

Testing Services (cont’d) Testing Products or Services - Offered by Other Organizations Developed with NIST Assistance

Strategy Conformance Tests • Reference Implementations Standards developed jointly with industry Diagnostic Tests Research to reduce costs of testing Reference Data

Reference Implementations 3 Purposes § Used as a metric of comparison in conformance testing § Used to sort out errors and ambiguities in standards § Used as a prototype

Aroma Pervasive Computing Project Team: Alden Dima Bill Majurski Mary Laamanen

Aroma Pervasive Computing Project n Emerging trend toward – numerous, easily accessible, often inconspicuous computing devices and sensors; mobile or embedded in the environment; connected to ubiquitous network structure n Industry is asking for – – – n Standards for service discovery, APIs, e-commerce services Testing and diagnostic tools to improve quality, interoperability Reference implementations Aroma = multi-division effort to: – Explore standardization and measurement issues – Attract industry interest and partnerships

Aroma Prototype (Aroma adaptor + Aroma projector) n Goals - To discover technical, standards and measurement issues related to: – – Connecting portable wireless devices to traditional networks Service discovery, self-configuration and dynamic resource sharing Mobile code and data Software infrastructure needed to create and manage pervasive services and applications

FY 00 Progress n n n Present papers at NIST PC 2000, Java. One, and International Pervasive Computing Workshop Co-chaired NIST Pervasive Computing 2000 conference Developed Aroma Adapter-Projector with Div. 892 Measured performance of Aroma adapter Developed conceptual model based on the OSI Reference Model Exploring low-end pervasive computing – Embedded and 8, 16 -bit processors – Collaboration with US Naval Academy

FY 01 Plans n n n Co-chair NIST Pervasive Computing 2001 conference Explore use of ADL in pervasive computing model Implement a virtual private smart space (VPSS) test bed Extend Aroma projector to include digital cameras and VPSS technology Continue performance measurements Develop diagnostic and testing tools

FY 00 Resources n n 1. 50 FTE $300 K (STRS, OA-DISA) FY 01 Resources n n n 3 FTE Travel required to conferences to present work and collaborate with colleagues Equipment: $15 K

Pervasive Computing Customers: – IT industry: the usual suspects – Consumers: e. g. , GE, Sunbeam, Phillips, Northeast Utilities, Xerox – Universities: USNA, UMD, UC Berkeley Impact: – Solicited industry needs for standards and measurements – PC 2000 conference launched awareness of NIST as key player – Papers presented at conference stimulated discussions on pc and standards – Ref imp demonstrated feasibility of PC – Enable widespread adoption and integration with software tools to create, manage, measure and test pervasive services and applications

DISCOVERY Registry Internet RETRIEVAL XML Registry/Repository (Reg/Rep) Project Team: Lisa Carnahan Len Gallagher Mike Kass Registry

XML Registry/Repository Reference Implementation Built to OASIS specification n Being used by both OASIS and UN/CEFACT (eb. XML) n Built using IMS Meta-data specification n Apply lessons learned to EC repositories n

FY 00 Progress Revised the OASIS Registry and Repository Specification n Implement a reference implementation n Participate in eb. XML Reg/Rep team n – Ensure testability of specification – ensure compatibility between eb. XML and OASIS specifications

FY 01 Plans Update reference implementation n Develop conformance test suite/data for OASIS specification n Coalesce the eb. XML and OASIS specifications n

FY 00 Resources n n 1. 5 FTE $325 K (STRS) FY 01 Resources 2 FTE n Travel required to attend standards meetings n

XML Registry/Repository n Customers – EC/Enterprise Integrators: Data Channel, Documentum, Oracle, Microsoft, Sun, IBM – Consumers: Vertical market partnerships and supply-chains (e. g. , automotive, insurance, travel, manufacturing) n Impact – Repositories allow efficient use of XML and eliminate duplication – Enable SMEs to participate in EC marketplaces and supply chains – Reduce cost of doing business through timeliness and increased automation

Internet Commerce for Manufacturing (ICM) (ITL, EEEL, MEP) Project team: Tom Rhodes Liz Fong Kathy Harvill

FY 00 Progress n n n Assumed project leadership (at request of EEEL) Defined standards-based framework for ICM testbed (reference implementation) Conducted workshops and presented papers/demos – NAMT, SIMA, and ATP reviews – Industry conferences: IPC, NEMI, ITEM 2000 n n n Collaborate with NEMI, IPC, Georgia Tech, UMBC Evaluating industry specifications (e. g. , Rosetta. Net, OAG, C-Net, e. Co FW) for B 2 B interactions Evaluating applicability of agent and web technologies

FY 01 Plans n n Continue industry collaborations Demonstrate distributed ICM testbed operations (with Ga. Tech, NEMI, IPC partners) n n Test and validate ICM information model in ICM testbed Continue evaluation of applicable standards and technologies and their incorporation in ICM testbed

FY 00 Resources n n 1. 5 FTE $350 K (STRS-EEEL, ATP) FY 01 Resources 2 FTE (other FTE from EEEL, MEP) n Travel required to collaborate, participate in standards meetings, and present work n

ICM n Customers – Consumers: PC assembly supply chain participants – Consortia: NEMI, IPC, Rosetta Net n Impact – Reduce the cost of circuit-board manufacturing by enabling the rapid exchange of build information and change notices for all supply-chain participants – Demonstrates the use of XML in a vertical industry

Strategy Conformance Tests Reference Implementations • Standards developed jointly with industry Diagnostic Tests Research to reduce costs of testing Reference Data

Forward-Looking Standards Role-Based Access Control (RBAC) Virtual Reality Modeling Language (VRML)/X 3 D XML and companion standards (DOM, CSS, etc) Java ADL IMS DASE/SMPTE OASIS Registry/Repository GIS

STANDARDS COMMITTEES AND RELATED ACTIVITIES NATIONAL • NCITS (Previously ANSI accredited X 3) Management Level JTC 1 -TAG Technical Committees H 2 (Database) H 3 (Computer Graphics) L 1 (Geographic Information Systems) L 8 (Information Representation) J 22 (Java) R 1 (Real Time Systems) IEEE Standards Board 1484 FEDERAL GOVERNMENT CIO Council Interoperability Committee Standards Working Group INTERNATIONAL • ISO SC 32 (NCITS-L 8, NCITS-H 2) SC 22 (NCITS-J 22) SC 24 (NCITS-H 3) TC 211 (NCITS-L 1) SC 36 (IEEE 1484) • ECMA TC 39 (Java. Script) OTHER • World Wide Web Consortium XML, DOM, XSLT, CSS • IMS Global Learning, Inc. • Object Mgmt. Group • OASIS • Web 3 D Consortium • eb. XML (UN/CEFACT)

Real-Time Java Project team: Lisa Carnahan Alden Dima

Real-Time Java n Assistance asked by industry providers and government users n Collaborate with Sun, HP, IBM, New Monics, FTL n Hosted continuing workshops for real-time Java requirements n Addresses real-time issues in both Java embedded and Java server application environments n NIST coalesced industry to produce one set of requirements leading to one spec n Open consensus forum for requirements n BOTH camps (Sun and HP, Microsoft, etc. ) have used NIST-led requirements document to produce specs

EE Times “Interestingly, both the splinter group and Sun are using as a serious basis for their efforts a ‘requirements document’ for real-time (Java) extensions prepared by a separate group working under the auspices of the National Institute of Standards and Technology (NIST). That is raising hopes that both standards might ultimately come together. ”

Real-time Java Requirements Group • “Requirements for Real-time Extensions to the Java Platform” (final release -September, 1999) • NIST’s Requirements Group effort dissolved after successful completion of its goal. Sun Microsystems’ sponsored RT Java Expert Group “The National Institute of Standards and Technology (NIST) has organized a real-time Java requirements group. This is an open group that is defining requirements for real-time Java. Their requirements document will be a valuable input to the experts group, …” JSR-000001 Real-time Specification for Java (Call forming the RT Java Experts Group sponsored by Sun Microsystems; NIST member) The J Consortium “The J Consortium uses real-time requirements developed by the Requirements Group for Real-Time Extensions for the Java Platform, an effort sponsored by the National Institute of Standards and Technologies (NIST)…. ” [Press Release -J Consortium™ Unveils First Public Real-Time Specification For The Java Platform] CUPERTINO, Calif. , September 24, 1999

FY 00 Progress n n n Real-Time Java Requirements document completed Worked with both spec development groups to help coalesce industry Member of Sun RT Experts Group contributed to specification Worked with J Consortium to help develop specification Project sunset

FY 00 Resources n n 1. 25 FTE $250 K (STRS) FY 01 Resources n 0 (Project sunset)

Real-Time Java n Customers – Sun, IBM, HP, New. Monics and users of real-time Java n Impact – Coalesced industry to produce one set of requirements – 2 spec efforts compatible – New book on Real-time Java mentions NIST – Presented to and endorsed by VCAT

ADL (Architecture Description Language) Project Team: Chris Dabrowski John Tebbutt

Architecture Description Language n n Focus on high-level structure of the overall application (i. e. , systems architecture) Provide features for modeling software systems architecture – E. g. , Structure, components, interfaces, interconnections n n Support automated analysis and verification of design Enable testing during design phase – Saves $ over testing at code phase

Architecture Description Language (ADL) Software Architecture: “. . an abstract system specification consisting primarily of functional components described in terms of their behaviors and interfaces and component-component interconnections” [Hayes-Roth, 1994] ADL: “A language that provides features for modeling a software system’s conceptual architecture” [Medvidovic 1996] Focus Why ADLs? • Complexity of s/w - formal verification Analysis Design Implementtation Testing Maintenance • Software re-use - strong architecture description • Testing during design phase - saves $ over testing at code phase Support

NIST’s Unique Role n Standards Role: Neutral Expert Applied Research: Contribute to ADL body-of-knowledge Standards Collaborations – Propose additional functionality to SAE (first industry standards effort) – Uniquely positioned to bring academic and industry communities together. – Apply developing AADL standard (based on Honeywell’s Meta. H) to Unmanned Ground Vehicle: build additional functionality into AADL standard – Explore ADL relationships with IEEE and Open Group efforts n Applied Research: Contribute to ADL body-ofknowledge – Research application of ‘formal approaches’ to ADLs – Enhance ADL features for verifying conformance of application systems architectures to reference architectures (e. g. RCS) – Publish successes and lessons-learned for industry awareness – Provide feedback to appropriate ADL tool developers

FY 00 Progress n n Participated in development of SAE’s Avionics ADL (AADL) standard Explore ADL standardization efforts in IEEE and Open Group Established partnerships with DISA, US Army, ITL/Security Division Identify domain areas for applying ADL to explore standardization and testing issues – E. g. , government smartcard, unmanned ground vehicle, pervasive computing

FY 01 Plans n n Provide support for the formalization of AADL to ensure its testability Develop prototype of US Army robotics architecture – collaboration with MEL – dependent on DISA funds n n n Develop prototype of Smartcard Architecture w. Div 893 Demonstrate and document the benefits and functionality of ADLs Produce a comparative study of different ADLs to support further development and standardization of ADLs

FY 00 Resources n n 1 FTE $200 K (STRS) FY 01 Resources n n n 2 FTE Equipment and Software: 10 K Travel required to standards meetings and to participate in conferences and industry meetings

Architecture Description Language n Customers – Consumers: Industries that rely on component-based systems (e. g. , manufacturing, avionics, real-time) – ADL tool vendors n Impact – Moved ADL from University research to industry standard – Provide uniform descriptions of the structure and function of software. – Increases software quality by reducing errors through testing in design phase. – Increase the producibility, quality, and reusability of component-based software systems.

CIO Council Standards Working Group Technology Transfer Act/OMB Circular A-119 direct federal agencies to use voluntary consensus standards in lieu of government unique standards NIST directed to coordinate federal technical standards activities with private sector technical standards activities NIST/ITL initiated effort to create Standards Working Group within Federal CIO Council Standards Working Group approved late last year by the CIO Council Interoperability Committee Standards Working Group to be focal point of discussion, definition, and coordination of federal requirements for voluntary IT standards, and will assess the results of voluntary IT standards in meeting requirements Standards Working Group chaired by a member of this Division, and will be supported by this Division in its efforts to provide a unified voice to the voluntary IT standards community for Federal IT standards technical requirements

FY 00 Progress n n Prototype of web based support software for collection, evaluation, and status reporting on Federal Government technical requirements for voluntary standards developed and demonstrated. Electronic Documents conference held featuring speakers from private industry, Federal Government, and voluntary standards organizations. Meetings of the Standards Working Group held on an as needed basis. Standards requirements forwarded to appropriate voluntary standards committees as the requirements are approved.

FY 01 Plans n n n Continue development of web based support software for Standards Working Group. Make software available for possible use by other Working Groups within the EIEIT structure. Meetings of the Standards Working Group held on an as needed basis. Standards requirements forwarded to appropriate voluntary standards committees as the requirements are approved. Consider holding second Electronic Documents conference.

FY 00 Resources n n 1. 5 FTE $325 K (STRS, OA-CIO Council) FY 01 Resources n 1. 5 FTE – Combination of time from Group Leader and others in Interoperability Group

CIO Council Standards Working Group n Customers – Other Federal Agencies n Impact – Federal agencies input requirements to standards groups and utilize voluntary consensus standards – Eliminates duplicative standards

Strategy Conformance Tests Reference Implementations Standards developed jointly with industry • Diagnostic Tests Research to reduce costs of testing Reference Data

Year 2000 Problems - Project Leader: Gary Fisher Published bulletin and produced Web site • Raised awareness of issues • Proposed methods for managing conversion Issued change notice to FIPS 4 -1 on date formats • Recommend use of four-digit format • Discouraged use of two-digit format Developed source code scanner Developed test assertions • Defined types of tests needed for date/time testing • Under evaluation by IEEE Held international symposium June 9 -10, 1997 • Management and technical issues • Market development of tools and services • Case studies by companies and governments Participate in Interagency Year 2000 Task Force

FY 00 Progress n n Y 2 K web site completed Embedded system study completed – Report sent to Koskinen – Problems highlighted and impacts reported – Produced new emphasis on embedded systems n Project sunset

FY 00 Resources n n 0. 25 FTE $50 K (STRS) FY 01 Resources n 0 (Project sunset)

Year 2000 n Customers – All businesses that use computers – Consumers who use computers n Impact – Y 2 K rollover went without incident

Strategy Conformance Tests Reference Implementations Standards developed jointly with industry Diagnostic Tests • Research to reduce costs of testing Reference Data

Automatic Generation of Spec Based Tests Using Formal Methods Project Team: Paul Black Bill Majurski Jim Lyle Micheal Koo

Industry Need • Software testing is enormous cost to industry: 30% - 50% of budget Development may be 1000’s of staff-years Testing cost may be $ millions • Current methods: mostly hand generated tests Inefficient Not rigorous enough • Software testing: key to reducing security incidents: “More than 75% of the incidents…are the direct result of …bugs. ” and “…more real testing - not 'beta testing' in the marketplace - will yield programs that are less vulnerable to attack. ” (Business Week, February 28: "Locking Out the Hackers")

Problems • Conventional software specifications are in natural language and so lack precision Ambiguities Inconsistencies Errors • Conventional software testing problems: Exhaustive testing not possible Test case selection subjective Result = costly and time-consuming testing

Current Test Development Methodology English Language Requirements review & inspect English Language Specification review & inspect Programming Language Source Code Human Programmers Software Tests

Advanced Test Development Methodology Mathematical Logic Requirements formal proof Mathematical Logic Specification Test Generator informal proof Programming Language Source Code Software Tests

Accomplishments • Method for automatically generating tests from formal specifications • Associated coverage metric • Publications and presentations ICFEM (98), WSEST(99), HASE (99), DASC (99) GMU, UMBC, College Park, Brisbane, NRL • Feasibility collaborations leading to commercialization Argus, a secure computer operating system Ford Motor Research Lab: powertrain control systems Northrop Grumman: fighter radar computers

FY 00 Progress n Feasibility tests on models from Ford and Argus n Initial formalization of BLAST (Basic Local Alignment Search Tool: identifies similar DNA sequences) Algorithm completed in collaboration with CSTL n Hosted Workshop on Testing Software Using Formal Methods n Hosted Tutorial on Formal Methods n 3 papers published n 2 papers submitted and accepted n 4 papers submitted n 9 talks (6 invited)

FY 01 Plans n Commercialize ATG Method with Ford and software company (tbd) n Continue feasibility studies with Argus n Complete BLAST formalization

FY 00 Resources n n 2. 5 FTE $500 K (STRS-Competence) FY 01 Resources n n n 2. 5 FTE Equipment: $5 K Travel required for validation collaborations and presentations

Automatic Generation of Tests n Customers – Software developers – Software end users, e. g. , web surfers – Security administrators – Researchers in software testing methodologies n Impact – Reduce the cost of software testing (50% of development costs) – Improve software quality – Reduce security incidents (75% due to bugs) – Ford and Argus interested in commercialization of ATG methodology – Strongly endorsed by VCAT

Software Testing by Statistical Methods (Competence Project) n Objective: develop new methods for software testing based on stochastic processes and metrics, in order to – Improve the quality of software development – Develop reliability estimates that software correctly adheres to its specification n Joint project – Software Diagnostics and Conformance Testing Division (897) – Statistical Engineering division (898) n Collaboration with other NIST Labs (MEL, EEEL, CSTL) Project Team: Len Gallagher Lynne Rosenthal

Problem • Exhaustive testing not possible Can show presence of errors, not their absence • Degree of correctness not quantified Can derive subjective level of confidence, not quantitative measure • Test case coverage not science Strategy is intuitive, based on anecdotal evidence

Conformance Tests Software Implementation 100% non conforming ? ? ? conforming

FY 00 Progress n n Published paper on Coverage Model of Mutation Testing (with Paul Black) Produced paper on applying statistical methods to component based testing Produce paper on Test Input Needed to Improve Reliability Analysis of effort to apply statistical method (partitioning) to new test development

FY 01 Plans Continue effort to apply statistical method (partitioning) to new test development n Apply statistical method to pervasive computing applications n

FY 00 Resources n n 1 FTE $200 K (STRS-Competence, plus $100 K given to ITL/SED) FY 01 Resources n n 1 FTE 200 K competence funding (plus 100 K given to ITL/SED)

Software Testing by Statistical Methods Customers: – Conformance Test Developers – Ourselves – Users of conformance tests Impact: – Improve our knowledge and skills in designing and developing quality software. – Quantitative measures associated with conformance testing.

Quantum Information Project Team: Paul Black Neva Carlson

Quantum Information n Goal: to develop information representations and computing machines that take advantage of quantum effects n Joint effort of physicists and computer scientists

FY 00 Progress n Invited to develop joint proposal for Nanotechnology Initiative with Physics n Invited by NIST Physicists to present Basic Concepts in Theory of Computation n Developed Quantum Information Demo n Developed Quantum Information Bibliography n Attended weekly Physics Quantum Information seminar

FY 01 Plans n Develop products for Nanotechnology Quantum Information project n Organize Workshop on Computer Science in Quantum Information n Research questions in testing Quantum Algorithms

FY 00 Resources n . 5 FTE n $100 K (STRS – Physics Lab? ) FY 01 Resources n 1. 5 FTE (depends on Nanotechnology Initiative or $ from Physics) n Equipment: $0 n Travel anticipated

Quantum Information n Customers – Researchers in computational theory, and algorithm design and analysis – Developers of large computer models, e. g. , global weather – Cryptanalists n Impact – Computing machines of unprecedented capability and small size – Solutions to intractable computational problems, e. g. , factoring, searching – ITL staff invited to present at Physics seminars and participate in Physics Lab efforts

Strategy Conformance Tests Reference Implementations Standards developed jointly with industry Diagnostic Tests Research to reduce costs of testing • Reference Data

Error, Fault, and Failure (EFF) Data Collection and Analysis Project Team: Dolores Wallace Michael Koo Tony Cincotta Jason Rager

Error, Fault, and Failure (EFF) Data Collection and Analysis • Repositories of reference data on software failures • Tools to create and manage repositories • Tools to analyze software failure data • Research in taxonomies to characterize software failures

FY 00 Progress n Repository systems: – Completed public repository with predefined queries – Completed internal repository with dynamic queries n Data: 1000 data points added n Paper on failures in medical devices finalized n Papers: report on NASA data submitted to NASA Project was sunset n

FY 00 Resources n n 1. 5 FTE $300 K (STRS, ATP - $160 K) FY 01 Resources n 0 (Project sunset)

Error, Fault, and Failures n Customers – Software Developers – Researchers in Software Development Methodologies n Impact – Identifies the stages in the software life cycle whose practices can be improved – Endorsed by the Software Division of the American Society for Quality – Project methodologies and tools used in graduate programs at Johns Hopkins and the University of Louisville – EFF Tool used by RST Corporation

Other Agency Projects n NIJ - Software and Tool Verification – Project Team: Gary Fisher, Len Gebase, Jim Lyle, Jackie Schneider, Mark Zimmerman n VA - Health Care Information Systems – Project Leader: Bill Majurski n GSA - EC Standards Roadmap – Project Leader: Mary Brady n EPA - Metadata Descriptions and Registries – Project Leader: Judy Newton

NIJ National Software Reference Library & Beta Testing

National Software Reference Library n n n GOAL: Reference Data Set of file signatures (hash) of COTS file Used during examination of digital evidence to identify pertinent files and eliminate others BOTTOM LINE: One computer may have 5, 00020, 000 files. NSRL data set can eliminate 40 -95% of those files and save hundreds of staff-hours. Beta Testing Project n n GOAL: Verification of computer forensic tools and their use in analyzing digital evidence Used to ensure tool usage yields objective, repeatable, reproducible, results that hold up in court.

FY 00 Progress n Developed NSRL data dictionary and database structure for hash codes Established contacts with Microsoft, Adobe, et al to supply software – CBD announcement, Press release Identified hash algorithms n Creating Reference Data Set (i. e, file signatures) n Developed draft test plan for beta testing Develop procedures and tests for imaging tools n n n

FY 01 Plans n Establish the reference data set and distribution mechanism – (SQL database with web interface) n n Develop test metrics for efficacy of hash codes used in the reference data set Test the NSRL web site and make it available to approved users Continue to develop tests and procedures forensic tools Issue test reports and test suites (publish on web site)

FY 00 Resources n n 3. 5 FTE $700 K (OA-NIJ) FY 01 Resources 3. 5 FTE n Travel required for training courses and presentations n

NIJ Customers: – Law Enforcement Agencies (e. g. , FBI, Treasury, DCFL, state and local agencies) – Software companies (e. g. , Adobe, Microsoft) Impact: – Improve the efficiency of forensic examination of suspect computer files/date (NSRL data set can eliminate 40 -95% of suspect files and save hundreds of staff-hours). – Provide testing procedures forensic software tools to ensure the analysis process and results holds up in court

Healthcare Information Systems Project Team: Bill Majurski Tony Cincotta Roger Gima John Barkley

GSA - EC Standards Roadmap

EPA - Metadata Registries

Industry Testimonials n n n “The OASIS-NIST XML Conformance Test Suite is critical for our industry. ” – Norbert Mikula, CTO, OASIS “NIST made strong contributions (X 3 D) and resolved our knottiest problems. ” – Don Brutzman, Board of Dir. , Web 3 D Consortium “Your expertise helped advance conformance and spec development. ” – Steve Griffin, IMS Technical Director “The high quality test suite and certification program is an invaluable resource for ATA. ” – Robert Peel, Director of Airworthiness and Standards “NIST’s participation on DASE is vital for the U. S. industry. ” – Lisa Mc. Knight, Multimedia Business Development

NIST Requested Division Presentations n n e. Commerce - XML and DASE – Presented to Undersecretary Shavers – Presented to NIST Director and Deputy VRML, Real-Time Java, Auto Generation – Presented to VCAT XML/DOM Conformance Testing – Presented at ITL Town Meeting VRML Conformance Testing – Presented at ITL Town Meting n Vm. View n – Presented to Congressional staff ICM Project Briefing – Presented at ATP National Meeting

NIST Testimonials n n “Software testing work. . . is highly appropriate serving a broad set of companies who would not, on their own make these investments. ” – VCAT Report “Way to go folks. You did us proud. ” – Brian Belanger, Exec. Director, VCAT “The model used in the software testing program - to partner with industry, rather than developing tests and transferring them is excellent and will become a model for other programs at NIST. ” – Ray Kammer’s feedback of the ITL Program Review “A wonderful testimonial to your work. ” – Ray Kammer’s response to the ATA letter.

Priorities Core Program XML Conformance testing 2. Interactive TV 3. Automatic Generation of Tests 4. Pervasive Computing 5. XML Registry/Repository -----------------------------6. Computer Forensics – NIJ 7. CIO Council Standards Working Group 1.

Priorities – Next Level 1. 2. 3. 4. 5. 6. 7. 8. ADL Health Care Information Systems – VA CGM Learning Technologies Internet Commerce for Manufacturing Software Testing by Statistical Methods Quantum Information Other OA Projects (GSA, EPA)

Division Issues We’re a victim of our success - we’re in constant demand n We need people!!!!! n We need more discretionary funding n – STRS vs. OA/ATP n n Travel demands are increasing Demonstrations - good and bad Membership in consortia is essential Collaborations with other Divisions

BARKLEY ROSENTHAL ROSEN SKALL We also consult with distinguished scientists

Remember, Al and the SDCT Division jointly invented the Internet.