Registration Management Committee RMC Workshop July 14

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Prepared/Presented by: Ben Tuley SAI Global - Senior Consultant (214) 274 -8646 – ben. tuley@saiglobal. com Lloyd (Sonny) Crile – Boeing Supplier Quality (316) 304 -5294 – lloyd. l. crile@boeing. com

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth • Why is this important? – ANAB auditors issued over 75 NCRs in 20 areas in 2005 -2007 for not auditing to sufficient depth – Many were repeat NCRs – Undermines the integrity of aerospace audits – Serious concern within aerospace community

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth • 75 NCRs in these 20 areas

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth • Why is this important? – OEMs beginning to conduct validation audits (independent checks of suppliers with AS 91 XX certifications) – OEMs can (& do) complain directly to ANAB regarding unresolved issues related to CB conformance to established aerospace requirements, including not auditing to sufficient depth – ANAB will respond very promptly (i. e. visit CB office within a few days)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth • What should auditors do? – Understand the requirements to which the audit must be conducted – Research/get training in topics they do not understand – Understand the processes and interactions and involvement of all stakeholders – Understand what objective evidence is required for an auditee to show conformance with a requirement – Accomplish fair, unbiased, detailed audits to the established criteria – Avoid “soft” auditing or “glazing over” topics

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth • What will we address in this session? – Document control (including configuration management) – Procurement (including purchasing and supplier control) – Product acceptance software

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Document Control and Configuration Management Requirements (13 ANAB NCRs in this area in the past three years, primarily in configuration management)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Document Control 4. 2. 1 General: The quality management system documentation shall include… f. quality system requirements imposed by the applicable regulatory authorities

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Control of Records 4. 2. 4 Control of Records • The documented procedure shall define the method for controlling records that are created and/or retained by suppliers. • Records shall be available for review by customers and regulatory authorities in accordance with contract or regulatory requirements.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Control of Documents and Records Questions on Document and Record Control?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management 4. 3 Configuration Management: The organization shall establish, document and maintain a configuration management process appropriate to the product. Note: Guidance on configuration management is given in ISO 10007

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management What is configuration management? Per ISO 10007: “configuration: interrelated functional and physical characteristics of a product defined product configuration information” “configuration management: coordinated activities to direct and control configuration”

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management What is configuration management? Working level definition: The cumulative actions an organization takes to ensure that the exact status of a unit or component is known at any given time and to ensure that the status is as planned Status refers to the makeup of the unit or component (i. e. which revision is being built? ; what pieces or items are in the unit or component? )

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management 4. 2. 3 Document Control vs. 4. 3 Configuration Management: Document control typically addresses controlling procedures, work instructions, contracts and other such documents Configuration management typically addresses controlling a product print or drawing and items that are used to build the product

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management • While we will address configuration management in some detail, ISO 10007 should be available at your registrar • Per AQMS Standards, configuration management process must be documented (may change in next rev) • Configuration management will move to clause 7. 1 in next revision

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Configuration management actions: Technical and organizational activities comprising: • configuration identification • configuration control • configuration status accounting • configuration auditing (next rev adds configuration planning)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Configuration Planning – Develop a plan – Describe CM actions, responsibilities and authorities throughout product life cycle Configuration Identification – Identify configuration items – Develop a baseline Configuration Control Define a change methodology Define the information storage procedure

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Configuration Status Accounting – Gather & record data on a build – Identify deviations Configuration Audits – Functional Configuration Audit • Verifies that a CI meets performance and functional requirements cited in the configuration documentation – Physical Configuration Audit • Verifies that the as-built unit conforms to product configuration documentation

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Some points an auditor should investigate: • • Is the process defined/documented? Has a CM plan been developed? Is there a methodology to accomplish CSA? Are the steps being accomplished? How are the audits being documented? Are audit findings being acted upon? Have roles and responsibilities been defined? What happens when CM is not maintained?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Some points an auditor should investigate (con’t): • How does configuration management ripple throughout the organization (i. e. purchasing, production, repair)? • What impact does a change in supplier or source of a material/component have on CM activities? • Is the CM system comprehensive enough for the product and/or contract flow down requirements? • How are drawings controlled, both while in engineering and after release to the program? • How do engineering changes interface with the configuration management process? • At what point does the CM function take control of the configuration from the design function? • How does this transfer take place?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Configuration Management Other questions on configuration management?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement Addresses supplier control, purchasing and verification of purchased product (10 ANAB NCRs in this area in the last three years)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1: The organization shall evaluate and select suppliers based on their ability to supply product in accordance with the organization’s requirements. Criteria for selection, evaluation and reevaluation shall be established. Does a supplier having an ISO 9001 or AS 9100 certification automatically satisfy this requirement?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1: The organization shall be responsible for the quality of all products purchased from suppliers, including customer-designated sources. • What should an organization do if a customer directed supplier is not performing as needed? • How should an organization proceed if a customer directs use of a supplier that the organization would not otherwise use?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1: The organization shall: a. Maintain a register of approved suppliers that includes the scope of approval How is scope of approval determined? – An example

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control To identify scope, we divide our suppliers by commodities. The commodities we use are:

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control Hardware and Software

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control How can scope of approval be cited? • Specific part numbers • Unique item numbers • Commodities – Let’s look at commodities

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control Commodity List AC – Adhesives and Coatings CB - Circuit Boards CH – Chemicals CS – Consumables DI – Distributor ML - Metals MP – Manufactured Parts PL – Plastics TP – Turned Parts Is this list sufficient?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control Commodity List AC – Adhesives and Coatings Adhesives – pressure sensitive adhesive, composite adhesive Coatings- metal coatings, optical coatings CH – Chemicals Dye, photo resist, cleaning solvents CS – Consumables What does this mean? DI – Distributor How are distributors handled? ML – Metals Round stock, bar stock, sheet stock, aluminum, steel, gold, platinum MP – Manufactured Parts Lathe, extrusion, mold Is this list sufficient?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1 b Periodically review supplier performance; records of these reviews shall be used as a basis for establishing the level of controls to be implemented How does this apply to customer directed suppliers? What are some examples of controls used for suppliers?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1 c Define the necessary actions to take when dealing with suppliers that do not meet requirements How does this apply to customer directed suppliers? What are some examples of actions that can be taken?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1 d Ensure where required that both the organization and all suppliers use customer-approved special process sources How can we tell when this is required? How can this requirement be met without flowing it down to suppliers?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Supplier Control 7. 4. 1 e Ensure that the function having responsibility for approving supplier quality systems has the authority to disapprove the use of sources What must be in place before we can ensure this requirement is met?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Purchasing 7. 4. 2 d. Include positive identification of product, applicable specs, drawings, process requirements, inspection instructions & other relevant technical data e. Include requirements for design, test, examination and related instructions for organizational acceptance f. Include requirements for test specimens (production method, number, storage conditions) for design approval, inspection, investigation or audit g. Include NCM (nonconforming material) notification requirements (supplier notification to organization and arrangements for organization approval of supplier NMC)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement - Purchasing 7. 4. 2 h. Include supplier notification to organization of product or process definition change and obtain organizational approval if required i. Include right of access by organization, customer, regulatory authorities to all involved facilities and applicable records j. Include supplier requirements flow down to subtier - applicable requirements, key characteristics

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement – Verifying Purchased Product 7. 4. 3 Verification of purchased product • • Positive verification must occur before supplied product is used - or release under positive recall When verification is delegated to the supplier, requirements will be defined and a register of delegations maintained Customer access to suppler & organization’s premises when required to verify subcontracted product meets requirements Customer verification is not evidence of good supplier control, does not relieve organization’s responsibility for supplying acceptable product & does not preclude customer rejection

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Procurement Questions or Comments on Procurement?

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Summary • What should auditors do to be effective? – Understand the requirements to which the audit must be conducted – Research/get training in topics they do not understand – Understand the processes and interactions and involvement of all stakeholders – Understand what objective evidence is required for an auditee to show conformance with a requirement – Accomplish fair, unbiased, detailed audits to the established criteria – Avoid “soft” auditing or “glazing over” topics – Continue to gather information/data until a conclusion can be reached – Review their actions and results to ensure effectiveness

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Summary • Existence • Adequacy • Conformance • Effectiveness

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Auditing to Sufficient Depth Product Acceptance Software

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” CMSC Overview Product Acceptance Software (PAS) and Quality System Requirements - Taking Ownership of One's PAS

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Agenda • • Product Acceptance Software Defined Processes Affected PAS Requirements/Guidelines Mathematical Computation Validation Methods Validation Summary What can you do and why Questions

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Product Acceptance Software Defined • What is Product Acceptance Software (PAS) – PAS is considered software that performs product or tooling acceptance without subsequent inspection – Software used to manage, store, or translate the sole-source authority copies of digital type design, and is able to impact the integrity of the type design – PAS may be commercial off the shelf software

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Processes Affected – Common PAS applications include • CMM software • PCMS applications (Laser Trackers, Laser Radar, Laser Scanners, Photogrammetry, PCMM’s) • Optical Laser Templates • CAD data translators • CAD Analysis Software • Post processors

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Requirements • FAA AC 21 -36 (1993) Quality assurance controls for product acceptance software. (FAA Req’mnt) • AS 9100 Quality management systems – Aerospace requirements – Section 7. 6 • D 6 -51991 Quality Assurance Standard for Digital Product Definition at Boeing Suppliers – Section 3. 0

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Guidelines • ISO 10012 Requirements for measurement processes and measuring equipment (Guidelines) – Section 6. 2. 2 • ASME B 89. 4. 10 Methods for performance evaluation of coordinate measuring system software (Guidelines) • ARP 9005 Aerospace Guidance for Non. Deliverable Software

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Requirements • FAA AC 21 -36 (1993) Quality assurance controls for product acceptance software. (FAA Req’mnt) – Provides information and guidance concerning control of software – “This AC addresses only those sections of FAR Part 21, Subparts F, G, X and O where information on CAM/CAI/CAT software would be helpful”.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Requirements • AS 9100_rev B_Sect 7. 6 f – When used in the monitoring and measurement of specified requirements, the ability of computer software to satisfy the intended application shall be confirmed. This shall be undertaken prior to initial use and reconfirmed as necessary. NOTE: See ISO 10012 -1 and ISO 10012 -2 for guidance. (Replaced by ISO 10012 rev 2003)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Requirements • D 6 -51991 section 3. 1. 4 (Boeing) – Supplier PAS must be verified prior to product acceptance use. – The supplier will establish and maintain a procedure independent of the software developer to determine that the software, and subsequent revisions, accomplishes its intended function.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Guideline • ISO 10012_rev 2003_Sect 6. 2. 2 Software used in the measurement processes and calculations of results shall be documented, identified and controlled to ensure suitability for continued use. Software, and any revisions to it, shall be tested and/or validated prior to initial use, approved for use, and archived. Testing shall be to the extent necessary to ensure valid measurement results.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Guideline • ASME B 89. 4. 10 -2000 – Methods for Performance Evaluation of Coordinate Measuring System Software • This document provides guidelines for evaluating the quality of solutions generated by CMS software and is concerned with testing the behavior of algorithm implementation. • This standard allows for either a “Digital Test” or a “test work piece“ (artifact) can be used in lieu of the listed tests.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” PAS Guideline • ARP 9005 Section 7 Verification & Validation – Verification is the process of evaluating developed software to assure meeting input requirements at the end of a development stage. Verification can occur at various times during software development. Verification includes review, analysis, inspection, and test. – Validation is the confirmation that the software requirements are fulfilled and the software works as intended in the target environment. Validation determines that the integrated software operates correctly, completely, and consistently with specifications and requirements.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Mathematical Computation What is required? • The ability of computer software to satisfy the intended application shall be confirmed. • Testing shall be to the extent necessary to ensure valid measurement results. • The supplier will establish and maintain a procedure independent of the software developer to determine that the software, and subsequent revisions, accomplishes its intended function.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Algorithms Testing • The algorithms are the mathematical calculations that the software uses to produce results or controls measurement hardware • Common algorithms that require testing include: – – – GD&T functions Shapes fits Best Fits/Least Squares Orientation Temperature compensation CAD translations Software that controls measurement hardware

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Algorithm Testing • ASME B 89. 4. 10 -2000 describes a methodology for testing PAS algorithms using 2 sets of point data. One dataset represents a geometric shape. When this data is analyzed by the PAS, the second set of data describes nominal values to validate the PAS algorithms analyzed the shape correctly.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Example of validation of circle algorithm with point dataset pairs obtained from NIST

Measurement Software analysis results Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Known NIST “Fit Dataset” • Values match !! • This algorithm has been validated.

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Validation Summary • Once the software has been validated; – Document the revision level tested – Maintain records of validation for each version of software tested – Validate and document any software revisions

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” History

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Expectations • Comprehensive assessment of suppliers to AS 9100 section 7. 6 • Consideration/Review of customers requirements. – D 6 -51991 (BOEING) – SQR 004 (Vought) – MAA 1 -10009 -1 (Spirit)

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Review • Requirements – FAA AC, AS 9100, D 6 -51991 • History – SER’s • Expectations – AS 9100 7. 6 – Customer Requirements

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” FAA Concerns • • Configuration control and traceability for Boeing Datasets and all derivatives (NC programs, CMM programs, Screen prints, Inspection plans, OLT’s etc). Verification of data translations Product Acceptance Software verification (PAS). Calibration records Internal Audit records for DPD/MBD Supplier control records (Approval, Maintenance and Flow down) FAI’s Training Records that apply to DPD/MBD

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” QUESTIONS

Registration Management Committee (RMC) Workshop – July 14 -15, 2008 “Auditing to Sufficient Depth” Contributors • • Sonny Crile – DPD PQIT Team Member Mike Sawyer – Tooling/DPD Rep Rick Harcourt – DPD PQIT Team Member Randy Becker – DPD PQIT Team Member