Americas Aerospace Quality Group Registration Management Committee Boeing

Americas Aerospace Quality Group Registration Management Committee Boeing Supplemental Oversight Assessment Results January 22, 2008 Presented by: Brian Simons Boeing Oversight Representative The Boeing Company ׀

Topics for discussion Boeing supplemental oversight process § Assessment Activities § Strengths and Issues § Corrective Action Expectations Assessment results § Statistics § Significance § Corrective Action Responses Questions and comments 2 ׀

Boeing Supplemental Oversight Process Supplemental assessment activities: § Semi-annual CB performance review to identify 6 month planned activity § Risk-Based assessment for each CB using: - Aggregate client (supplier) performance (product and QMS) - Previous Boeing oversight activity - Total quantity and change of client base § Ad Hoc or unplanned (Throughout the year) - Primarily based on unsatisfactory supplier performance - BOR (Boeing Oversight Representative) Support Requests - Management requests - Field Representative questions and inquiries § Approximately 80% of Boeing’s supplemental oversight is Ad Hoc 3 ׀

Boeing Supplemental Oversight Process Y Axis = Influence Jan – June 2008 - America’s Sector - Oversight Opportunities X Axis = Performance 4 ׀

Boeing Supplemental Oversight Process Types of supplemental oversight activity: § OASIS database and certificate information validation § Audit report review (Audit summary, AS 9101 checklist, nonconformances, corrective action, etc. ) § Client validation (on-site, including audit report review) - Required for risk CBs identified on RAT § CB auditor witness audit § CB/ANAB office audit 5 ׀

Boeing Supplemental Oversight Process Strengths and Issues (S&I) document: § Issue (CA required) or Comment Issues documented in a 3 part format: § Requirement – Reference clause number and verbatim statement of requirement (e. g. AS 9104 clause 8. 4 The AQMS audits shall be documented on the Aerospace Assessment Report…. ). § Issue (Nonconformance) – Restatement of requirement in the negative (e. g. The AQMS audits are not documented on the …. ) § Evidence – Documentation of evidence collected, proceeded by “For example” (e. g. For example, no evidence of completed AS 9101 checklist for surveillance assessment # 123, dated 1/22/08) The BOR team will begin providing ANAB with a courtesy copy of all S&Is issued to CBs beginning 2008 6 ׀

Boeing Supplemental Oversight Process Sample S&I - Issue #1 Requirements: AS 9104 clause 8. 3 Nonconformity: No certificates or approvals to AQMS Standards or any combination of AQMS Standards with ISO 9001 shall be issued unless all major and minor nonconformities have been satisfactorily corrected with the root cause analysis and corrective action verified by the CRB. IAF Guidance on the Application of ISO/IEC Guide 62: G. 3. 8. 5. The implementation of the corrective action should not be deemed to have been completed until its effectiveness has been demonstrated and the necessary changes made in the procedures, documentation and records. Issue: The CRB does not verify the satisfactory implementation, completion and effectiveness of corrective action for nonconformities, including demonstration that the necessary changes were made in the procedures, documentation and records, as necessary. Evidence: For example, The client corrective action for NCRs #2 and #3 from the recertification audit were not completed prior to CB X closing the NCRs. NCR #2 client corrective action states that inspector training will be completed by February 15, 2007. The training record provided by the client for this action is dated 4/4/2007. NCR #2 was closed on 2/5/07. NCR #3 client corrective action states that before February 10, 2007 the client will implement a revision to the procedures manual addressing AS 9100 clause 7. 5. 1. 3. As of November 1, 2007, there is no evidence of a revision to the client's Quality Procedure Manual for clause 7. 5. 1. 3 (Procedure does not exist) or the Quality Assurance Manual (current revision December 2003). NCR #3 was closed on 2/5/07. 7 ׀

Boeing Supplemental Oversight Process S&I Corrective Action expectations (5 part format): § Immediate Correction - Address and correct the evidence - Identify the Direct Cause - Notify affected personnel § Root Cause Statement - Identify fundamental process deficiency - Personnel error or mistake is unacceptable § Root Cause Corrective Action (CA) Plan - Address the Root Cause - Identify what, who and when - Include reference to revised QMS policy, process, etc. (title/number) 8 ׀

Boeing Supplemental Oversight Process S&I Corrective Action expectations (5 part format): (Continued) § Verification of Corrective Action Plan - Verify RCCA actions have been completed per commitments - QMS process updated, training completed, clients assessed, etc. § Follow-up audit - Determine of CA plan is effective in preventing recurrence - Conducted 6 -9 months following verification 9 ׀

Boeing Oversight Assessment Results BOR oversight activity from January 1, 2006 through December 31, 2007: § 19 out of 34 domestic Certification Bodies Assessed (CBs associated with Boeing suppliers) § CBs assessed based on RAT and Ad Hoc requests § 69 out of 79 oversight activities resulted in at least 1 issue (87% rejection rate) § 32 - OASIS database and Certificate validations § 26 - Client Validations (includes audit report review) § 9 - Audit Report Reviews (Desktop review) § 1 - Office Audit § 1 - Witness Audit § 143 total issues identified (Nonconformance to industry requirement) 01 ׀

Boeing Oversight Assessment Results 11 ׀

Boeing Oversight Assessment Results Supplemental oversight from January 1, 2006 through December 31, 2007 : § Inadequate QMS assessment (31) § OASIS update > 30 days (21) § Audit report deficiency (16) § Multi-site issue (13) § Assessment days below minimum requirement (11) § CB contract deficiency (7) § Certificate deficiency/error (7) § Certificate issued with incomplete client corrective action (7) § NC soft grading (6) § OASIS errors (5) 21 ׀

Boeing Oversight Assessment Results Supplemental oversight from January 1, 2006 through December 31, 2007 : Certificate validity § Inadequate QMS assessment (31) § Audit report deficiency (16) § Multi-site issue (13) § Assessment days below minimum requirement (11) § Certificate issued with incomplete client corrective action (7) § NC soft grading (6) CB Administration § OASIS update > 30 days (21) § CB contract deficiency (7) § Certificate deficiency/error (7) § OASIS errors (5) 31 ׀

Boeing Oversight Assessment Results Inadequate Quality Management System Assessment (31): § Ineffective QMS management processes (Client validation) - Management Review, Quality Objectives, Internal Audit, Corrective Action, Analysis of Data, Continual Improvement, etc. § Ineffective product control processes (Client validation) - Configuration Management, Production Documentation, Identification, Accountability, Control of Nonconforming Product, etc. § Completed audit reports lack sufficient evidence to justify “satisfactory” result - Prompts Boeing oversight investigation and results comparison § Unsatisfactory Boeing supplier performance - Compare Boeing 2 nd party QMS audit results vs. CB audit results - Prompts Boeing oversight investigation 41 ׀

Boeing Oversight Assessment Results Certification Body Corrective Action Response: § Corrective action response due within 30 days (typical) § Initial submittal rejection rate approx. 90 -100% - 5 part format not followed Evidence cited in S&I not addressed and corrected in Immediate Action Root Cause identifies human error or mistake Root Cause is an explanation or rationalization of the condition Root Cause Corrective Action does not address long term process improvement CA Verification and Follow-up are confused and mixed together § BOR policy is to elevate unsatisfactory S&I CA to ANAB for resolution - 2 S&Is were elevated in 2007 for ANAB investigation and action 51 ׀

That’s all folks! § Questions or Comments? 61 ׀