Скачать презентацию 2008 Auditor Workshop Output Matters 14 July 2008 Скачать презентацию 2008 Auditor Workshop Output Matters 14 July 2008

3b079857c30eb066b550da36a16a8ef9.ppt

  • Количество слайдов: 22

2008 Auditor Workshop “Output Matters” 14 July 2008 Dan Conlon 951. 273. 5210 daniel. 2008 Auditor Workshop “Output Matters” 14 July 2008 Dan Conlon 951. 273. 5210 daniel. [email protected] mil

Agenda • AS 9100 Importance to MDA • MDA Assurance Provisions (MAP) • MDA Agenda • AS 9100 Importance to MDA • MDA Assurance Provisions (MAP) • MDA Audit Process • AS 9100 Findings • Analysis • Examples of AS 9100 Audit Findings • Common AS 9100 Issues • How You Can Help • Conclusion 2

AS 9100 Important to MDA • AS 9100 is a key QMS document – AS 9100 Important to MDA • AS 9100 is a key QMS document – MDA Core standard • The MAP builds on AS 9100 requirements/disciplines to create a very rigorous set of QMS implementation requirements • MAP includes additional Quality, Safety, and Mission Assurance (QSMA) disciplines for developing mission and safety critical items such as: – Systems Engineering – Reliability, Maintainability, and Availability – Parts and Materials Management – Software – Risk Management – Configuration Management – Safety and Environmental Management – Interface Management 3

MDA Assurance Provisions (MAP) • A standardized set of 14 QSMA provisions, requirements, and MDA Assurance Provisions (MAP) • A standardized set of 14 QSMA provisions, requirements, and processes for mission and safety critical items: MDA Assurance Provisions (MAP) • 3. 1 Management • 3. 2 Design and Development • 3. 3 Software and Firmware – – Goal is to implement on all MDA contracts • 3. 5 Reliability, Maintainability, and Availability – • Authored by joint Gov/Industry Team 160 pages, 1800+ shall statements • 3. 6 Parts and Materials Control Program Provide methods to measure, verify, and validate mission success through: – Collection and analysis of metrics – Risk Assessment, technical evaluations, independent assessment, reviews – • Conduct of qualification, ground, and flight tests Promote continual process improvement to ensure QSMA, reduce cost, and improve productivity • 3. 4 Technical and Mission Assurance Reviews • 3. 7 Integrated Test and Evaluation Program • 3. 8 Test, Meas. , and Diagnostic Equip. & Stds. • 3. 9 Interface Management • 3. 10 Configuration Management • 3. 11 Control of Nonconforming Items & Materials • 3. 12 Fabrication and Quality • 3. 13 Supplier Management • 3. 14 Safety Proven Mission Assurance Requirements on Contracts 4

MDA MAP Program Usage • As part of the MDA Acquisition Strategy the MAP MDA MAP Program Usage • As part of the MDA Acquisition Strategy the MAP was developed for MDA Program Office implementation and flow down to prime contractors and suppliers – Use existing processes, when adequate, to achieve and sustain mission success – Tailored by the MDA Program Offices for each program – Further tailored for subsystem components through supplier management • Implementation of MAP (including AS 9100) assessed via: – Mission Assurance audits – Supplier audits – On-Site Mission Assurance Representatives (MARs) – Focus Teams – (VAFB Flight Test Team, PMRF, NGST) 5

Why Audit? First MDA/QS Mission Assurance Audit performed 1997 1998 1999 2000 2001 2002 Why Audit? First MDA/QS Mission Assurance Audit performed 1997 1998 1999 2000 2001 2002 2003 2004 Aegis FTR-1 7/00 IFT-01 1/97 THAAD IFT-04 1/00 BV-3 12/01 BV-2 8/01 BV-5 1/04 IFT-10 12/02 IFT-14 2/05 No Interceptor Failures IFT-13 c 12/04 17 Flight Test / 0 Failures FT-08 5/98 FT-07 3/97 2008 FM-5 6/03 IFT-05 7/00 NMD/GMD 2005 2006 2007 (0% failure rate) FT-09 3/99 22 Flight Test / 10 Failures (45% failure rate) QS Organization founded in Late 2002 32 Flight Test / 4 Failures (13% failure rate) Majority of Failures Were Lack of Process Discipline 6

GMD Flight Test-03 a 28 Sept 07 • Engages - Target launch from Kodiak GMD Flight Test-03 a 28 Sept 07 • Engages - Target launch from Kodiak - Interceptor launched VAFB - In midcourse phase of flight - EKV discrimination 7

MDA QS Audit Program • Purpose – Conduct Mission Assurance audits of MDA prime MDA QS Audit Program • Purpose – Conduct Mission Assurance audits of MDA prime contractors and suppliers • MDA Audit Objectives – Validate that the end item at a selected level conforms to engineering design requirements – Validate that the end item at a selected level is qualified – Compare fabrication methods and acceptance test levels against MDA and industry standards as appropriate for MDA use – Validate production readiness – Assess contractor/government Mission Assurance practices, procedures, and implementation • MDA Audit Findings – Deficiency - a non-compliance to a contractual requirement or internal procedure/process • Includes AS 9100 deficiencies (if AS 9100 is on contract or if audit site is certified or compliant) – Observation - when an MDA or Industry “best-practice” should be used 8

All MDA Audits CY 05 -CY 08 9 All MDA Audits CY 05 -CY 08 9

AS 9100 Findings 10 AS 9100 Findings 10

Analysis • Majority of MDA deficiencies can be tied to an AS 9100 element Analysis • Majority of MDA deficiencies can be tied to an AS 9100 element • AS 9100 related deficiencies are primarily in the following areas: – Production and Service Provision: Quality Records, Procedural Discipline, Foreign Object Debris/Damage (FOD) , and Electrostatic Discharge (ESD) – Documentation Requirements: Control of Documents (Change and Approval), Quality Management System Records – Purchasing: Supplier Requirements Flow Down, Supplier Rating System – Design and Development: Inputs/Outputs, Traceability, Design Reviews, Design Verification/Validation – Control of Monitoring and Measuring Devices: Design and Selection, Calibration, and Tool Control – Planning of Product Realization: Establishment and Documentation of Processes – Control of Nonconforming Product: Nonconformance Disposition and Nonconforming Material Identification and Segregation 11

Finding Examples • Production and Service Provision, AS 9100, paragraph 7. 5 – Drawing Finding Examples • Production and Service Provision, AS 9100, paragraph 7. 5 – Drawing parameters and/or pass/fail criteria were not accurately represented in the build paper. – Foreign Object Debris/Damage (FOD) was found in the Building Powder Blend Area. – Limited life materials were expired and available for use in the production of deliverable hardware. – Electrostatic discharge equipment continuity and environmental limits are not adequately controlled in the Bonded Stores and Electrical Inspection Areas. – Desiccant used to minimize moisture in motor manufacturing is not used per Manufacturing and Inspection Records (M&IR) or drawing requirements. – Preventive Maintenance Work Orders are not verified and Repair Work Orders are not generated to correct deficiencies. 12

Finding Examples • Documentation Requirements, AS 9100, paragraph 4. 2 – The audited program Finding Examples • Documentation Requirements, AS 9100, paragraph 4. 2 – The audited program did not implement enterprise company practices for requirement traceability, and did not have an approved process for the unique implementation of their particular practices. – X-Ray inspection procedures are not controlled. – Changes to a Mix and Cast Building checklist were not approved before use. – Receiving Inspection Records were missing required documentation (Inspection report, Government Source Inspection Stamp, Material Certification). – The contractor could not provide supporting information that correlated or related to waivers and deviations. – Uncontrolled and improperly marked documentation were posted in the Process Control Lab. 13

Finding Examples • Purchasing, AS 9100, paragraph 7. 4 – Parts that did not Finding Examples • Purchasing, AS 9100, paragraph 7. 4 – Parts that did not comply with all supplier quality assurance requirements were accepted at Receiving Inspection. – The contractor’s Approved Vendor List does not include some vendors and does not reflect the vendor’s quality rating. – The prime contractor Statement of Work imposes an incorrect quality requirement. – The contractor did not impose core quality requirements (corrective action system, inspection procedures, and quality records) on suppliers. – The contractor accepted a supplier certificate of analysis that certified material for a storage temperature in conflict with the material specification. 14

Finding Examples • Design and Development, AS 9100, paragraph 7. 3 – Contractor Automated Finding Examples • Design and Development, AS 9100, paragraph 7. 3 – Contractor Automated Test Station configuration identification does not reflect the current approved configuration. – The contractor does not have a formal process for documenting and tracking design review action items and responses. • Control of Monitoring and Measuring Devices, AS 9100, paragraph 7. 6 – Calibrated equipment at a work station in the Destructive Lot Acceptance Test Area had broken integrity seals. – Test and Measuring Equipment with expired calibration was in the receiving inspection area. 15

Finding Examples • Planning of Product Realization, AS 9100, paragraph 7. 1 – The Finding Examples • Planning of Product Realization, AS 9100, paragraph 7. 1 – The contractor’s software development practices were not documented and were dependent on the cognizant engineer. – The mass test limit for the Inertial Measurement Unit was incorrectly noted on the test sheet in the Engineering Note Book. • Control of Nonconforming Product, AS 9100, paragraph 8. 3 – The contractor processed “Use as Is” dispositions for hardware without proper review and approval, including the customer. – Undocumented items (various mechanical, electrical and optical parts) were noted in the bonded Material Review Board lockers and cages which appear to be scrap. 16

Common AS 9100 Issues • Interpretation varies between contractors and government organizations – AS Common AS 9100 Issues • Interpretation varies between contractors and government organizations – AS 9100 Paragraph 7. 5. 1. 1, Production Documentation: Production operations shall be carried out in accordance with approved data • MDA Interpretation: Mission and Safety Critical processes shall be carried out in accordance with approved detailed plans and procedures – AS 9100 Paragraph 8. 3, Control of Nonconforming Product: The organization shall ensure that product that does not conform to product requirements is identified and controlled to prevent its unintended use or delivery • MDA Interpretation: Nonconforming product is conspicuously identified (tagged), segregated, and retained in a hold status until dispositioned by formal review process – AS 9100 Paragraph 4. 3 Configuration Management: The organization shall establish, document and maintain a configuration management process appropriate to the product. NOTE: Guidance on configuration management is given in ISO 10007 • MDA Interpretation: A formal Configuration Management Program includes: a CM Plan which addresses Configuration Planning, Configuration Identification, Configuration Change Management (Waivers/Deviations), Configuration Status Accounting, Configuration Verification / Audit, and CM of Digital Data 17

Common AS 9100 Issues Continued • Compliance and interpretation varies from contractor to contractor Common AS 9100 Issues Continued • Compliance and interpretation varies from contractor to contractor – Some contractors manufacture mission and safety critical hardware per an approved drawing, while others use an approved drawing, traveler, and detailed procedures – Contractors advertise AS 9100 Certification, however, implementation varies based on programs and funding • Certification achieved at division level, but not implemented across all programs – MDA experience: Many contractors have stated that AS 9100 requires them to have a documented system, but the documented system does not have to be effective or implemented for compliance. 18

AS 9100 How You Can Help • Strengthen Implementation Requirements – MDA experience: There AS 9100 How You Can Help • Strengthen Implementation Requirements – MDA experience: There is a gap between system assessment and implementation. AS 9100 certification should include assessment of critical processes and the associated detailed planning and procedures. • Audit/Assess MDA Programs as part of Certification – Identify MDA Programs with MAP on Contract – Assess effectiveness of Corrective Actions from MDA Audit Findings – Provide feedback for high risk MDA Suppliers 19

Conclusion • Majority of MDA deficiencies can be tied to an AS 9100 Element Conclusion • Majority of MDA deficiencies can be tied to an AS 9100 Element • Most audit findings are violations of internal processes – Not following their own procedures or processes • Our audits have indicated gaps in PDCA implementation – Most sites do the “Plan” portion – They do less well with the “Do” portion – Even do less well with the “Check” portion – Most “Act” actions are short term, focusing on immediate fixes rather than well thought out long term actions that address systemic issues and improve performance across the enterprise Output of Requirements Must be Defined 20

Backup Backup

MDA Audit Findings Nonconformances MAP to AS 9100 Correlation 22 MDA Audit Findings Nonconformances MAP to AS 9100 Correlation 22