21 Flavors of Software Audits G Gordon

“ 21” Flavors of Software Audits G. Gordon Schulmeyer, CDP PYXIS Systems International, Inc. www. pyxisinc. com ggs@pyxisinc. com 410 -729 -0416 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 1

“ 21” Flavors of Software Audits • • • What is a software audit Software audit preparation Flavors of software audits Software audit reporting Software audit ramifications Conclusions 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 2

What is a software audit • Audit may be applied to any type of standard (including software or IT) • Do auditing of software processes • Internal audits should look for conformance to ISO, to internal software standards, and an effective system 1 • Audit for an effective system • Integrate audits in an organization (example systems and software) • Apply audits at a strategic level (quality at same time as configuration management as development as test)2 1 Modified from All About Auditing, Russell, J. P. , Quality Progress, May 2000 2 Modified from The Future of Auditing, Okes, Duke, APLOMET, 2000 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 3

Software audit preparation 1. Ensure that critical documentation is available (Do not try to hide, misplace, or make documents unavailable to the Auditor) 2. Ensure that all affected parties are familiar with, and properly trained in standards being audited 3. Prepare an Audit Agenda 4. Generally use a pre-defined audit checklist 5. Go into all audits enthusiastically 6. As the auditor, prepare yourself in the requirements and regulations 7. Feel confident of your auditing ability and the company’s ability to meet and exceed the audit requirements 1 Modified from 4/18/2004 Preparing for a Computer Systems & 21 CFR Part II Audit, www. auditing. com Copyright © 2004 PYXIS Systems International, Inc. 4

Flavors of software audits Major Categories • Software development (Engineering) audits • Software support (Information Technology – IT) audits • Hybrid - Internal software / IT company standards audits 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 5

Software development (engineering) audits • CMM®-based • CMMI®-based • ISO-based • Internal software standards-based 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 6

CMM®-based (Nearly Obsolete) • Checklists / Questionnaire • SPA [Software Process Assessment] • CBA-IPI [CMM-Based Assessment for Internal Process Improvement] • SCE [Software Capability Evaluation] 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 7

CBA-IPI [CMM-Based Assessment for Internal Process Improvement] • Follows CAF [CMM Appraisal Framework] – A CAF compliant appraisal method shall identify an asset (such as guidance for a particular activity) that must be provided during a CAF compliant appraisal 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 8

CBA-IPI [CMM-Based Assessment for Internal Process Improvement] Plan & Prepare Appraisal Perform Appraisal Report Appraisal Results 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 9

SCE [Software Capability Evaluation] • SCE Method V 3. 0 is a CAF-compliant method • SCEs are used: – as a discriminator to select suppliers – for contract monitoring – for evaluation of internal processes. 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 10

CBA-IPI vs. SCE 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 11

®-based CMMI Three Classes of Appraisals Characteristics Class C Class B Class A (SCAMPI) Pulse taking Amount of objective evidence Ratings generated Resource needs Team size 4/18/2004 Mini-appraisal Benchmarking Low Medium High Documents / Interviews / Presentations No Low Small No Medium Yes High Large Copyright © 2004 PYXIS Systems International, Inc. 12

Class B & C Outputs Required Completed Appraisal Disclosure Statement (ADS) Expected Strengths and/or Weaknesses Data collection technique Practice used Implementation Characterizations Data collection sessions conducted Appraisal Plan: time, effort, cost; model coverage; organizational coverage 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. Optional Detailed Data Summaries Orientation / Training 13

Class A - SCAMPI • Appraisal Requirements for CMMI, Version 1. 1 (ARC, V 1. 1)* – Defines each requirement for an A, B or C appraisal • Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Version 1. 1: Method Definition Document (MDD) – The MDD describes the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) * ARC requirements address the intent of assessment requirements levied by ISO/IEC TR 15504 -3 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 14

Class A – SCAMPI MDD Contents 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 15

Three Classes of ARC Appraisals Class A • Full, comprehensive method • Thorough model coverage • Provides maturity and/or capability levels Class B • Less comprehensive, less expensive • Initial, partial, self-assessment • Focus on areas needing attention • No rating Class C • Quick look • Checks for specific risk areas • Inexpensive, little training needed Revised from Software Productivity Consortium (SPC) 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 16

ISO-based – ISO 9001, Quality Systems - Model for Quality Assurance in Design, Development, Production, Installation, and Servicing auditing done on all elements integral to the products – Tick. IT (ü Information Technology) on the development of software integral to products – Tick. IT auditor is specifically knowledgeable in software development 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 17

Tick. IT (ü Information Technology) • Verifies that software design, development, production (replication, duplication, etc. ), installation and servicing (maintenance) comply with ISO 9001 • Uses for guidance – ISO 9000 -3, Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software – The Tick. IT Guide - A Guide to Software Quality System Construction and Certification Using ISO 9001: 2000 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 18

ISO-based • External ISO Audit 1. Perform planning with registrar responsible for the ISO audit 2. Focus on software readiness for Tick. IT audit 3. Prepare staff for what to expect from the ISO audit 4. Usually provide a presentation to start the project audit 5. Concisely respond to the Tick. IT auditor 6. Take results and do follow-up for corrective actions as necessary 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 19

ISO-based • Internal ISO Audit 1. Allocate ISO 9001 clauses to auditors 2. Interpret the ISO 9001 clause in light of the guidance form ISO 9000 -3 3. Determine if the activity or work product(s) are compliant with ISO 9001 4. Report results with reference to appropriate ISO paragraph 5. Perform follow-up to assure corrective actions performed in a timely manner 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 20

Internal software standards-based [Hybrid] 1. 2. 3. 4. 5. 6. Document Review of software standards Prepare exploratory questions Interviews Consolidate data Findings presentation Package results 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 21

1. Document Review of software standards 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 22

2. Prepare exploratory questions • Based upon the results of the document review, questions are prepared to better discover if the standards are understood and being used 3. Interviews • Conduct interviews for clarification of items reviewed (usually takes 0. 5 to 1 hour / person) 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 23

4. Consolidate data • Based upon the document review and the interviews the team meets to decide where there are findings 5. Findings presentation • Findings are drafted with the team focused on the internal software standards for reference 6. Package results • A Final Report is written and provided to the project manager with an indication of the corrective action(s) due date 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 24

Software support (Information Technology – IT) audits • Software licensing • System security • System usage • Internal IT standards-based 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 25

Software licensing • Business Software Alliance (BSA) • Software & Information Industry Association (SIIA) • Either: – Download free software tool (Key Audit) to do internal audit of whether software is legal or not – Suspected software license infringement from BSA, SIIA, or Vendor (serious) How to Survive a Software Audit through Effective Software Management, Tomeny, John, 2002, www. sassafras. com 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 26

System security Software security auditing objectives are: Verify correct functioning of a software product from a security standpoint, in terms of - problems and/or errors which may have occurred during its implementation, or - its relationship to the system’s other components 1 1 Software 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. Security Auditing, Final Report 27

System security • Software security auditing methods are: – Tests to • try to penetrate security measures • obtain greater privileges • compromise the correct functioning of the application or security domains 1 (Success or Failure of logins, file accesses, deleting files, changing security settings Recommend auditing only servers and domain controllers [login failures])2 1 Software 2 Creating 4/18/2004 Security Auditing, Final Report an Audit Policy, Posey, Brian, Nov. , 2000, http: //networking. earthweb. com Copyright © 2004 PYXIS Systems International, Inc. 28

System usage • CIO looks at: – Identify all software in use on organization’s computers – Assess actual quantity and type of software needed by the org. (see next side) Step-by-Step Guide to Managing Your Organization’s Software, Business Software Alliance, www. bsa. org, 2004 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 29

System usage survey • List software programs used most often in your work ___________________ hours per day _________ • What software do you need that you do not currently have • What software do you want that you do not currently have • Are you currently using any work software on your home computer? • Are you currently using any personal software on your office computer? • Provide any relevant comments here: Step-by-Step Guide to Managing Your Organization’s Software, Business Software Alliance, www. bsa. org, 2004 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 30

Internal IT standards-based [Hybrid] 1. 2. 3. 4. 5. 6. Document Review of IT standards Prepare exploratory questions Interviews Consolidate data Findings presentation (Same as steps Package results 4/18/2004 for software based) Copyright © 2004 PYXIS Systems International, Inc. 31

Software audit reporting • Provide an out briefing • Provide an Audit Report • Every finding must be tied to a software or IT requirement (Don’t let them bite your head off) • Provide Opportunities for Improvement 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 32

Software audit ramifications re / IT he softwa Makes t 1. etter process b tion of to correc Leads ered in 2. es uncov i deficienc / IT software greater tion has Organiza oftware 3. eness of s awar ents requirem / IT back ave to go jects h Some pro ng board to 4. awi to the dr iance ve compl achie 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 33

Conclusions Flavors of software audits: • Categories [software development, software support, hybrid-internal software / IT company standards audits] • Types [SCAMPI, CBA IPI, SCE, ISO, Internal] 4/18/2004 Copyright © 2004 PYXIS Systems International, Inc. 34