- Количество слайдов: 88
Quality Assurance & Improvement Program: Audit Process Versus Program: The Difference…and Why It Matters Presenter: Brian E. Kruk, CIA, CCSA, CGAP, CCA, CISA Director Contract and Construction Audit Union Pacific Railroad Topeka Chapter April 5, 2016
Today’s Agenda • A brief history of QA • Discuss the available QA&IP guidance • Examine common misconceptions in QA&IP development • Explore the differences between basic internal audit processes and effective components of a QA&IP • Utilization of the Old IIA PA 1311 -2 to create an appropriate, right-sized QA&IP • Understand how a CMM can be used to facilitate the path to quality
Today’s Focus - Has anyone recently completed a QA? - Has anyone performed as a Validator? - Is anyone working on their Internal Assessment or Self Assessment? - What do you want out of today’s session? - Are there any questions before we begin?
“Quality is not an act – it is a habit. ” ~Aristotle “Quality means doing it right when no one is looking. ” ~ Henry Ford
Quality Assessment Defined The process of evaluating the efficiency and effectiveness of an internal auditing organization through a comprehensive, qualitative review of audit procedures, leading to recommendations for improving controls, reducing risk and the introductions of successful innovative best practices. It should also provide assurance conformity with the International Standards for the Professional Practice of Internal Auditing and other relevant organizational and departmental policies and procedures.
Synopsis of QA History - Other professions have required peer reviews - IIA first publication on QA in 1984 - IIA recommended peer reviews in previous Standards - IIA began conducting QAs in 1986 - Some QAs also conducted by other providers - GTF Brings Focus to Quality Initiative - QA Manual, 4 th Edition, released in 2002 - QA Manual, 5 th Edition, released in 2006 - QA Manual, 6 th Edition, released in 2009 - QA Manual, 7 th Edition, released in 2013
A Vision for the Future: Professional Practices for Internal Auditing Report of GTF to IIA Board of Directors – Adopt New Framework – Revise Definition of IA – Update Code of Ethics and Standards – Establish Oversight Committee – Develop Guidance to Support the Standards
Professional Practices Framework - 2002 The “Path to Quality” gets its formal start with the creation of: 7 New Quality Standards & 5 Practice Advisories OH 2 -3
Continuous Improvement Highlights Onward and Upward
Continuous Improvement Highlights Examples of Shortfalls • Addressing the applicability of the Standards for specialty groups • Further clarification of Assurance & Consulting services • Need for some level of basic fraud (Red Flags) • Knowledge of key IT risk, controls and technology-based audit techniques • Periodic Internal and External QA and ongoing monitoring as part of QA&IP • Inclusion of overall opinion and/or conclusion where appropriate, in final communications
Continuous Improvement Highlights • By January 2004 -24 changes to the PPF • 11 New Standards • 13 Additions to Glossary • 11 New Practice Advisories • 5 Revisions to PA’s
Continuous Improvement Highlights July 2007 - Arrival of the New International Professional Practice Framework
Continuous Improvement Highlights • By the end of 2009 - changes to the IPPF • • 6 New Standards 19 New Interpretations 13 Additions to Glossary Practice Advisories reduction to 58 3 New Practice Guides, New 13 GTAG’s New 3 GAIT’s
Continuous Improvement Highlights • 2010 to 2011 - changes to the IPPF • • 3 New 1 Deleted 15 Revised Standards 9 New and Revised Interpretations 5 Revisions to Glossary 13 New Practice Advisories 8 New Practice Guides, 3 New GTAG’s
Continuous Improvement Highlights
Continuous Improvement Highlights The New IPPF • Mandatory Guidance – – Core Principles Standards DIA COE • Recommended Guidance – Implementation Guidance – Supplemental Guidance (PGs, GTAGs, & GAITs)
IIA - Core Principles • • • Demonstrates integrity. Demonstrates competence and due professional care. Is objective and free from undue influence (independent). Aligns with the strategies, objectives, and risks of the organization. Is appropriately positioned and adequately resourced. Demonstrates quality and continuous improvement. Communicates effectively. Provides risk-based assurance. Is insightful, proactive, and future-focused. Promotes organizational improvement.
Attribute Standards address the attributes of organizations and individuals performing internal auditing. - 1000: Purpose, Authority and Responsibility - 1100: Independence and Objectivity - 1200: Proficiency and Due Professional Care - 1300: Quality Assurance and Improvement Program
Performance Standards describe the nature of internal auditing and provide quality criteria against which the performance of these services can be measured. - 2000: Managing the Internal Audit Activity 2100: Nature of Work 2200: Engagement Planning 2300: Performing the Engagement 2400: Communicating Results 2500: Monitoring Progress 2600: Management’s Acceptance of Risks
QA Related Standards 1300: Quality Assurance and Improvement Programs The chief audit executive should develop and maintain a quality assurance and improvement program that covers all aspects of the IAA and continuously monitors its effectiveness. The program should be designed to help the internal auditing activity add value and improve the organization’s operations and to provide assurance that the internal audit activity is in conformity with the Standards and the Code of Ethics. Note: 2 nd half drop in new Standard: See Interpretation next slide
QA Related Standards Standard 1300 – Interpretation A quality assurance and improvement program is designed to enable an evaluation of the IAA’s conformance with the Standards and an evaluation of whether internal auditors apply the COE. The program also assesses the efficiency and effectiveness of the IAA and identifies opportunities of improvement.
QA Related Standards Original 1310: Quality Program Assessments The internal audit activity should adopt a process to monitor and assess the overall effectiveness of the quality program. The process should include both internal and external assessments. Revised 1310 – Requirements of the QA&IP The QA&IP must include both internal and external assessments.
QA Related Standards Original 1311 - Internal Assessments Should include: • Ongoing reviews of the performance of the IAA; and • Periodic reviews performed through self-assessment or by other persons within the organization, with knowledge of internal auditing practices and the Standards. Revised 1311 - Internal Assessments must include: • Ongoing monitoring of the performance of the IAA; and • Periodic self-assessment or assessments by other persons within the organization with sufficient knowledge of internal audit practices.
QA Related Standards 1311 - Internal Assessments Interpretation: Ongoing monitoring is an integral part of the day-to-day supervision, review and measurement of the IAA. Ongoing monitoring incorporated into the routine policies and practices used to manage the IAA and uses processes, tools and information considered necessary to evaluate conformance with the DIA, COE and Standards. Periodic reviews are assessments conducted to evaluate conformance with the DIA, COE and Standards. Sufficient knowledge of IA practices requires at least an understanding of all elements of the IPPF.
QA Related Standards Original 1312: - External Assessments External assessments such as quality assurance reviews, should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization.
QA Related Standards Revised 1312: External Assessments External assessments should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization. The potential need for more frequent external assessments as well as the qualifications and independence of the external reviewer or review team, including any potential conflict of interest, should be discussed by the CAE with the Board. Such discussions should also consider the size, complexity and industry of the organization in relation to the experience of the reviewer or review team.
QA Related Standards Current 1312 : External Assessments External assessments must be conducted at least once every five years by a qualified independent assessor or assessment team from outside the organization. The CAE must discuss with the board: • The form and frequency of external assessment; and • The qualifications and independence of the external assessor or assessment team, including any potential conflict of interest.
QA Related Standards 1312 - External Assessments Original Interpretation: A qualified reviewer or review team consists of individuals who are competent in the professional practice of internal auditing and the external assessment process. The evaluation of the competency of the reviewer and review team is a judgment that considers the professional internal audit experience and professional credentials of the individuals selected to perform the review. The evaluation of qualifications also considers the size and complexity of the organizations that the reviewers have been associated with in relation to the organization for which the IAA is being assessed, as well as the need for particular sector, industry, or technical knowledge. An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs.
QA Related Standards 1312 - External Assessments Revised Interpretation: A qualified reviewer or review team demonstrates competence in two areas: the professional practice of internal auditing and the external assessment process. Competence can be demonstrated through a mixture of experience and theoretical learning. Experience gained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of a review team, not all members of the team need to have all the competencies; it is the team as a whole that is qualified. The CAE uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified. An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs.
QA Related Standards 1312 - External Assessments Proposed Interpretation: External assessments enhance a complete QA&IP and may be accomplished through a full external assessment, or a selfassessment with independent validation. The external assessor must conclude as to conformance with the Standards; the external assessment may also include operational and strategic comments. 3 rd paragraph adjustments “ real or an apparent ” changed to read “ actual or a perceived” conflict of interest ____* Added 3 rd sentence: The CAE should encourage board participation in the QA&IP to reduce perceived or potential conflicts of interest.
QA Related Standards Original 1320 – Reporting on Quality Program The chief audit executive should communicate the results of external assessments to the board. Revised 1320 – Reporting on Quality Program The CAE must communicate the results of the QA&IP to senior management and the board. Review interpretation narrative
QA Related Standards 1320 - Reporting on the QA&IP Interpretation: The form, content and frequency of communicating the results of the QA&IP is established through discussions with the senior management and the board and considers the responsibilities of the IAA and CAE as contained in the IA Charter. To demonstrate conformance with the DIA, the COE, and the Standards, the results of external and periodic internal assessments are communicated upon completion of such assessments and the results of ongoing monitoring are communicated at least annually. The results include the assessor’s or assessment team’s evaluation with respect to the degree of conformance.
QA Related Standards Original -1330: Use of “Conducted in Accordance with the Standards” Internal auditors are encouraged to report that their activities are “conducted in accordance with the International Standards for the Professional Practice of Internal Auditing. ” However, internal auditors may use the statement only if assessments of the quality improvement program demonstrate that the internal audit activity is in compliance with the Standards. Current -1321: Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing” Indicating that the IAA conforms with the ISPPIA is appropriate only if the results of the QA&IP supports such a statement.
QA Related Standards Original 1340: Disclosure of Noncompliance Although the IAA should achieve full compliance with the Standards and internal auditors with the Code of Ethics, there may be instances in which full compliance is not achieved. When noncompliance impacts the overall scope or operation of the IAA, disclosure should be made to senior management and the board. Current – 1322: Disclosure of Nonconformance When nonconformance with DIA, the COE, or the Standards impacts the overall scope or operation of the IAA, the CAE must disclose the nonconformance and the impact to senior management and the board.
QA Related Practice Advisories • 1300 - 1 Quality Assurance & Improvement Program • 1310 – 1 Requirement of the QA&IP (Deleted from IPPF) • 1311 - 1 Internal Assessments • 1311 - 2 Internal Assessment: Establishing Measures to Support Reviews of IAA (Deleted from IPPF) • 1312 - 1 External Assessments • 1312 - 2 External Assessment- SAWIV • 1312 - 3 Independence of External Assessment Team – Private • 1312 - 4 Independence of External Assessment Team – Public • 1320 -1 Reporting Results of QA&IP • 1321 - 1 Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing” • 1322 - 1 Disclosure of Nonconformance w/ the ISPPIA • 2120 - 2 Managing the Risk of the IAA
QA Related Practice Advisories PA 2120 - 2 Managing the Risk of the Internal Audit Activity 1. Managing the risk of not achieving IA Objectives 2. IAA must manage its own risk 3. 3 categories: audit failure, false assurance, and reputation risks 4. Where were the internal auditors? 5. IAA can implement the practices to mitigate its risk: – – – QA&IP Periodic reviews of audit plan Effective planning Effective audit design Effective management review and escalation Proper Resource Allocation 6. 6 through 14 - additional topics of further guidance
External Assessments Areas of focus: - Review IA Activity’s charter, audit plans, policies and procedures - Review a sample of audit reports, special projects and supporting work papers - Review staff composition, supervision, professional development and response to client needs
External Assessments Areas of focus: - Assess staff and client satisfaction through interviews and surveys - Specifically interview audit committee chairperson, a representative sample of officers, senior executives and management clients and the external auditing partner - Risk assessment methodology - Approach and adequacy of IT audit coverage
External Assessment Activities Tools Review • Self Study/Benchmarking • Customer/Staff Survey • On-site Activities – Interviews (Board, Management, External Auditor, Staff) – QA Program – Work Paper Reviews • Issue Report
QA - Assessment Objectives - Assess the efficiency and effectiveness of the internal audit activity in light of: - Its charter and mission - Expectations of the board, senior management, audit clients, and the CAE - Identify opportunities and offer ideas and counsel to the CAE and staff for: - Improving their performance - Increasing the value they add to the enterprise - Provide an opinion on the internal audit activity’s conformance to the spirit and intent of the Standards
QA - Assessment Approach - Self Study & Audit Management Questionnaire - Survey of Clients and Staff - Interviews with Senior Managers & Staff - Review Tools (Programs) üOrganization of the Internal Audit Activity üRisk Assessment and Engagement Planning üStaff Professional Proficiency üInformation Technology üProduction and Value Added üSample of Workpapers and Reports - Rating of Conformity with IIA Standards
QA – Conforming Evaluation Definitions • GC – “Generally Conforms” means the assessor has concluded that the Activity’s charter, structure, policies, and procedures, as well as the processes by which they are applied, are judged to be in conformity with a majority of the Standards with some opportunities for improvement being possible. • PC – “Partially Conforms” means the assessor has concluded that a good faith effort exist but deviations from conformity for a majority of the Standards exists and corrective action is needed. These deviations are not, however, significant enough to preclude the Activity from carrying out its responsibilities in an acceptable manner. • DNC – “Does Not Conform” means the evaluator has concluded that the Activity is not aware of, is not making good-faith efforts to comply with, or is failing to achieve conformity with the majority of the Standards, thus impacting its ability to carry out its mission.
QA Overall Evaluation OVERALL EVALUATION- Generally Conforms (GC) Attribute Standards 1000 1100 1200 1300 Purpose, Authority & Responsibility Independence & Objectivity Proficiency and Due Professional Care Quality Assurance and Improvement GC GC PC Performance Standards 2000 2100 2200 2300 2400 2500 2600 Managing the IA Activity Nature of Work Engagement Planning Performing the Engagement Communicating Results Monitoring Progress Communicating the Acceptance of Risk IIA Code of Ethics GC GC GC
QA - Potential Issues Reporting Categories • Opportunities to Improve Conformity with Standards • Opportunities for IA Consideration • Suggestions for Senior Management • Verbal Comments
QA – Validation Reporting Process Two Options: – Validator signs internally prepared report – Validator prepares separate report referencing internally prepared report
Quality Assessment Process Map (IIA Manual 7 th Edition)
Quality Assessment Process Map (IIA Manual 7 th Edition) • • IA Governance (1000, 1100, 1300, COE, & DIA) IA Staff (1200) IA Management (2000, 2100, & 2600) IA Process (2200, 2300, 2400, & 2500)
5 Minute Break
QA Related Standards - Revisit Original 1311 - Internal Assessments Should include: • Ongoing reviews of the performance of the IAA; and • Periodic reviews performed through self-assessment or by other persons within the organization, with knowledge of internal auditing practices and the Standards. Revised 1311 - Internal Assessments must include: • Ongoing monitoring of the performance of the IAA; and • Periodic self-assessment or assessments by other persons within the organization with sufficient knowledge of internal audit practices.
Internal Assessment Exercise #2 (5 minutes) What type of items would you see as part of your on-going program? What type of items would you see as part of your periodic program?
Internal Assessment 1311 - Internal Assessment Program – Ongoing Performance Reviews of the IA Activity • • • Work Paper Reviews Performance Evaluations Actual vs. Budgeted Analysis Various Monitoring Metrics Customer Surveys – Periodic Reviews • Self-Assessment – Annually – Covering all Standards over 5 years – Quarterly/Semi-Annual – Portions of Standards each year – Assess compliance with IA Activity Charter
Internal Assessment • Ongoing Assessments – …routine policies and practices used to manage the IA activity… • Engagement supervision • Checklists and other means • Feedback from IA clients/stakeholders • Project budgets, timekeeping systems, audit plan completion, cost recoveries and other performance metrics (e. g. cycle times and recommendations accepted) • Conclusions, follow-up, and implementation
Internal Assessment • Periodic Assessments (Snapshot In Time) – Non-routine special purpose reviews and testing • More in-depth interviews & surveys of stakeholder groups • May be performed via self-assessment or by other competent audit professionals within organization • May include self-assessments, preparation of materials and benchmarking subsequently reviewed by others • Can facilitate & reduce external assessment costs • Conclusions, follow-up, and implementation • Communicating Results – Share with various appropriate stakeholders
QA&IP Design Individual exercise: Please list 3 components or task performed by your IAA that you feel illustrate your working QA&IP.
QA&IP Design What would an effective QA&IP (performance measurement and reporting process) include?
Why is QA&IP Important? • Reasons for setting up QA&IP – Know where your group stands at all times – Potential external QA cost savings – Reduce risk of external QA “surprises” – Improve the IA environment/process – Reasonable assurance to audit committee – Quality does matter (i. e. Org. initiatives & SOX) – Required by the Standards • What reasons do you see out there?
QA&IP Design Program vs. Process -----Differing Perspectives
QA&IP Design IIA Sample QA&IP
QA&IP Design Deleted - Practice Advisory 1311 -2 Establishing the performance measure process The CAE Should: • Identifying critical performance categories • Identifying performance category strategies & measurement • Establish process for measurements to be monitored, analyzed & reported • Ensure measures used are appropriate to size & type of IAA
QA&IP Design Identifying critical performance categories Suggested categories: • • Key stakeholder satisfaction Internal audit processes Innovation Capability
QA&IP Design Key stakeholder satisfaction: Who are the stakeholders? • Internal – Audit committee – Executive management – Operating management – Internal audit clients – Audit staff – External government bodies and/or regulators • External auditors
QA&IP Design Key stakeholder satisfaction: How do you identify stakeholders? Consider the following: • Products & services being provided • Extent to which organization is regulated • Relationship with internal & external parties • Nature of the organization (public vs. private)
QA&IP Design Key stakeholder satisfaction: • Satisfaction levels must be assessed and gaps identified! – Interviews – Facilitated sessions – Questionnaire • Develop appropriate plan for corrective action • Execute, monitor & re-evaluate periodically
QA&IP Design Internal audit processes: • Risk assessment • Annual & long range planning • Engagement planning & performance – Proper scope, objectives, timing & resources – Conducted using established methodologies & practices • On-going communications • Reporting • Follow-up • Consulting • Fraud investigations
QA&IP Design Innovation & capability: • Training & competence – Documented training plan by position – Minimum annual training hours – Certification requirements & levels attained • Utilization of technology – Staff training goals – Audit staff satisfaction – Data extraction & analysis, automated work papers • Industry knowledge – Periodic staff interaction – Employee loan programs – Formalized rotation programs
Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency • Defining Internal Audit Effectiveness & Efficiency • Internal & External Stakeholders • Internal Audit Performance Metrics/Measures of Effectiveness & Efficiency • Monitoring and Reporting Results
Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency Selected Narrative – Executive Summary: • “To maintain and enhance IA credibility, its effectiveness and efficiency must be monitored. ” • “Identify key performance measures for IA activities that stakeholders believe add value and improve the organization’s operations. ” • “Effectiveness and efficiency measurements can be quantitative and qualitative. ” • “Adequacy of engagement planning and supervision. ”
Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency Selected Narrative – Defining IA Effectiveness & Efficiency: • “A general description of E &E is the degree (including quality) to which established objectives are achieved. ” • IA E&E should be monitored and assessed periodically as part of the IA process. ” Selected Narrative – Internal & External Stakeholders: • “Specific feedback will provide insight into; understanding of purpose, adequacy, deliverables, expectations, priorities, & shortcomings. ”
Supplemental Guidance - Practice Guide Measuring Internal Audit Effectiveness and Efficiency Selected Narrative – IA Performance Metrics/Measures of E&E: • “Identifying critical performance categories such as stakeholder satisfaction, IA processes, and innovation and capabilities. ” • “Routinely monitoring , analyzing, and reporting performance measures. ” Selected Narrative – Monitoring and Reporting: • “E&E should be reported to stakeholders periodically. ” • “ Consistent processes are needed for gathering, summarizing, & analyzing measurement data. “
QA&IP Implementation should include: • Measuring alignment with IIA Standards, key strategic objectives, & applicable laws & regulations • Timely gathering, summarizing & analyzing data • Ensure measurements kept current & consideration for changing expectations, conditions, priorities & objectives • Effective, efficient on-going reporting to stakeholders • Annual reporting on IA effectiveness to AC • Appropriate internal resourcing • Documented methodology • Staff involvement & buy-in
QA&IP Should Reveal IAA is: • Efficient & effective • Structured & staffed appropriately • Has an approach that is adequate & meet stakeholder expectations • Fully complying with the Standards • Utilizes sound testing techniques, methods & technology • Considers innovative practices & adopted them, when appropriate
Guiding Concepts • • • Design a program that fits your IAA Utilize available internal resources Treat as a project, start with a detailed plan Promote total team involvement Hold regularly scheduled update meetings Educate all constituencies (IA staff, executive management, & the audit committee) on objectives & progress • Make the process as transparent, objective & participatory, as possible • Conceptualize on synergies with external QA
Supplemental Guidance – 2 nd PPG Quality Assurance and Improvement Program
Capabilities Maturity Model Example
Performing the Validation - Key Points for Consideration General considerations Planning and preparation Interviews Self-assessment fieldwork Self-assessment results, recommendations and implementation plans
Performing the Validation Key Points for Consideration – Perception of lower cost – More time invested by IA Activity – Project timeline controlled by IA Activity – No or limited best practice enhancements – Less independent as much of the work is done by the IA Activity – Key Point-Validator should be qualified – Interview and survey limitations
Performing the Validation Overview and details: - General considerations - Planning and preparation - Interviews - Self-assessment fieldwork - Self-assessment results, recommendations and implementation plans
Performing the Validation General considerations: - Alternative means for complying with Standard 1312 external assessments - Benefits - Economics/Practicality - Expand external assessments to more IA activities
Performing the Validation General considerations: - Scope Limitations - Scope more targeted/limited than full external assessment - Focused on basic IA expectations - Fulfillment of IA mission - Conformance to the Standards - Areas where in-depth analyses may be curtailed or excluded
Performing the Validation Planning and preparation: - Designate project leader and team Select external independent validator Agree on scope and responsibilities Prepare self study Consider/ Conduct client surveys Select audit/consulting engagements for review - Select interview candidates for team and validator
Performing the Validation Interviews: - Audit Committee Chair - Executive to whom the CAE Reports - Senior and Operating Manager - CAE - IA Staff - External Auditor
Performing the Validation Fieldwork: - Departmental structure and organization - Risk assessment and engagement planning - Staffing skills and experience - IT review - Assessing productions and value added - Individual W/P file review Utilize Same Concepts (Tools) as External Summarized QAS Tool 12. doc Summarized QAS Tool 13. doc Summarized QAS Tool 14. doc Summarized QAS Tool 15. doc Summarized QAS Tool 16. doc Summarized QAS Tool 17. doc
Performing the Validation Results, recommendations & implementation plans - Major results/findings with emphasis on: - Opportunities for process improvement - Enhancing customer relations - Evaluation summary - Conclusion on conformity to the Standards
Performing the Validation - Recap Validation Process: - Independent validation of the self-assessment - Advance Prep review - AMQ review - Report review - On-site review - Documentation of self-assessment - Limited testing - Evaluation summary - Draft report/communication - Interviews - Memorandum/Closing conference/report
REMEMBER!!!!!! “You manage what you measure. ” ~Brian E, Kruk
Questions on QA&IP?
Thanks for your participation! Brian E. Kruk, CIA, CCSA, CGAP, CCA, CISA Director Contracts and Construction Union Pacific Railroad 402 -637 -1199 [email protected] com