Assessing Quality Systems An Overview
Course Objectives n At the conclusion of this course, participants will be able to: n Discuss n n the purpose of Quality Systems Assessments (QSA) the benefits of conducting QSAs how QSAs differ from other types of assessments Demonstrate a process for planning, conducting, and reporting the results of QSAs
Quality Systems Assessment n WHAT n a systematic, independent, and documented examination that uses specified criteria to assess quality systems n WHY n EPA Order 5360. 1 n 48 CFR Part 46 and 40 CFR Parts 30, 31, and 35
Assessment vs. Audit n Assessment n the evaluation process used to measure the performance or effectiveness of a system and its elements n Audit n a systematic and independent examination to determine whether quality activities comply with specifications and whether these specifications are implemented effectively
Types of Assessments Process Project/Program System Process Audits - Field and Lab - I. T. - Databases -Software Project/Program - Project -Technical
QSA Perception Check True or False n Planning and conducting a QSA requires no expertise in an organization's activities and how it operates n QSA results will tell management where to look to uncover problems (e. g. , performance issues, fraud) n A QSA can serve as a key deterrent to organizational improprieties n New organizations don’t need an assessment of the quality system n A QSA can indicate the effectiveness of communicating quality policies in an organization
QSA Benefits QSA results can be indicators of the: § Adequacy and implementation of programs developed to achieve an organization’s mission § Quality of products and services § Compliance with governing specifications
Practice What You Learned For each scenario, determine: • What type of assessment is appropriate (system, project/program, process)? • Should the assessment be internal or external? • What benefits may be achieved from the assessment? n A recent grantee has water sampling scheduled through a QAPP that the funding organization has just approved. Their QMP had few details, and it is suspected that they have inadequate processes for overseeing the work. n A large organization, with many divisions and several QA staff have recently amended their QMP to change the way they delegate QAPP approval. n An organization had been subject to regular assessments. Each set of assessment results contains the same findings for insufficient documentation of the quality system. However, improvement in the project management QA responsibilities is noted.
Assessment Framework QSA Results Reporting Conducting Planning
The Assessment Cast n Authorizing Entity n Assessee n Assessment Team Leader n Assessors
Authorizing Entity Roles and Responsibilities n Authorize and provide resources for the assessment n May set the scope of the assessment n Review assessment results and ensure corrective actions Actions BEFORE the QSA n Authorize the QSA and provides resources n Approve the assessment plan
The Assessee Roles and Responsibilities n Acknowledge the QSA communications n Work with the assessor to accomplish the QSA Actions BEFORE the QSA n Provide preliminary documentation and scheduling assistance n Review the assessment plan
Assessment Team Leader Roles and Responsibilities n Select team members n Prepare the assessment plan n Complete and submit the assessment report n Address comments to the report Actions BEFORE the QSA n Understand the need, authority and resources for the QSA n Prepare written and oral communication, including assessment plan n Prepare interview/document review checklists
The Assessors Roles and Responsibilities n Identify and accurately document n n non-compliance noteworthy practices n Review and comment on draft reports to ensure accuracy Actions BEFORE the QSA n Understand the purpose of the QSA and assigned areas n Review relevant documentation n Assist with assessment plan and checklists, etc.
Assessment Planning "Would you tell me which way I ought to go from here? " asked Alice. "That depends a good deal on where you want to get, " said the Cat. "I really don't care where" replied Alice. "Then it doesn't much matter which way you go, " said the Cat. Lewis Carroll, Alice's Adventures in Wonderland (1865), novelist and poet (1832 -1898).
Initial Planning n The initial planning process involves: n gaining knowledge and understanding of the assessee organization n determining the lines of assessment inquiry n formalizing assessment objectives n considering available resources
Initial Planning Steps n Determine criteria for the assessment n Determine the assessment scope and available resources n Select the assessment team
Initial Planning Steps Determine Assessment Criteria Assessment criteria can be set by policy or directives: n Quality System Specifications n n n Assessee specifications/policy n n n Order 5360. 1 ANSI/ASQ E-4 the assessee’s Quality Management Plan assessee reports Regulatory requirements n n contracts and assistance agreements program specific requirements
Initial Planning Steps Determine the Assessment Scope n The scope is set by the: n directive of the authorizing entity n allocated resources n onsite considerations
Initial Planning Steps Select the Assessment Team n Select team members that: n have appropriate skills and knowledge n are competent and credible n have no conflict of interest
Documenting the Assessment Process n The Assessment File contains: n planning documents n relevant correspondence n working papers n assessment reports
Documenting the Assessment Process Tracking Tools n HOW n Tools for managing document flow and status from planning, observations and actions taken, and filing. n n Paper notation Electronic data base n WHY n Papers flow through many channels n A need to effectively manage assessment information
Assessment Communications n Initial contact with the assessee to begin scheduling n Approval of the assessment plan by the authorizing entity n Formally notify the assessee of the planned assessment n Reporting, tracking corrective actions, closeout
Case Study – Part I n Review the Case Study n Appoint a spokesperson n Complete the Initial Planning Worksheet
Detailed Planning Whatever failures I have known, Whatever errors I have committed, Whatever follies I have witnessed in private and public life Have been the consequence of action without thought. Bernard Baruch, stock broker, advisor to presidents Woodrow Wilson, Harry S. Truman,
Detailed Planning n Information Review n Issue Selection n Identifying Interviewees and Onsite Documentation
Information Review the organization’s quality system n Quality Management Plan n Applicable regulations for environmental programs n Reports from previous assessments n Quality Assurance Annual Report and Work Plan/Fiscal reports n Published products n Information about personnel (chart, CVs)
Information Review cont’d n What can the documentation tell you? n QMP n Mission/quality policy n QA support/resources n Specifications of the quality system- roles (e. g. , description of programs and products) n Regulations n QA context for work n Previous results n Prior problems n Annual reports n Resources, activities, plans n Fiscal reports n relative importance n Published products n conformance, status n Staff information n qualifications, experience
Issue Selection n Items that help to define/select assessment issues: n assessment issues pre-defined by the authorizing entity. n quality system processes affecting costly or important products n quality system specifications having a major effect on environmental data operations n processes appearing difficult or expensive to implement n recently implemented policies and procedures n processes having been problematic for similar organizations
For Example n Roles and responsibilities from the QMP n The Envirumdata Group, composed of 4 programmers and 2 data entry professionals, is responsible for ascertaining the quality of all analytical data entered into the organization’s database n Specification of the quality system n All data (collected by other organizations) in the database must be assessed prior to use n Relevance of specification to the program n Data are used in support of rule making
Resulting Questions/Issues n n What is the process for data quality assessment? How is the process documented? How are the results documented? Do they have adequate expertise to assess data quality?
Tips for Issue Selection Look for: § incomplete and unclear quality system documentation § processes that seem too extensive for the organization’s size and mission § processes that are not clearly described either in full or by reference § end products that lack review and approval processes § issues that pass the “So What” test
Where do you draw the line? ? RISK
QSA Data Gathering n Primary QSA data gathering tools n Interviews n Interaction with the interviewee provides the team with direct information about the quality system. n Document reviews n Supporting documentation can be consulted and questions explained and clarified as necessary. Remember: The right people need to be asked the right questions.
QSA Data Gathering Select the Right Documents and the Right People n Define what documents contain information relevant to the selected issues. Include information that: n n may be requested prior to the onsite will be available during the onsite review n Identify the appropriate interviewees for the selected issues n usually those who dictate the specifications for, create, contribute to, review, approve or use the documentation.
QSA Data Gathering Get the Right Type and Number Selection considerations: size of organization n primary functions n personnel factors, including availability, experience, knowledge of issues selected, length of time in position, number in each job category n document factors, including importance of work products, size of document pool, appropriateness to the time period and process being assessed. n
Think About It… n Your assessment plan instructs you to ask 26 people whether they are familiar with their organization’s QMP. What assessment finding could you make? n You review QA documentation processes and understand the specifications (what is required, roles and responsibilities for preparation, review, approval, and record keeping according to the QMP). You then examine 26 examples and ask interviewees about the process they followed. What assessment finding could you make?
LUNCH!
CASE STUDY PART II
AFTER the PLAN is Done…… n Assessment plan approval n Confidentiality considerations n Assessment tools
Review and Approval of the Assessment Plan n Who? n Your management n Authorizing entity n Why? n Provides an opportunity to raise questions about the approach n Approval of the plan affirms the authority, credibility and scope of the assessment process n Encourages buy-in and ownership
Confidentiality Considerations n Dissemination of assessment findings “generally only to involved parties. ” n Interview notes are discarded when the report is final n CBI – important in RCRA, CAA, FIFRA
Assessment Tools Checklists n Used to: n organize interviews and document review n assure completeness and consistency in reviews n promote objectivity among assessors n Design the checklist to: n fill information gaps in Quality System information n verify Quality System specifications n address specific observable practices
Assessment Tools Matrix n Suited for complex assessments n Serves as a visual check for the assessment team; shows where noncompliance may be clustered or how wide spread a practice may be
Ready, Set……………… Just as there are three R's there also three A's of business life. They are: Ability, Ambition, and Attitude. Ability establishes what a worker does and will bring him a paycheck. Ambition determines how much he does and will get him a raise. Attitude guarantees how well he does. Wilbert Sheer
Conducting the Assessment n Logistics n Onsite Activities n Reporting
Logistics n BEFORE YOU GET THERE, CONFIRM: n n Building access Assessment schedule/issues n n n Opening and closing Meetings Interview schedules Assessment team meeting times Availability of meeting rooms/work space File review access n TO GET THERE n n n Travel (arrival and departure) itinerary for all team members Lodging (phone, address) Local city transportation (maps, taxi rates)
The Onsite Conduct: n The Opening Meeting n The Onsite Review n n Documents and Records Review Interviews
Opening Meeting n Typical Format n Introductions n Brief description of the scope of the assessment n Explain the assessment team’s schedule n Answer any questions regarding the assessment n Limit time to one hour or less
The Onsite Review n Document and Records Review n Review documents listed in the assessment plan n Focus on documents that verify policies and processes n Verify supplemental information (e. g. , databases, archived information) n With permission, make copies of relevant information
Interviews n Elements of the interview n Planning n Opening n Conducting n Closing
Planning n Establish information needs n What information is needed n Who needs to be interviewed n Format assessment questions
Types of Questions n Open ended n Prompt the speaker to provide detailed information n Close ended n Questions that offer a set of alternative answers (e. g. , yes or no, true or false, scaled answers)
Formulating Questions To find out what they know Ask them to describe something To find out how they do something Ask them to describe an activity To find out what they think Ask them for an opinion
Tips n Questions should be: n as neutral as possible n asked one at a time n worded clearly n Use "why" questions sparingly
Interview Scheduling n Scheduling considerations n Schedule in advance n Plan no more than one hour for each interview n One on one interviews (preferred) n Try to avoid the hour right before or after lunch
Opening the Interview n Sets tone for the interview n Make introductions n Keep small talk to a minimum n Re-state the purpose of the interview and how you will proceed
Opening the Interview cont’d n Tell them: n who will get access to their answers and how their answers will be analyzed n the format of the interview n how long the interview usually takes n how to get in touch with you after the interview n that you will take notes during the interview
Note Taking Tips n Use a notepad n Record main ideas n Periodically read back notes to check for accuracy n Notes are about PROCESSES, not PEOPLE
Conducting the Interview n Remain as neutral as possible n Encourage responses n Control your behaviors when note taking n Provide transition between major topics n Don't lose control of the interview
Closing the Interview n Briefly summarize your information n Thank the interviewee for his/her time n Ask if the interviewee would be willing to follow up with you if necessary
PRACTICE WHAT YOU LEARNED The character of the questions you ask will influence the appropriateness of the answers. Ø You ask the interviewee about his interaction with the QA Manager and he immediately displays resentment. You ask…. . "So I guess you don’t like him much, huh? " Ø You should ask………. Ø You ask a database operator …. . "Do you know how to work this thing? " Ø You should ask…………. .
PRACTICE WHAT YOU LEARNED n One More for Good Measure! Ø You ask a field sampler…. "Have you been following this SOP? “ Ø You should ask…………….
Think About It Based on your draft assessment plan, formulate 2 questions for each of the following categories: - Are Quality System policies implemented by staff and management? - Are the Quality System policies effective? - Do personnel understand the Quality System requirements?
Working Papers n Assessment Working Papers n Notes from document review n Copies of documents (with permission!) n Interview notes n Cross reference to criteria n Date each document
Now What? ? ? n Meet with your team each day to review working papers n Develop preliminary findings
Finding An assessment conclusion that identifies a condition having a significant effect on an item or activity
Other Assessment Results n Noteworthy practices n Positive practices implemented by the organization n Observations n Practices noted during the review. No corrective action required.
Example: Finding 1 n The QA Coordinators need training in QA concepts and theories
Example: Revised Finding 1 n The QA Coordinators are unable to perform the review and approval of QAPPs as specified in the QMP because they have not been trained in the process.
Draft Preliminary Findings n Presented to summarize what you have observed during the onsite n Observations should be factual and to the point n Findings can change during follow up review (HINT - You’re not finished!)
Exercise!
Closing Meeting n Brief management on your preliminary findings n Explain your reporting process and timetable n Remember: No Surprises! n Preliminary findings must not be frivolous n Keep the meeting to no more than 1 hour n Try to address immediate concerns but DON’T n n n Make promises you can’t keep Usurp your management’s authority
The Black Hole Phenomenon
The Report n The assessment report communicates assessment findings to management. It should clearly identify: the type of assessment, n the assessor, n the assessee, n what was assessed, n the findings, n if requested by the authorizing entity, recommendations for corrective actions n
Effective Reporting n Clear and concise writing n Address significant deficiencies in a constructive manner n Report the organization’s actions, not the actions of specific individuals n Prioritize assessment findings and any need for corrective actions
Example Report Outline Background and Purpose II. Assessment Scope III. Assessment Results and Rationale References Appendices I. n n n A. List of Personnel Interviewed B. List of Documents and Records Reviewed (optional) C. Corrective Action Plan
Close Out
Close Out n BEFORE you can close the assessment you need: A response from the assessed organization with an acceptable corrective action plan (if necessary) and an indication that the corrective actions have been completed n A review of the assessment file to ensure that it is complete before it is archived. n
SUMMARY n Now you can discuss: n the purpose and benefits of Quality Systems Assessments (QSAs) n how QSAs differ from other types of assessments n You can demonstrate a process for planning, conducting, and reporting the results of QSAs I can give you a six-word formula for success: "Think things through - then follow through. " Edward Rickenbacker
Скачать презентацию Assessing Quality Systems An Overview Course Objectives
fd8f9e3e442cb1d169c09b5205372338.ppt