How To Investigate The Bias on Your CAP

How To Investigate The Bias on Your CAP Proficiency Testing (PT) Evaluation Reports? By: Muain Haseeb: Ph. D. (BA), MS (HSA), BSMT (AMT), LQMSC (CLSI), CKPIP Consultant Laboratory Quality & Safety Mobile: 055 694 4056 Email: Dr. Muain@yahoo. com * The. Quality. Doctor@gmail. com

Learning Outcomes: n n n Introduction What is PT (Proficiency Testing)? What Do You Do When You Receive You PT Shipments? How To Enter Your PT Results In CAP E Lab Solution? How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports?

Introduction: n For Non US Laboratories, if you are a CAP (College of American Pathologists) accredited lab, or thinking of becoming a CAP accredited, you must have or order (buy) PT (Proficiency Testing) from them (CAP). n CAP will not accept PT from another source unless approved by them.

Introduction:

What Is PT? n Evaluation of participant (laboratory or individual) performance against pre-established criteria by means of interlaboratory comparisons. § In some countries, the PT programs for clinical laboratories are called "external quality assessment" programs.

What Do You Do When You Receive You PT Shipments? n Log them on a form (calendar)

What Do You Do When You Receive You PT Shipments? Give them to the laboratory section to run Note: Sections read instruction well before running n Run PT samples like patients’ samples n Record / Log on CAP work sheets n Techs performed the survey and Medical director or designee must review / sign the Attestation Page n Enter and send PT results to CAP by or before due date (deadline) n Record date sent and who did the survey on the form (calendar) n

What Do You Do When You Receive You PT Shipments?

How To Enter Your PT Results In CAP e Lab Solution? n n n n n Log In into CAP Website. Click on “E Lab Solution” Click on “Results Form Data Entry” A new widow opens, click on the survey you want to enter the results for A new window opens, Click on “Enter Data” Enter your results; Save Go back and Click on “Pending Approval” Click on “View/Edit” Click “Save” Your PT status need to say “RECEIVED”

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n To Print Your Evaluation Reports: n n n Log In into CAP Website From e Lab Solution, click on “Evaluation Reports” You can look a report for a PT (survey) in 2 ways: n If it is on the list of the most recent PT reported, click on it, and it will open n If it is not on the list, click on: n “Click here for filter options to modify the list of kits displayed or search for other kits”, n enter the Kit Number, n click “GO”. n A new window open up. n If the report is available, it will come up. Either way, under available reports, select “Printed Evaluation”. Click on view report, the report will come up; print it.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n Evaluation of Results (Reading the Report): **REVISED** 08/17/2016 n COM. 01700 PT Evaluation n There is ongoing evaluation of PT and alternative assessment results, with appropriate corrective action taken for each unacceptable result. NOTE: n n n n Phase II Primary records related to PT and alternative assessment testing are retained for at least two years (five years for transfusion medicine). These include all instrument tapes, work cards, computer printouts, evaluation reports, evidence of review, and records of follow-up or corrective action. For laboratories outside the US, PT failures relating to problems with shipping and specimen stability should include working with local customs and health regulators to ensure appropriate transit of proficiency testing specimens. Evidence of Compliance: Records of ongoing review of all PT reports and alternative assessment results by the laboratory director or designee AND Records of investigation of each "unacceptable" PT and alternative assessment result including records of corrective action appropriate to the nature and magnitude of the problem

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n CAP PT Evaluation Report Includes One or More of the Followings: n Graded Results: n n Ungraded Results (Exceptional Codes): n n Acceptable with no bias or exceptional codes (ungraded) Acceptable, but shows trends (bias results) Unacceptable (counted against you) Might or might not be counted against you For sake of time, in this presentation, I am only going to talk about a challenging issue which is “investigation of PT results that are “Acceptable, but shows trends (bias results)”.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n n Acceptable, But Shows Trends (Bias Results): CAP requires a review and (if needed) an investigation and corrective action of all acceptable (satisfactory) PT results that shows bias or trend.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? Acceptable, But Shows Trends (Bias Results) Cont. : n n Used for Quantitative Data, that are “within limits” but might be experiencing a slowly developing bias that may result in future failures (unacceptable results). n Standard Deviation Index (SDI) is an excellent tool to access the PT bias. n As applicable, CAP calculates and gives each participant laboratory the SDIs on the evaluation report for their PTs.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports?

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n How do you calculate the SDI? n SDI value is obtained by subtracting the Group Mean from Your (Lab) Result and then dividing by the Peer Group SD (Standard Division): n SDI = Your Results – Group Mean Group SD n To calculate, from the above report, the Peer Group Mean for SE 11 is 356 IU/m. L and the Peer Group SD is 22. 6 IU/m. L, and the Lab Results is 371 IU/m. L. So: n SDI = 371 356/22. 6 = 0. 7 n But, what does 0. 7 mean?

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n Use the below guidelines to know if your lab SDIs are acceptable or not: n An SDI of zero (0. 0) indicates a perfect comparison with the peer group. n If the average SDIs is ± 1. 5 or less, it is considered acceptable. n If the average of the SDIs results is more than ± 1. 5, this may indicate a significant systematic error, n If one of the SDIs is greater than ± 3, this may indicate a significant random error. n If the total SDI of 2 SDIs is greater than 4 (i. e. 1 SDI is 2 and 1 is +2. 5 for a total of 4. 5), this may indicate a significant random error. n To investigate the bias on the PT evaluation report, for each analyte, review, the SDI score reported (if given by CAP).

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports?

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? § Document your evaluation and investigation (if indicated), on forms “Form Lab. Gen 029 Proficiency Survey Evaluation” and “form 030 Proficiency Survey Investigation”. n Make sure the medical director or designee (section head) reviews and signs all proficiency testing evaluations and investigations.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n Another optional tool to evaluate the quantitave data on the PT evaluation report is to evaluate the allowed deviations on the graphical summary using the relative distance of your results from the target. This distance is called the Allowed Deviation. n CAP calculates and graphs the % of Allowed Deviation for 3 PTs (if you have) on your PT Evaluation reports.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports?

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n The % of Allowed Deviation may be calculated as follows: n If your result is greater than the target mean: n n If your result is less than the target mean: n n (%) of Allowed Deviation = 100 X Your Result Target ______ Upper Limit Target Mean (%) of Allowed Deviation = 100 X Your Result Target_____ Target Mean Lower Limit If an analyte on the evaluation report falls outside of the acceptable range, it will be flagged by a star.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports?

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? Patterns in PT Evaluation Graphs for a Single Mailing Rule One result in a mailing exceeds ± 75% of the allowed deviation Comments Review results to rule out possible problems; identify possible errors from non-analytical sources such as clerical errors for results that exceed ± 100% of the allowed deviation. Suggested Actions If the data fail either of the rules below, follow the suggested actions for systematic or random error, as appropriate All results are on one side of the target values Shows bias indicating a possible calibration drift; there would be with at least 1 difference exceeding ± 50% of less concern if the relative differences were all close to 0 the allowed deviation See listing of suggested action for evidence of systematic error below Large positive and negative differences; Shows possible random error combined lengths of longest positive and negative bars is > 140 out of total range of 200 See listing of suggested actions for evidence of random error below Time Trends in PT Evaluation Graphs over Multiple Mailings Rule Persistent results on one side of the target values Comments Show persistent bias, even if small; recalibration should have occurred within this time frame Suggested Actions See listing of suggested actions for evidence of systematic error below Results flip from one side of the target to bars Shows impact of system and/or process changes; longer bars are See listing of suggested actions for evidence of increase of more concern systematic error Over time, length of bars increase A sudden shift may show impact of system and/or process changes; may reveal new source of either systematic or random error Follow the suggested actions for systematic or random error, as appropriate Over time, length of bars decrease Shows impact of system and/or process changes, particularly as a result of corrective action Retain as documentation that corrective action has been successful

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n Suggested Actions If There is Evidence of Systematic Error (If Applicable): n Review internal QC performance: n n n n n Perform instrument maintenance. Recalibrate instrument, if it has not already occurred. Review reagent/ sample storage; e. g. refrigerators. Prepare fresh reagents and re run sample. Check pipettes. Perform staff training. If participating in an external QC performance program, review comparative reports for QC performance: n n Look for trends or shifts that may not yet trigger your rejection rules. Assess the process of setting and changing QC target values. If the laboratory performance on a lot of QC material is at consistent variance with the group performance mean, further investigation is warranted. Re test the PT sample again to evaluate performance.

How To Check For, Print, Evaluate, & Investigate The Bias On Your PT Evaluation Reports? n Suggested Actions If There Is Evidence of Random Error (If Applicable): n n Rule out errors from non analytical sources (transcription error, misplaced specimens, calculation error). Re run the sample, if the result of repeat analysis is acceptable then corrective action is not required. If the issue persists, investigate possible sources of systematic error. Investigate components of the analytical system (sample probes, reaction cells, reagents). Review internal QC performance: n n n Look for trends or shifts that may not yet trigger your rejection rules. Assess the process of setting and changing QC target values. If all possible causes have been excluded, a single unacceptable result is most likely due to random error.

Summary: n All CAP PT failures and biases need to be investigated & documented in a timely manner. n So, print your evaluation reports ASAP, review, and investigate as needed. n For All Quantitative PT: if you are a CAP accredited lab, and you are not investigating the PT bias already, I suggest, you go back to all of your PT for 2016, and review and investigate the BIAS, as needed. n If you are not a CAP accredit lab yet, start doing it.

Questions?

How To Investigate The Bias on Your CAP Proficiency Testing (PT) Evaluation Reports? By: Muain Haseeb: Ph. D. (BA), MS (HSA), BSMT (AMT), LQMSC (CLSI), CKPIP Consultant Laboratory Quality & Safety Mobile: 055 694 4056 Email: Dr. Muain@yahoo. com * The. Quality. Doctor@gmail. com

References: n CAP. ORG; E LAB Solutions 2016 n All Common CAP Checklist, 08/17/2016 n CAP Proficiency Testing (PT) Manual 2016 n Quality Management IPPs KKUH Lab