- Количество слайдов: 40
Is it Method Validation, Verification or or Just Semantics? Michael Brodsky Consultants [email protected] com
Objectives n n Distinguish method validation from verification Consider the requirements for method validation and in-house verification from an ISO Standard 17025 perspective Review AOACI OMA and RI SLV and Multi-Lab Collaborative Study validation protocols as applicable to “in-house” validation Outline the requirements for a verification protocol that defines a method’s fitness-for purpose
What is Test Method Validation? Validation is the establishment of one or more performance characteristics for a test method n By single laboratory validation (SLV) n By multi-laboratory collaborative study n Always by comparative analysis to a “Reference” Method n
Verification n n Demonstration of Analytical competency. Performance characteristics of the method, when performed, as prescribed, by laboratory analysts, conforms to the performance characteristics of the method as established during the validation study. Ensures that the method is fit for its intended purpose Includes determination of uncertainty of measurement (quantitative methods).
Performance Characteristics of Microbiological Methods n n n n n Relative Accuracy (%Recovery) Precision (Repeatability and Reproducibility) Specificity (Selectivity for target analyte) Sensitivity (Distinguishing target from non target) Inclusivity (Range of target analytes detected by method) Exclusivity (Range of non-target analytes excluded) False Positive and False Negative Rates Limits of Detection (LOD) Limit of Quantitation (LOQ) Scope of application
Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study Performance Characteristic Identification Quantitative Qualitative (P/A) Relative Accuracy No Yes No Matrix Effects (Scope) No Yes Precision No Yes No Sensitivity Yes Yes Specificity Yes Yes Inclusivity Yes Yes Exclusivity Yes Yes False Positive Rate No Yes False Negative Rate No Yes LOD No Yes LOQ No Yes No Ruggedness Yes Yes Linearity/Range No Yes No
Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method Performance Characteristic Identification Quantitative Qualitative (P/A) Verification (Where Applicable) Relative Accuracy No Yes Matrix Effects (Scope) No Yes No Precision No Yes Sensitivity Yes Yes Specificity Yes Yes Inclusivity Yes Yes No Exclusivity Yes Yes No False Positive Rate No Yes No False Negative Rate No Yes No LOD No Yes No LOQ No Yes No No Ruggedness Yes Yes No Linearity/Range No Yes No No
17025 Requirements n Methods requiring validation are: n Modified official methods n In-house developed methods n Previously validated methods extended to a component, analyte or matrix not previously tested or included in the validation study
Food Categories and Types
Food Categories and Types (Based on Physiochemical and Innate Microbial Characteristics)
Food Categories/Matrices 15. Raw milk and dairy products Heat processed milk and dairy products Ready-to-eat, ready-to-reheat meat products Raw poultry and ready-to-cook poultry products Ready-to-eat, ready-to-reheat meat poultry products Eggs and derivatives Raw and ready-to-cook fish and seafoods (unprocessed) Ready-to-eat, ready-to-reheat fishery products Fresh produce and fruit Processed fruits and vegetables Infant formula and infant cereals Dried cereals, fruits, nuts, seeds and vegetables Chocolate, bakery products and confectionary Multi-component foods or meal components Pet food animal feed 16. Environmental samples (food or feed production) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
Food Categories and Types 9. Fresh produce and fruit Types: i. iii. iv. v. vii. Cut ready-to-eat fruit Cut ready-to-eat vegetables Produce grown in or in contact with the ground Sprouts Raw fruit/vegetable juices (unpasteurized) Leafy greens Vegetables and fruits (unprocessed) not described above
Food Categories and Types 15. Pet food animal feed Types: i. iii. iv. v. viii. ix. Animal origin ingredients Plant origin ingredients Other ingredients Dry food (aw ≤ 0. 7) Wet food (aw > 0. 7) Canned Animal feeds (bovine, pig) Animal feeds (poultry) Animal feeds (fish)
AOACI Pre-Collaborative/PTM Study Design (SLV) n For Each Food Category to be claimed: n n ≥ 2 food types/category If claiming if applicable to a broad range of foods: n Test ~20 samples from 9 different Food categories
AOACI Pre-Collaborative/PTM Study Design (SLV) (Quantitative Method)* 20 analyses for each food type: n 5 portions of each matrix @ n Low, medium and high levels of contamination and uncontaminated n Comparison of recovery to reference method using ANOVA and comparison of means *Use as guide for “in-house’ validation n
AOACI Pre-Collaborative/PTM Study Design (SLV) (Qualitative Method)* n 30 analyses for each food type: n n n 5 replicate test portions per level the high inoculation level, 20 for the fractional positive level 5 for the uncontaminated level Comparison to reference method using Chi- square analysis &/or Probability of Detection (POD models) *Use as guide for “in-house’ validation
AOACI Collaborative Study – Quantitative n n n 10 - 12 laboratories n Minimum of 8 labs with acceptable data Test samples* n 1 Matrix n Low, medium and high contamination +uncontaminated n Five samples tested at each level n Both naturally and artificially contaminated food n Compare against a reference method n Repeatability, Reproducibility and differences between means *Use as guide for “in-house’ validation
AOACI Collaborative Study (Qualitative Method) n 12 - 15 laboratories n n Minimum of 10 labs with acceptable data Test samples* 1 matrix n 12 test portions per high analyte level n 12 test portions per fractional Positive samples n 12 uncontaminated test portions *Use as an alternate guide for “in-house’ validation n n Comparison to reference method using Chisquare analysis &/or Probability of Detection (POD models)
"I think you should be more explicit here in step two. "
17025 Requirements n n n Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format. Do not need to be fully validated. The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs
Verification of Quantitative Microbiological Methods • • • Precision (Repeatability and Reproducibility) Uncertainty of Measurement Fitness-for-Purpose
Validation Data from Collaboratively Studied Methods Provides reference values for RSDR and RSDr Ø e. g. Pour Plate counting (SMEDP) Ø e. g. RSDr ≤ 7. 7% (0. 077) (within analysts) Ø e. g. RSDR ≤ 18. 2% (0. 182) (between analysts) Ø
Validation Data from Collaboratively Studied Methods Can also be used to estimate uncertainty of measurement Ø e. g. Pour Plate counting (SMEDP) Ø e. g. RSDr ≤ 7. 7% (0. 077) (within analysts) Ø e. g. RSDR ≤ 18. 2% (0. 182) (between analysts) Ø
Calculation of Combined Uncertainty Ø Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the “root sum of squares” - RSS).
Validation Data from Collaboratively Studied Methods Ø Calculation of Combined Uncertainty (Uc): Root Sum of Squares: √(RSDr)2 + (RSDR)2 For Pour Plate (HPC) Ø Sum of Squares: (0. 077)2 + (0. 182)2 =0. 0371 Uc = √(0. 0371) = 0. 193=19. 3% Ø Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19. 3% = 38. 6% *≥ 30 Observations
Verification of Qualitative Microbiological Methods • • • Specificity Sensitivity Fitness-for-Purpose
How Many Samples Are Needed for Verification? ?
How Many Samples Are Needed for Verification? n n n No fixed number But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable 10 is the minimum requirement
Who’s Doing What n n ISO: Microbiology of food animal feed – Method Verification– Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing
ISO Protocol for the verification of reference and alternative methods implemented in a single laboratory (under review ) n n Minimum of 10 artificially inoculated samples Use CRM, if available Ø For Qualitative methods inoculate with 1 -5 CFU per test portion. Ø For Quantitative methods, the levels of contamination shall cover the range of the method n Analyse samples on 10 different occasions or days n be performed by at least 2 technicians working independently and with separate samples and reagents.
How to Interpret Verification? n For Quantitative methods, ISO proposes to accept: Ø participation in interlaboratory comparisons such as proficiency testing and plotting z-scores to show any trends; Ø use of microbiological RM or CRM; Ø recovery experiments with spiked samples using a non-selective method.
Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing by Health Canada (April 2015) For Qualitative Methods, n Analyze 3 -5 samples spiked at 3 -5 times the reported or determined LOD (1 -3 CFU/analytical unit) For Quantitative Methods n using selective medium, measure repeatability by analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated
How to Interpret Verification? For Qualitative methods, ISO and HC agree: 100% (comparative) sensitivity All samples must be correctly identified Ø For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2 r < R. If R value if published. Ø If there is no published performance characteristic data, ? ? ? Ø Calculate Ue Ø
17025 Requirements n n Fitness-for-Purpose The data obtained from the method approval process must show it is fit for the intended use and relevant to customers’ needs. Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?
References n n AOAC International Methods Committee Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J. The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group How to Meet ISO 17025 Requirements for Method Verification, 2015, AOAC Technical Division for Laboratory Management (TDLM), the Analytical Laboratory Accreditation Criteria Committee (ALACC) ISO/FDIS 16140 -2 Microbiology of the food chain — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method (ISO/TC 34/SC 9) 2014 -09 -05 (Draft)
Thank you for your attention Any Questions? [email protected] com