Choosing and validating new kits and equipment Keith

Choosing and validating new kits and equipment Keith Perry, Unit Head Microbiological Diagnostics Assessment Service [Mi. DAS] Evaluations & Standards Laboratory BSMT meeting, Management in Microbiology Laboratories November 2005 www. hpa-midas. org. uk

Choosing & validating kits & equipment www. hpa-midas. org. uk

Choosing & validating kits & equipment Today’s presentation • Background to Mi. DAS • Costs / Procurement • CE Marking • Evaluations • Validations • Issues common to Evaluations & Validations • Practical considerations www. hpa-midas. org. uk

Mi. DAS Evaluations & Standards Laboratory [ESL] Evaluations Unit [Mi. DAS] Quality Control Reagents Unit Evaluate kits & equipment and write reports on findings Provide laboratories with virology /serology IQC reagents Standards Unit Quality System Unit Write and co-ordinate the consultation and distribution of standard methods and clinical testing algorithms Advise on quality systems and audit www. hpa-midas. org. uk

Mi. DAS Associated organisations / contracts National Blood Service Centre for Evidence-based Purchasing Evaluation Centre for Microbiological IVDDs World Health Organisation Mi. DAS [ESL] Input into NBS Kit Evaluation Group committee Commercial contracts WHO Collaborating Centre for Diagnostic and Laboratory Support www. hpa-midas. org. uk

www. hpa-midas. org. uk

Procurement • Recent formation of Centre for Evidence-based Purchasing (CEP-PASA) • Focus on regional procurement hubs • Improved support for clinical networks by increasing their involvement in purchasing • Joining up device performance data with purchasing cycles • Strategic role of procurement in supporting adoption of new technology/ideas www. hpa-midas. org. uk

www. pasa. nhs. uk/evaluation www. hpa-midas. org. uk

CE Marking IVDD Directive – 98/79/EC Common Technical Specifications www. hpa-midas. org. uk

CE Marking does not cover: - • Ease of Use • Monitoring of internal QCs • External QA • Use of combinations ie kits/automated platforms • Comparative data • Evaluation and validation www. hpa-midas. org. uk

Evaluations A generic term for any study that measures the performance capabilities of an assay. While the main focus of Mi. DAS is evaluation of commercial devices, our processes can also contribute to high quality evaluation of innovative and in-house assays. Aims • To undertake comparative evaluations of devices intended for the in-vitro diagnosis and management of microbiological infection • To focus on device performance • To be independent and unbiased • To undertake evaluations in an accurate, efficient and timely manner • To provide informative reports www. hpa-midas. org. uk

Evaluations Why do we need evaluations? • Best practice • Relevant to important public health concerns • Standardisation of methods • Managed introduction of new technology • Encourages development www. hpa-midas. org. uk

Evaluations www. hpa-midas. org. uk

Evaluations Choosing HIV kits Timing of detection of primary HIV following seroconversion 0 Biotest Anti. HIV TETRA ELISA 5 VIDAS HIV DUO = combined antigen-antibody Enzygnost HIV Integral *Earliest anti-HIV detection Earliest HIV detection GENSCREEN PLUS HIV Ag. Ab = immunometric Biotest HIV 1/2 recombinant Ortho Ab-capt. ELISA Access HIV 1/2 NEW Ax. SYM HIV Ag/Ab Combo Murex HIV 1/2 VK 84/85 Vitros ECi anti-HIV 1/2 Vironostika HIV Uni-form II Ag/Ab Murex HIV Ag/Ab combination Innotest HIV-1/-2 Enzygnost HIV 1/2 plus Biotest Anti. HIV TETRA ELISA 10 Murex HIV 1. 2. 0 GE 94/95 15 20 days Murex ICE HIV 1+2 Abbott 3 rd gen Plus Clonesystems Detect-HIV v 1 Vironostika IMx HIV Uniform II HIV 1/2 Wellcozym plus O III plus e Pasteur Genscreen Version 2 Anti-HIV Ax. SYM HIV 1/2 g. O = Class specific antibody capture = antiglobulin / indirect www. hpa-midas. org. uk

Evaluations Choosing HIV kits Combined scores for commercial seroconversion panels www. hpa-midas. org. uk

Evaluations Choosing HIV kits Sensitivity Specificity www. hpa-midas. org. uk

Evaluations Good evaluation practice www. hpa-midas. org. uk

Evaluations Defined procedures Prioritisation system Literature Review 16 stages Preparation Setup 10 11 stages Technical Assessment 13 stages Report 13 stages Post publication 8 stages Customer Service & feedback www. hpa-midas. org. uk

Evaluations Collaborations www. hpa-midas. org. uk

Evaluations Benefits of collaborations • To incorporate latest developments and test algorithms into evaluation design • To maximise mutual benefit • To avoid conflicts in output • To bring together information on all evaluations undertaken • To work to publish results in a timely manner • To use mechanisms already in place and to avoid compromising good evaluation practice www. hpa-midas. org. uk

Evaluations Collaborations: Chlamydia trachomatis NAATs www. hpa-midas. org. uk

Extent of NHS use Evaluations Prioritisation – CEP ‘Hopper’ system Anyone can submit proposals Adoption/purchase Evaluation project proposals Prioritisation Board Evaluations Special Interest Group ? Time • Chair • Head of CEP • Business Planning Manager • Head of Commissioning and Delivery • Technical Advisor • PASA / Hubs (2 reps) • NICE (1 rep) • NIIII (1 rep) • HTA (1 rep) • NPSA (1 rep) • MHRA (1 rep) Priorities to be aligned to customer requirements & peer-organisations www. hpa-midas. org. uk

Evaluations Prioritisation Evaluation priority survey (pilot) – Serology kits Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs www. hpa-midas. org. uk

Evaluations Prioritisation Evaluation priority survey (pilot) - Nucleic Acid Tests Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs www. hpa-midas. org. uk

Validation: with special thanks to Dr. Ian Sharp & Sally-Ann Finn Definition of validation Validation is the evaluation of a process to determine its fitness for a particular use. Validation is the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled (ISO 17025: 2005) www. hpa-midas. org. uk

Validation Evaluation of the process • It is not the kit or reagent in isolation that is being validated but the whole process that it is being used in to produce the correct result • You are validating your ability to achieve acceptable results with the assay in question www. hpa-midas. org. uk

Validation www. hpa-midas. org. uk

Validation Why do we need to validate assays? • Good laboratory practice • Protection from litigation • CPA requirement • F 1. 2 Examination procedures shall be validated for their intended use prior to introduction, and the methods used and results obtained, recorded. www. hpa-midas. org. uk

Validation What do we need to validate? • Commercial assays –Can use the term commissioning • In house assays • Modified commercial assays • Changes to assays • Equipment-assay combinations www. hpa-midas. org. uk

Validation Commercial assays • Just because they are CE marked does not mean that they do not need to be validated (commissioned) to ensure that they are fit for purpose • Most assays are CE marked following self-declaration by the manufacturer –May be certified to ISO 13485 www. hpa-midas. org. uk

Validation What’s needed? Project team VALID ATION FILE Project lead Project manager Molecular skills Serology skills Statistical skills Validation plan www. hpa-midas. org. uk

Validation file www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel - Numbers www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel - Numbers Confidence intervals should always be quoted www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel - Numbers Carley, Dosman S, Jones SR, Harrison M (2005): Simple nomograms to calculate sample size in diagnostic studies Emerg Med J 2005; 22: 180 -181 www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel – Minimising bias • Compare results with a suitable reference standard / algorithm • Avoid including specimens pre-screened by kits that are part of the evaluation • Avoid including specimens known to share false reactivities with particular groups of kits • Be aware of discontinued use of challenging specimens within a panel (eg due to low volume) which may apparently enhance sensitivity/specificity calculations of kits tested at a later date. www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel – Representative types Include specimens with High, Medium and Low reactivity, and unreactive specimens www. hpa-midas. org. uk

Issues common to Evaluations & Validations Challenging specimen panel – in practice depends on: • • Specimen availability Specimen volume Number of kits to evaluate Specimen quality (eg freeze/thaw cycles) Ethical issues Access to diagnostic specimens Funds eg commercial seroconversion panels www. hpa-midas. org. uk

Practical Considerations • Technical expertise required to perform the assay • Training • Test throughput • Reliability of the supplier • Service provision & continuity • Requirement for specialised equipment • Maintenance contracts • Laboratory space www. hpa-midas. org. uk

Challenges • Responding to the need for a wide range of microbiological device evaluations • Joining device performance with procurement • Earlier assessment and usage of innovative technologies • Improved access to available evaluation results eg National Evaluations Register • Improved sharing of information www. hpa-midas. org. uk

Acknowledgments Evaluations & Standards Laboratory: Ian Sharp Sally-Ann Finn Ruhi Siddiqui Joe Vincini Valerie Bevan Mi. DAS-ESL: Katrina Barlow Michelle Cole Johanna Curtis Laura Dean Galit Gonen Fu Li www. hpa-midas. org. uk

Can You Help? Acquiring specimens www. hpa-midas. org. uk