Скачать презентацию CAESAR Computer Assisted Evaluation Screening And Reporting Скачать презентацию CAESAR Computer Assisted Evaluation Screening And Reporting

bdeaf8a200c82617d71e57482051df1e.ppt

  • Количество слайдов: 60

CAESAR Computer Assisted Evaluation – Screening And Reporting The Cervical Screening Wales Focal Point CAESAR Computer Assisted Evaluation – Screening And Reporting The Cervical Screening Wales Focal Point Project Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th, 2009

Acknowledgements • The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Acknowledgements • The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Wrexham • Bryan Rose • Helen Beer and the CSW information team • David Addington-Hall and Karen Winder for advice during the scoping phase • Drs Nick Dallimore, Sally Williams and Tom Hockey • Wilma Anderson and Luke Nottage of

Background • Rapidly changing programme – HPV Vaccination – New technology (HPV testing, Computer Background • Rapidly changing programme – HPV Vaccination – New technology (HPV testing, Computer Assisted Screening!!) – Modernisation (Sc. HARR 1 and NHS CSP 2 guidance, WAG pathology modernisation initiatives 3) 1. 1. 2. Option appraisal: Assessment of a seven-day turn-around for the reporting of cervical smear results. School of Health and Related Research, University of Sheffield. Jan 2006 NHS CSP: Achieving a 14 day turnaround for results by 2010. Advice to the NHS. April 2008. Welsh Assembly Government. Getting Results: A Strategy for Diagnostic Services in Wales, 2004. Diagnostic Services Programme: The Future Delivery of Pathology Services in Wales, 2007

Background • Pressures on the Laboratory Service – Staff leaving – Staff recruitment and Background • Pressures on the Laboratory Service – Staff leaving – Staff recruitment and training – Fewer cases being screened – can we maintain our skills? – Challenges around service delivery – CAS is expensive technology and if beneficial, will require a laboratory service that is structured to take full advantage of it

Background What is Rapid Quality Assurance screening? • Designed to improve the quality of Background What is Rapid Quality Assurance screening? • Designed to improve the quality of primary screening in the cytology lab • Can be either preview or review • Also used to detect false negative and positive cases • Is therefore a requirement in calculating individual and laboratory sensitivity and specificity

Why CAS? • Since the introduction of LBC, screener sensitivity has improved. Calculation of Why CAS? • Since the introduction of LBC, screener sensitivity has improved. Calculation of sensitivities is becoming increasingly invalid as the manual QA methods have become unfit for this purpose • Anecdotally, Rapid Preview/Review isn't particularly well liked by staff

 • Some staff perform better at QA screening than others – leads to • Some staff perform better at QA screening than others – leads to inconsistencies which will affect the calculation of individual and laboratory sensitivities • Automated screening might therefore provide a more sensitive QA approach that is also more consistent in its application

Introducing the Becton Dickenson Focal Point Slide Profiler Introducing the Becton Dickenson Focal Point Slide Profiler

Focal Point GS – An Overview • The BD Focal. Point™ slide profiler is Focal Point GS – An Overview • The BD Focal. Point™ slide profiler is an automated imaging system designed to assist in the primary screening of Sure. Path™ and conventional Papanicolaou cervical cytology slides. • Designed to detect slides with evidence of squamous carcinoma and adenocarcinoma and their usual precursor conditions. • Up to 300 individual features are evaluated by the BD Focal. Point™ by using both morphometric and densitometric algorithms. • The BD Focal. Point™ identifies up to 25% of successfully processed slides as requiring No Further Review (NFR). • When used in conjunction with a BD Focal. Point™ Guided Screener Review Station the 10 highest scoring Fields Of View (FOV) from each slide are presented to the screener for review via an automated stage fitted to a standard laboratory microscope

Loading Focal. Point Add barcode labelled Sure. Path™ slides to a slide tray (8 Loading Focal. Point Add barcode labelled Sure. Path™ slides to a slide tray (8 slides per tray) LOAD & LEAVE Load slide trays into input hopper of BD Focal. Point™ slide profiler (max 36 slide trays at any one time = 288 slides) Racks can be loaded and unloaded at any time Minimum 120 slides per run There is no maximum number of slides per run

BD Focal. Point™ Slide Profiler processing steps • 20 x Scan (High Resolution) • BD Focal. Point™ Slide Profiler processing steps • 20 x Scan (High Resolution) • Each FOV scanned twice • A high resolution image is acquired for each of the 1000 FOV’s • FOV processors - segment all objects in each acquired FOV image (segmentation = separation of meaningful objects from background) • Objects classified as single cells, groups and thick groups • A single cell, group and thick group score is given for each FOV • The FOV scores are accumulated and integrated into a final slide score between 0 and 1

BD Focal. Point™ GS Imaging Workflow: GS Review Station BD Focal. Point™ Slide Profiler BD Focal. Point™ GS Imaging Workflow: GS Review Station BD Focal. Point™ Slide Profiler Database

BD Focal. Point™ GS Imaging data transfer: Focal. Point GS GS Review Station Data BD Focal. Point™ GS Imaging data transfer: Focal. Point GS GS Review Station Data transfer via Data transfer Removable hard-drive or VPN connection Server PC GS Review Station

Principal Hypothesis • The Focal Point GS slide profiler technology is a acceptable replacement Principal Hypothesis • The Focal Point GS slide profiler technology is a acceptable replacement for the currently establis processes of quality assurance in primary screeni

The Project Aims • Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility The Project Aims • Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles • Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

The Project Aims • Evaluate Focal Point GS as a means of assessing sample The Project Aims • Evaluate Focal Point GS as a means of assessing sample quality – establishing baseline parameters for assessing adequacy of samples by creating a reproducible parameter for the assessment of sample taker performance • Evaluate Focal Point GS across an integrated laboratory network – assessing sample identification, tracking and transportation processes, workload allocation patterns and logistics and user friendliness

The Project Aims • Monitoring screener performance using the Focal Point Slide Profiler – The Project Aims • Monitoring screener performance using the Focal Point Slide Profiler – This technology has the capability to monitor operator/observer functions such as screener opinion and action, time taken to perform screen. – These data items - gathered using the Location Guided Screener workstation (LGS) can be compared against primary screening and/or final outcome in order to provide screener performance indicators.

Examine Operational Issues • Identify sample identification issues e. g. bar codes and maintaining Examine Operational Issues • Identify sample identification issues e. g. bar codes and maintaining “chain of custody” across a network • How does the technology deal with inadequate samples – does this compare with current adequacy criteria? • Does the technology compare with manual techniques in terms of operator time?

PROCESS PROCESS

Process • Samples received and processed at laboratory as normal • Despatched to Medical Process • Samples received and processed at laboratory as normal • Despatched to Medical Solutions for FP scanning • Returned with data disc for Location Guided Screen (LGS) assessment in laboratory Automated Arm) • Manually screened independently of the LGS evaluation (Manual Arm) • Results correlated before final report issued

The ‘Automated Arm’ • Consisted of screener evaluation of 10 fields of view (FOV) The ‘Automated Arm’ • Consisted of screener evaluation of 10 fields of view (FOV) on each sample • Difficult cases were referred to a checker who also examined the 10 FOV • Findings were recorded on the proforma illustrated overleaf • Outcome was recorded on LIMS (Tele. Path)

The ‘Manual’ Arm • Slides were primary screened and QA screened as usual following The ‘Manual’ Arm • Slides were primary screened and QA screened as usual following standard laboratory protocols • Codes and reports were recorded on Tele. Path • Automated arm report codes were recorded – FPQ or FPQQ codes were recorded depending on whether or not a discrepancy was noted between the two arms • Cases were authorised in the normal manner • CSW retrieved FP/LGS data as part of weekly downloads

CERVICAL SCREENING WALES FOCAL POINT PROJECT - PROCESS MAP AUTOMATED ARM COURIER SAMPLE TO CERVICAL SCREENING WALES FOCAL POINT PROJECT - PROCESS MAP AUTOMATED ARM COURIER SAMPLE TO LAB SLIDES TO NOTTINGHAM PROCESS HARD DRIVE + PRINT OUT Process Review data INDIVIDUALS CREENERS OPINION OF 10 F. O. V. CODE RESULTS ON TELEPATH RESULTS AND SLIDES RETURNED TO LAB SLIDE WIZARD NETWORK DATA TRANSFER AVAILABLE DESIGNATED PROCEDURE CODES FULL SCREEN Via Manual Arm SLIDES ARE READ ON FOCAL POINT QC DOUBLE SCREEN/ RAPID REVIEW RESULTS TO CSW MANUAL ARM PROJECT EVALUATION

Data and Morphology Anomaly Checking • Automated: manual arm mismatches were highlighted by the Data and Morphology Anomaly Checking • Automated: manual arm mismatches were highlighted by the CSW Information department as and when they arose • Any resultant slide morphology issues were addressed in team assessment sessions at Med Sol in Nottingham and resolved by: - LGS/multi-header microscope sessions involving representatives from Labs / CSW/ Med Sol

RESULTS RESULTS

Total Slides Processed Total Slides Processed

QA beyond the Lab Focal Point GS as a means of assessing sample quality QA beyond the Lab Focal Point GS as a means of assessing sample quality • FP reports on the Squamous and Endocervical components detected in the sample • Potentially of use to the screening programme as an indicator of smear taker proficiency – but emotive!!! • Wales monitors TZ reporting – however generally a wide spread – between 80% - 96% reported by NW labs • The study has shown that the FP reported endocervical component detection rates in NW labs only varied between 79% and 85% • Possibly screener interpretation – CAS may provide a more uniform assessment • Further work is ongoing in this area

Focal Point GS performance across an integrated laboratory network • VPN data transfer means Focal Point GS performance across an integrated laboratory network • VPN data transfer means that scanned slide information is available anywhere within the combined laboratory service • Economies of scale are definite considerations when planning the introduction of this technology – possible consideration for LBC procurement • Therefore, in this operational scenario – probably more efficient to combine the sample processing and CAS on one processing site.

Teaching and Case review using the Focal Point GS • LGS can mark and Teaching and Case review using the Focal Point GS • LGS can mark and record interesting features within a slide and direct the user to these areas on the slide on demand • With a system configured using VPN these slides can be viewed anywhere on the network • Considerable potential for teaching and MDT use when combined with a multi-head microscope/projector

Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles

Sorting & Ranking (Illustration) The Focal. Point™ SORTS and RANKS slides between the values Sorting & Ranking (Illustration) The Focal. Point™ SORTS and RANKS slides between the values of 0 and 1 : based on the likelihood of abnormality being present. 0 is negative : 1 is abnormal x < 25% Archive > 15% QC > 75% Review Negative Abnormal Q 5 Q 4 Q 3 Q 2 Q 1

FP Quintile Distribution Note: this category Contains PR as Well as NFR samples FP Quintile Distribution Note: this category Contains PR as Well as NFR samples

FP Quintile Distribution FP Quintile Distribution

Focal Point Scant Cellularity Category (FPSC) Compared to a Final Report of Inadequate –Comparison Focal Point Scant Cellularity Category (FPSC) Compared to a Final Report of Inadequate –Comparison of the inadequacy levels showed that the manual arm had a similar inadequate rate – not what was expected with Wales operating a 15000 cell threshold –Interestingly the majority of cases did not correspond between the manual and automated arms

“NFR” – DOES IT WORK? Assessment of the Focal Point Slide Profiler NFR (No “NFR” – DOES IT WORK? Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

Samples Assigned for NFR Compared to Final Result Samples Assigned for NFR Compared to Final Result

What was missed by NFR? 1. Cases from all labs, NFR negative, finally reported What was missed by NFR? 1. Cases from all labs, NFR negative, finally reported as abnormal 78 cases in a total of 2842 (Picked up by 10 screener, reviewer or checker) 2. 74% of cases missed by NFR – False Negatives

What was missed by NFR? 1. In over 50 000 cases manually rapid previewed What was missed by NFR? 1. In over 50 000 cases manually rapid previewed by 4 Welsh Labs during Jan – September 2009 (Picked up by 10 screener, reviewer or checker) The overall False Negative Rate was 3. 5% Statistically – Significant difference from 1. NFR is therefore at least as effective as Manual Rapid Pre-screen as a QA tool for primary screening of cervical samples.

Overall Sensitivity • In total, the automated QC screening arm did not “identify” 12 Overall Sensitivity • In total, the automated QC screening arm did not “identify” 12 HG cases (histologically confirmed) in a total of 134 reported Sensitivity (134 -12)/134 = 91% • Compare this to a quoted range of 75 -95%* HG sensitivity for conventional Rapid Preview QA methods • 202 LG cases were missed or consigned to NFR in 788 reported LG samples Sensitivity (788 -202)/788 = 74. 4% * See references and CSW data

But………. ! • On review, morphological abnormalities were present in the majority of 10 But………. ! • On review, morphological abnormalities were present in the majority of 10 FOVs in the HG cases “missed “ by the automated arm. • So potentially therefore, 91% is the minimum sensitivity rate that can be expected • This is because the Focal. Point GS presented the abnormalities to the LGS user, but not identified by the user for a number of possible reasons • This may be due to a number of reasons e. g. unfamiliarity and further analysis currently being performed on this data

In Summary……… • Automated QC screen appears to be no less effective than rapid In Summary……… • Automated QC screen appears to be no less effective than rapid pre-screen • Most (161/219) cases not identified by automated arm were borderline HPV cases – FP does not seem to recognise koilocytes! – especially with minimal nuclear abnormalities. • Despite this finding, sensitivity of the automated QC screen appears to be appreciably greater in comparison to available data on manual QC methods.

User experience is a large factor in the efficiency of the screen • A User experience is a large factor in the efficiency of the screen • A high proportion of the undetected abnormalities were shown to be present in the FOV’s on review – but not identified by the screener. • Staff had only received 6 -8 months experience of the technology - further work is currently ongoing to see if screener performance improved with time/experience

Monitoring laboratory and screener performance using the Focal Point Slide Profiler Monitoring laboratory and screener performance using the Focal Point Slide Profiler

Individual Laboratory Process Review Results Process Review: Is an indicator of the number of Individual Laboratory Process Review Results Process Review: Is an indicator of the number of failed slide scans in a given FP run It reflects technical quality, bar code integrity etc. Note the low value for YGC, which has been mainly attributed to tape coverslipping

Screening times for 10 FOV Note the remarkable similarity of the mean screening times. Screening times for 10 FOV Note the remarkable similarity of the mean screening times. Note also that NFR category accounts for 22% of samples - no FOVs, therefore potential productivity increase.

Manual QC Screening Times (includes data entry on LIMS) Manual QC Screening Times (includes data entry on LIMS)

OBSERVATIONS OBSERVATIONS

Observations • Thought that FP operates on TBS inadequacy threshold of 5 -6 k Observations • Thought that FP operates on TBS inadequacy threshold of 5 -6 k cells, therefore we hypothesised that the manual arm would identify a correspondingly higher number of inadequate samples, however……………. . – Comparison of the inadequacy levels showed that the manual arm had only a slightly higher inadequate rate – not as expected with Wales operating a 15000 cell threshold – Interestingly the majority of cases did not correspond between the manual and automated arms – possibly indicative of the fact that the FPSC report is applied to cases that exhibit other criteria, such as:

Observations – The slide cannot be focussed on correctly because of bubbles, too thick, Observations – The slide cannot be focussed on correctly because of bubbles, too thick, too few cells in the centre of the prep. etc. – It is therefore likely that the actual number of cases reported as FPSC for reasons related to cellular adequacy are lower than first thought – Further work will be undertaken to determine threshold values of the criteria that make up the FP report “Insufficient Reference Cells” to determine if this could be compared with the manual arm report of “Inadequate due to scant cellularity”

Observations • Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby Observations • Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby reducing the process review rate and decreasing the number of rejected slides. • Selecting a large coverslip size on the tape cover slipper extended the FP scan time from approx. 11 slides/min on glass coverslips to 9 slides/min using tape coverslips. Reducing the tape size reduced this to 10 slides/min* • Staff used to primary screening on a x 20 objective may have initial difficulties in operating the LGS which only displays FOVs at x 10 • *Note: Updated versions of the FP GS allow faster scan rates

Observations • Staff have varying ability to correctly interpret the 10 FOVs and like Observations • Staff have varying ability to correctly interpret the 10 FOVs and like manual screening, are only as good as the experience they have – both as primary screeners and users of the LGS • A small proportion of slides simply cannot be evaluated on 10 FOVs – these should be subjected to a full scan and further checking, if necessary • Despite best efforts to blind manual and automated arms interpretational bias may have resulted as staff were aware that this was a study

Observations • The study resulted in additional workload for staff which might also have Observations • The study resulted in additional workload for staff which might also have affected interpretation • Clearly defined protocols are mandatory for routine use • Transport of slides and data via external couriers posed a few challenges! – eventually settled down

RECOMMENDATIONS RECOMMENDATIONS

Recommendations • A number of issues have yet to be fully evaluated, including: – Recommendations • A number of issues have yet to be fully evaluated, including: – Completing comparisons between available manual QC data and project results – Full evaluation of NFR – do we need to continue with manual QC for these cases? – Subsequent effects on workload/staffing – Implementation and logistical issues – Full Cost/Benefit analysis

Recommendations • Cost/benefit analysis – Need to further evaluate the potential of the NFR Recommendations • Cost/benefit analysis – Need to further evaluate the potential of the NFR category – With demonstrated > 99% False Negative rate for HG disease in 22% of cases – potentially labour saving whilst improving quality! – This is in the region of 1100 hours screener time in Wales with an annual workload of 200 000+ cases

CONCLUSIONS CONCLUSIONS

Conclusions Analysis of the findings of this study are not yet complete. At this Conclusions Analysis of the findings of this study are not yet complete. At this stage initial results indicate that the technology is suitable for routine use, with potential quality and service management benefits to the cervical screening programme in Wales

Recommendations for Further Investigation • CSW AWMG have approved an extension of the project Recommendations for Further Investigation • CSW AWMG have approved an extension of the project and in collaboration with Medical Solutions will be working towards full use of FP as a QA system. The network of laboratories will be extended to include a fourth laboratory from South Wales – at the Royal Gwent Hospital, Newport – project commenced July 2009 • Define implemention plan – Networking issues – the technology seems particularly suited to networks. Wales is currently exploring the introduction of managed networks in cytology • Develop protocols – Specifically issues include standardisation of manual rapid QA methods across labs to rapid pre-screen – IT coding protocols require total standardisation – part of N Wales objectives. Data extraction will become more consistent

Recommendations for Further Investigation • Implementation – QA use initially – follow on project Recommendations for Further Investigation • Implementation – QA use initially – follow on project to implement FP as a replacement for QC screening and further assessment as a primary screening tool, especially in the light of the changing screening programme – Move to full adoption subject to satisfactory outcome of project and final approval by AWMG – Outcome will inform LBC procurement process

“Any sufficiently advanced technology is indistinguishable from magic. ” Arthur C. Clarke – The “Any sufficiently advanced technology is indistinguishable from magic. ” Arthur C. Clarke – The Lost Worlds of 2001 (1972)