Guide to Selecting Qualified WET Laboratories Robert N

Guide to Selecting Qualified WET Laboratories Robert N. Brent, Ph. D. Dyn. Corp Science and Engineering Group 3/19/2018

Why is Lab Selection Important? § You, the permittee, are responsible for: • Meeting monitoring frequency requirements • Certifying the quality of data – “I certify under penalty of law …the information submitted is, to the best of my knowledge and belief, true, accurate, and complete…” • Living with the test results – Test failures can initiate additional testing, TIEs, TREs, enforcement actions, fines, public scrutiny

Why is Lab Selection Important? § You are the one paying • In a 1998 WERF Study – C. dubia chronic test: $100 - $2, 300 – Fathead minnow chronic test: $238 - $5, 500 • With multiple outfalls, monthly testing requirements, and multispecies monitoring, test costs can be significant – 3 outfalls X 12 samples/yr X 3 species X $1000/test = $108, 000 • Know what you are paying for!

Why is Lab Selection Important? § Quality does matter • There is a wide range of laboratory performance – Ex: EPA’s Variability Guidance Document Percentiles of Intralaboratory CVs (coefficient of variation) for C. dubia Chronic Test

A Tale of Two Labs Ex: WET Interlaboratory Variability Study

A Tale of Two Labs Ex: WET Interlaboratory Variability Study $9 36 / te st $5 00 / te st

Obstacles to Selecting Qualified Labs § Procurement Regulations • Many permittees are required to select lowest responsive and responsible bidder • This doesn’t always mean selecting lowest bidder – Must be responsive to everything that you request in the solicitation – Must be responsible for delivering exactly what you specify in the contract • Solution: You determine the exact specifications for what you want – Government Ex: $600 hammer – WET Ex: lab must demonstrate intralab variability of <27% • Talk to your procurement specialist about incorporating detailed specifications into the lab solicitation process

Obstacles to Selecting Qualified Labs § Insufficient knowledge and resources for identifying qualified laboratories • Solution: This workshop will provide – Practical tools for evaluating laboratory quality – Things to look for in a qualified lab – Questions to ask – Sample test data sets to quiz your lab

General Evaluation Criteria § Capacity • Will the lab be able to test your samples when you need them tested? • What is the labs maximum and typical capacity (tests per week)? – Look for maximum capacity to greatly exceed typical capacity – Quality usually decreases when operating near capacity limits • What limits capacity? (space, staff, organisms, equipment) – Look for limits that can be easily remedied

General Evaluation Criteria § Staff • What is the level of experience and education of staff from top to bottom? – Remember: it is the staff at the bottom (technicians) that will have the most contact and interaction with your samples – Experienced and well-trained staff is critical to consistently generating high quality data • What is the turnover rate? – Look for low turnover at the top – Turnover at the bottom may be high, but look for rigorous training program

General Evaluation Criteria § Organizational Structure • What is the chain of command? • Who will you be interacting with? – Look for a dedicated account manager – Look for a laboratory contact as well § Certification • Does the lab carry any State or national certifications? – Not all States have lab certification programs, but beware of labs without certification in a State where there is a certification program

General Evaluation Criteria § Historical Performance • How many tests does the lab run annually? – Look for 50 – several hundred (for common test methods) • What percentage of tests were successfully completed (met TAC) without retesting? – Look for 90 -100% • How many other clients do they have with similar waste streams (municipal, industrial, etc. ) to yours? – Familiarity with similar waste streams may lead to better, quicker problem resolution

General Evaluation Criteria § Reporting • Does the lab’s reporting standards and format meet the requirements of your regulatory authority? • Does the lab’s reporting format meet your needs – Look for simple but detailed reports s Ex: You should be able to find the test result (NOEC, IC 25, etc. ) within 60 seconds s Ex: Report should be detailed enough for someone to completely recalculate the test results s Ex: You should be able to find the temperature of treatment A on Day 4 of the test

General Evaluation Criteria § Awareness • Is the lab aware of recent developments in WET testing? – Look for lab to be aware of: s Proposed WET method changes and upcoming new versions of WET manuals s WET Method Guidance Document (EPA 821/B-00/004) s WET Variability Guidance Document (EPA 833/R-00/003)

Quality Control Evaluation Criteria § Organism Source and Quality • Where does the lab get their organisms? – In-house cultures vs. commercial suppliers – Look for consist source • How many organisms are available on a daily basis? – Look for capacity to exceed usage • What QC measures are used to assess organism health? – Look for reference toxicant testing and other measures (reproduction and survival in cultures) • How often are culture crashes experienced?

Quality Control Evaluation Criteria § Dilution Water Source and Quality • What dilution water types does the lab use? • Where does the dilution water come from or how is it prepared? – Look for consistent high quality source – Look for experience with different dilution water types • How does the lab assess dilution water quality? – Look for chemical testing and toxicity testing of new dilution water batches

Quality Control Evaluation Criteria § Reference Toxicant Control Charts • Reference toxicant control charts are one of the best ways to assess laboratory performance (if you know what to look for) • Labs are required to conduct monthly reference toxicant tests • Test results are plotted on a chart to show variability over time

Evaluating Reference Toxicant Control Charts § Look for control charts of LC 50 (for acute methods) or IC 25 (for chronic methods), not NOECs § Look for 20 points § Look for curved limits

Evaluating Reference Toxicant Control Charts § Look at scale Different Scale Same Scale

Evaluating Reference Toxicant Control Charts § Look at CV • CV = Standard deviation / Mean * 100% • Compare CV to national percentiles (Variability Guidance Doc Table B-1) CV = 40% CV = 17%

Evaluating Reference Toxicant Control Charts § Look for results beyond control limits • 1 in 20 will exceed limits by chance alone • Beware of >2 exceedances in 20

Quality Control Evaluation Criteria § Control Charts for Additional QC Measures • Look for control charts of: – Control performance over time (i. e. , control reproduction) – Control CV over time – PMSDs (percent minimum significant difference) over time • Compare with national percentiles from Variability Guidance Document (Tables B-7 to B-8)

Evaluation of PMSD Control Charts § Look for PMSD mean versus national percentiles § Look for PMSDs that exceed 90 th percentile

Quality Control Evaluation Criteria § Statistical Analysis and Endpoint Calculation • Can the laboratory correctly perform the recommended statistics and properly calculate test endpoints? – EPA found that a large percentage of laboratories in the WET Interlaboratory Variability Study made one or more errors in the calculation of test results • Test laboratories with sample data sets – 3 sample data sets provided – Ask laboratories to calculate LC 50, survival NOEC, sublethal NOEC, and IC 25 using the recommended statistical flowcharts and concentration-response evaluation guidance (where appropriate)

Sample Test Data Set #1

Sample Test Data Set #2 § Requires use of concentration-response guidance

Sample Test Data Set #3 § Requires use of concentration-response guidance

Conclusions § It’s your money and your reputation at stake: • Choose a quality laboratory § When you find a quality laboratory, stick with them § When you don’t know, ask