DQO Training Course Day 3 Module 16 The

DQO Training Course Day 3 Module 16 The EPA 7 -Step DQO Process Step 7 - Optimize Sample Design Presenter: Sebastian Tindall 60 minutes 1 of 50

Step 7: Optimize Sample Design Step 1: State the Problem Step 2: Identify Decisions Step 3: Identify Inputs Step 4: Specify Boundaries Step 5: Define Decision Rules Step Objective: Identify the most resource effective data collection and analysis design that satisfies the DQOs specified in the preceding 6 steps Step 6: Specify Error Tolerances Step 7: Optimize Sample Design 2 of 50

Terminal Objective To be able to use the output from the previous DQO Process steps to select sampling and analysis designs and understand design alternatives presented to you for a specific project 3 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 4 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop alternative sample designs The outputs should provide information on the context of, requirements for, and For each design option, select needed constraints on data collection mathematical expressions design. Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 5 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Go back to Steps 1 - 6 and revisit decisions. Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Based satisfies the Select the optimal sample size that on the DQO outputs from Steps 1 -6, for DQOs for each data collection design option each decision rule develop one or more sample designs to be Check if number of samples exceeds evaluated in considered and project resource constraints Step 7. Optimal Sample No Yes Design 6 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed For each option, pay close attention mathematical expressions Gray Region to the Step 4 outputs defining the population to be represented Select the optimal samplewith the data: size that satisfies the Remember: DQOs for each data collection design option method • Sample collection Sampling Uncertainty is decreased • Sample mass size when sampling density is increased. • Sample particle Check if number of samples exceeds size Go back to Etc. project resource • constraints Steps 1 - 6 Optimal and revisit Sample No Yes decisions. Design 7 of 50

Sampling Probability-samples (P-S), for which sampling errors can be calculated, and, for which the biases of selection and estimation are virtually eliminated or contained within known limits. Judgment-samples (J-S), for which the biases and sampling errors can not be calculated from the sample but instead must be settled by judgment. Deming, W. E. , 1950, Some Theory of Sampling, Dover Publications, New York 8 of 50

Sampling Non-Probabilistic selection: When certain constituent elements of the lot to be evaluated have a zero probability of being taken into the sample. Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 28 9 of 50

Types of P-S Designs Simple Random n Systematic Grid with random start n Geometric Probability or “Hot Spot” Sampling n Stratified Random n – Stratified Simple Random – Stratified Systematic Grid with random start Statistical Methods for Environmental Pollution Monitoring, Richard O. Gilbert, 1987 10 of 50

Simple Random Definition- choice of sampling location or time is random n Assumptions n – Every portion of the population has equal chance of being sampled – Population is “relatively homogeneous” n Limitations: – may not cover area 11 of 50

Simple Random n To generate a simple random design: – Either grid the site - set up equalateral triangles or equal side rectangles and number each grid, use a random number generator to pick the grids from which to collect samples – Randomly select x, y, z coordinates, go to the random coordinates and collect samples 12 of 50

Example - Simple Random Using Coordinates 13 of 50

Systematic Grid, Random Start n n Definition-taking measurements at locations or times according to spatial or temporal pattern (e. g. , equidistant intervals along a line or grid pattern) Assumptions – Good for estimating means, totals and patterns of contamination – Good when population is not “relatively homogeneous” 14 of 50

Systematic Grid, Random Start (cont. ) n Limitations – Biased results can occur if assumed pattern of contamination does not match the actual pattern of contamination – Inaccurate if have serial correlation n NPDES outfall – Periodic recurring release; time dependent n Groundwater: – seasonal recurrence; water-level dependence 15 of 50

Systematic Grid, Random Start (cont. ) Remember: Start at random location Move in a pre-selected pattern across the site, making measurements at each point 16 of 50

Geometric Probability or Hot. Spot Sampling Uses squares, triangles, or rectangles to determine whether hot spots exist n Finds hot spot, but may not estimate the mean with adequate confidence n 17 of 50

Geometric Probability or Hot. Spot Sampling (cont. ) n n Number of samples is calculated based on probability of finding hot area or geometric probability Assumptions – Target hot spot has circular or elliptical shape – Samples are taken on square, rectangular or triangular grid – Definition of what concentration/activity defines hot spot is unambiguous 18 of 50

Geometric Probability or Hot. Spot Sampling (cont. ) n Limitations – Not appropriate for hot spots that are not elliptical – Not appropriate if cannot define what is hot or the likely size of hot spot 19 of 50

Example Grid for Hot-Spot Sampling 20 of 50

Action Limit = 50. 0 ppm Average = 33. 8 ppm 21 of 50

Geometric Probability or Hot. Spot Sampling (cont. ) n In order to use this approach the decision- makers MUST – Define the size of the hot spot they wish to find – Provide rationale for specifying that size. – Define what constitutes HOT (e. g. , what concentration is HOT) – Define the effect of that HOT spot on achieving the release criteria 22 of 50

Stratified Random n n Definition-divide population into strata that are “relatively homogeneous” and collect samples in each strata randomly Attributes – – n Provides excellent coverage of area Need process knowledge to create strata Yields more precise estimate of mean Typically more efficient then simple random Limitations – Need process knowledge – Assumes population is “relatively homogenous” 23 of 50

Example - Stratified Simple Random Strata 1 Strata 2 24 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Go back to Steps 1 - 6 and revisit decisions. Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the DQOs for each data collection design option 1. Statistical Method/Sample Size Formula 2. Cost Function Check if number of samples exceeds project resource constraints Optimal Sample No Yes Design 25 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the DQOs for each data collection 1. Statistical Method/Sample Size Formula design option Define suggested method(s) for testing the statistical hypothesis and define sample size formula(e) that correspondsexceeds Check if number of samples to the method(s). Go back to project resource constraints Steps 1 - 6 Optimal and revisit Sample No Yes decisions. Design 26 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the DQOs for Perform a preliminary DQA: each data collection design option • Generate frequency distribution histogram(s) for each population • Select one or more statistical number of samples exceeds PSQs Check if methods that will address the • Go back to List the assumptions for choosing these statistical methods project resource constraints • Steps 1 - 6 appropriate formula for calculating the number of List the and revisitn samples, No Yes decisions. Optimal Sample Design 27 of 50

3 Approaches for Calculating n “Symmetrical” distribution approach n Skewed distribution approach n FAM/DWP approach n – Badly skewed or for all distributions use computer simulation approach • e. g. , Monte Carlo Show Monte Carlo simulation (Coming Soon!) 28 of 50

Begin With the Decision in Mind Contaminant Concentrations in the Spatial Distribution of the Population Frequency Distribution Correct Equation for n (Statistical Method) Data • field • onsite methods • traditional laboratory , , , Alternative Sample Designs Optimal Sampling Design How Many Samples do I Need? The end 29 of 50

Logic to Assess Distribution and Calculate Number of Samples 30 of 50

Logic to Assess Distribution and Calculate Number of Samples 31 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the 2. Cost Function optimal sample size that satisfies the DQOs for each data collection design option For each selected sample size, develop a cost function that relates the number of samples to the total cost of sampling and analysis. exceeds Check if number of samples Go back to project resource constraints Steps 1 - 6 and revisit No Yes decisions. Optimal Sample Design 32 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Review DQO outputs from Steps 1 -6 to be sure they are internally consistent Information OUT To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the In order to develop the cost function, the aggregate unit cost per DQOs for each data cost of collecting one sample must be determined. This is thecollection design option sample and conducting all the required analyses for a given decision rule. Check if number of samples exceeds Go back to project resource constraints Steps 1 - 6 and revisit No Yes decisions. Optimal Sample Design 33 of 50

Aggregate Unit Sampling and Analysis Cost j AUSCA$ = USC$ + USA$ i i=1 Where (here): USC$ = Unit Sample Collection Cost USA$ = Unit Sample Analysis Cost AUSCA$ = Aggregate Unit Sample Collection and Analysis Cost j = Number of analytical methods planned 34 of 50

Step 7 - Optimize Sample Design Information IN Actions Information OUT From Previous Step Review DQO these methods, calculate Using the formulae appropriate to outputs from Steps 1 -6 to be required, varying , consistent the number of samples sure they are internally for a given . Repeat the same process using new s. Decision Error Develop alternative sample designs Tolerances Review all of calculated sample sizes and along with To Next Step their corresponding levels of , , and . For each design option, select needed mathematical expressions Gray Region Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 35 of 50

Step 7 - Optimize Sample Design Information IN Actions Merge the selected sample size outputs with From Previous Step Unit Sample Collection and the Aggregate Review DQO outputs from Steps 1 -6 to Analysis cost output. sure they are internally consistent be Information OUT To Next Step Decision Error a table that shows the product This results in Develop alternative sample Tolerances of each selected sample size and the AUSCA$. designs This table is used to present thedesign option, select needed For each project managers and decision makers with a range mathematical expressions Grayanalytical costs and the resulting Region of uncertainties. Select the optimal sample size that satisfies the DQOs for each data collection design option From the Check select table, if number of samples exceeds Go back to the optimal sample size that project resource constraints Steps 1 - 6 meets the project budget and revisit and uncertainty requirements. No Yes decisions. Optimal Sample Design 36 of 50

Step 7 - Optimize Sample Design Observe the uncertainty a given cost (and number of samples) Actions Information OUT will “buy”. From Previous Step To Next Step Review DQO outputs from Steps 1 -6 to be sure they are have acceptable levels Select those sample sizes that internally consistent of , , and (uncertainty) associated with them. Decision Error Develop alternative sample Tolerances will exceed the project sampling and designs budget: If this analysis Information IN -Increase the budget For each design option, select needed -Relax the error tolerances (accept greater uncertainty) mathematical expressions Gray Region Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 37 of 50

SHOW SCA EXCEL File 38 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop design meets the error tolerances If no sample alternative sample designs within the budget: relax one or more of the constraints or request option, select needed For each design more funding, etc. mathematical expressions Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 39 of 50

Iterative Process Steps 1 - 6 Step 7 Optimal Design 40 of 50

Step 7 - Optimize Sample Design Information IN Actions Information OUT From Previous Step Review DQO outputs from Steps 1 -6 to Justification for bejudgmental sampling design a sure they are internally consistent To Next Step Timeframe Decision • Error • Qualitative. Develop alternative sample designs consequences of an inadequate sampling Tolerances design (low, moderate, severe) • Re-sampling access after decision has been made For each design (accessible or inaccessible) option, select needed mathematical expressions Gray Region Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 41 of 50

Sampling Probability-samples, for which sampling errors can be calculated, and, for which the biases of selection and estimation are virtually eliminated or contained within known limits. Judgment-samples, for which the biases and sampling errors can not be calculated from the sample but instead must be settled by judgment. Deming, W. E. , 1950, Some Theory of Sampling, Dover Publications, New York 42 of 50

Sampling Non-Probabilistic selection: When certain constituent elements of the lot to be evaluated have a zero probability of being taken into the sample. Sampling for Analytical Purpose, Pierre Gy, J. Wiley & Sons, 1998; pg 28 43 of 50

WARNING!! If a judgmental design is selected in lieu of a statistical design the following disclaimer must be stated in the DQO Summary Report: “Results from a judgmental sampling design can only be used to make decisions about the locations from which the samples were taken and cannot be generalized or extrapolated to any other facility or population, and error analysis cannot be performed on the resulting data. Thus, using judgmental designs prohibits any assessment of uncertainty in the decisions. ” 44 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop alternative sample designs is the most The output resource-effective design for the study that For each design option, select needed is expected to achieve mathematical expressions the DQOs. Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 45 of 50

Data Quality Assessment Step 1: Review DQOs and Sampling Design n Step 2: Conduct Preliminary Data Review n Step 3: Select the Statistical Test n Step 4: Verify the Assumptions of the Test n Step 5: Draw Conclusions From the Data n Guidance for Data Quality Assessment, EPA QA/G 9, 2000 46 of 50

Summary To succeed in a systematic planning process for environmental decision making, you need Statistical Support: One or more qualified statisticians, experienced in statisticians environmental data collection designs and statistical data quality assessments of such designs. 47 of 50

Summary (cont. ) n n Going through the 7 -Step DQO Process will ensure a defensible and cost effective sampling program In order for the 7 -Step DQO Process to be effective: – Senior management MUST provide support – Inputs must be based on comprehensive scoping and maximum participation/contributions by decision makers – Sample design must be based on the severity of the consequences of decision error – Uncertainty must be identified and quantified 48 of 50

Step 7 - Optimize Sample Design Information IN From Previous Step Decision Error Tolerances Gray Region Actions Information OUT Review DQO outputs from Steps 1 -6 to be sure they are internally consistent To Next Step Develop alternative sample designs For each design option, select needed mathematical expressions Select the optimal sample size that satisfies the DQOs for each data collection design option Go back to Steps 1 - 6 and revisit decisions. Check if number of samples exceeds project resource constraints Yes No Optimal Sample Design 49 of 50

End of Module 16 Thank you Questions? 50 of 50