SDC for continuous variables under edit restrictions Natalie

SDC for continuous variables under edit restrictions Natalie Shlomo & Ton de Waal UN/ECE Work Session on Statistical Data Editing, Bonn, September 2006

Contents l l l The problem Evaluation data SDC techniques l l l Additive noise Microaggregation Rounding Rank swapping Conclusions

The problem l l Statistical disclosure control (SDC): microdata need to be protected against disclosure before release Several SDC-techniques available for continuous microdata l l l Do not take edit constraints into account Inconsistent microdata lead to loss of utility and pinpoint potential intruders to protected data Problem: extend SDC techniques for continuous microdata to take edit constraints into account l l Micro edits – record level inconsistencies Macro edits – overall loss of utility (bias and variance)

Evaluation data l l l 2000 Israel Income Survey with three continuous variables (gross, net and tax) and one control variable (age) 32, 896 individuals of which 16, 232 earned income from salaries Edits: l l l E 1 a: E 1 b: E 1 c: E 2: E 3: gross ≥ 0 net ≥ 0 tax ≥ 0 IF age ≤ 17 THEN gross ≤ 6, 910 net + tax = gross

Additive noise l l Generate random value and add to value to be protected Random value can be drawn in several ways, depending on l l Aiming to preserve variances or not A single variable or multiple variables

Additive noise for a single variable using standard approach l Adding standard noise: perturb Y as follows l l l Y* = Y + e, e drawn from N(0, σ2) Adding random noise to gross with σ2 = 0, 2 x. Var(gross) resulted in 1, 685 failures of E 1 and 119 failures of E 2 Adding standard noise in groups l l Define 5 equal groupings (quintiles) by sorting Within each group applying above method resulted in 66 failures of E 1 and no failures of E 2

Additive noise for a single variable using correlated noise Perturb value Y as follows (Natalie’s trick): l Y* = d 1 Y + d 2 e, l l l d 1 = (1 - δ 2)1/2, d 2 = δ for positive parameter δ e drawn from N((1 -d 1)/d 2 x mean(Y), Var(Y)) Note that l l l E(Y*)= E(d 1 Y) + E(d 2 e) = E(Y) Var(Y*) = (1 - δ 2)Var(Y) + (δ 2)Var(Y) = Var(Y) Linear equations are preserved

Additive noise for multiple variables and linear programming l l l Perturb each variable Yi separately, resulting in Yi* Adjust perturbed values Yi* slightly so that all edits become satisfied (LP-trick) Minimize Σi |Yi* - Yi, final| subject to edit constraints l l Yi, final are final perturbed values Problem is simple linear programming problem

Additive noise for multiple variables using correlated noise Perturb vector Y by applying Natalie’s trick l Y* = d 1 Y + d 2 e, l l d 1 = (1 - δ 2)1/2, d 2 = δ for positive parameter δ e drawn from N((1 -d 1)/d 2 x mean(Y), Var(Y)) mean(Y) mean vector of Y; Var(Y) covariance matrix of Y Means, covariances and equations are again preserved l l l E(Y*) = E(Y) E(Var(Y*)) = E(Var(Y)) Linear equations are preserved

Microaggregation l l l Replace value to be protected by average value in small group Reduction in variance due to elimination of “within” variance Microaggregation can be applied in several ways: l l l Standard version of microaggregation Microaggregation followed by adding noise (to preserve original variance) and using linear programming to ensure preservation of linear equations (LP-trick) Microaggregation followed by adding correlated noise to ensure preservation of linear equations (Natalie’s trick) l Avoids need for LP- trick but does not raise variance to expected level

Results for microaggregation Var. tax SD SD micro- SD original aggregation random noise 2, 119 2, 082 2, 115 SD and LP 2, 103 SD and Natalie’s trick 2, 091 net 5, 137 5, 114 5, 134 5, 129 5, 119 gross 7, 181 7, 174 7, 171

Rounding l l Round value to be protected to multiple of rounding base Rounding can be applied in several ways: l l l Random rounding Controlling totals and additivity, and selecting all rounded values within base of original value

Random rounding l l l Univariate rounding with rounding base b res(X) = X – largest multiple of b less than X Round X up with probability res(X)/b and down with probability 1 - res(X)/b Expectation of rounding is zero In expectation totals are preserved

$Random rounding: controlling totals l Select fraction of res(X)/b random entries to be rounded$

Random rounding: controlling totals l Select fraction of res(X)/b random entries to be rounded upward and round the rest downward l l total is exactly preserved gross is calculated as sum of rounded tax and net l l gross may jump a base apply reshuffling algorithm to correct this

Results for rounding Var. Total Diff. random rounding Diff. controlled totals and totals in base -7 -87 tax 25, 443, 623 -787 net 86, 724, 755 -575 -105 gross 112, 168, 378 -1, 362 -22 -192

Rank swapping l l l Sort variable to be protected and construct groupings, select random pairs in each group and swap values between pairs Different group sizes lead to different results Evaluation criteria: l l AD = Σi |Xi, orig – Xi, pert|/nr l where i is cell in age group (14) x sex (2) x income group (22) BV = Σj nj (averagej(X) – average(X))2/(p-1) l with j=1, . . , p in age group (14) x sex (2)

Results for rank swapping Groupings of 10 Number and Percent of Cells with Differences AD Ratio of BVpert/BVorig Groupings of 20 106 (22%) 166 (34%) 0. 224 0. 338 -0. 03% 0. 58%

Conclusion l l Standard perturbation methods can be extended so they take (micro and macro) edit constraints into account “Best” method to protect data set is to some extent subjective choice l l Must provide protection against disclosure risk according to tolerable risk threshold Must provide fit for purpose data according to needs of users