Outline Test bias definitions

Outline ¢ ¢ ¢ Test bias – definitions The basic issue: group differences What causes group differences? Arguments that tests are not biased Differential item functioning analysis Criterion-related sources of bias

Outline ¢ Other approaches to testing minority groups Chitling test l BITCH test l SOMPA Models of test Bias l Regression l Constant Ratio l Cole/Darlington l Quota l ¢

Test bias – definition ¢ A test is biased if it gives a systematically wrong result when used to predict something. ¢ So, an intelligence test would be biased if, for example, it underestimated one group’s probability of success in a given endeavor.

Test bias – the basic issue ¢ Various groups within society differ in their average scores on some psychological tests. ¢ We don’t know what causes these differences.

What causes group differences? ¢ Some candidate accounts: ¢ ¢ ¢ Genetics Socioeconomic factors Caste Culture Stereotype threat

Arguments that tests are not biased ¢ ¢ Major tests have been subjected to impressive scrutiny for decades Enormous resources are devoted to this purpose ¢ Criterion validity has been established very securely for the major intelligence tests – they do predict college and job performance

Arguments that tests are not biased ¢ It is not appropriate to focus on individual items on a test, which some critics of testing do ¢ Items should be drawn from a variety of domains, not all of which will be familiar to anyone

Arguments that tests are not biased ¢ Test developers evaluate tests on the basis of overall patterns of prediction utility ¢ They’re futureoriented, not pastoriented: l l “How will you do in college or in a job? ” Not “have you had the opportunity to learn? ”

Arguments that tests are not biased ¢ Do you think of test score results as “outcomes” or as “information” (predictors)? ¢ ¢ Test developers say, results are the beginning, not the end – they are information that will guide us Opponents see test results as outcomes

Arguments that tests are not biased ¢ Systematic studies have asked whether biased items produce group differences on tests such as Stanford-Binet and Wechsler tests ¢ These studies found no evidence that group differences disappeared when allegedly biased items were removed

Argument that tests are not biased ¢ Group differences just as large on what is considered the most culture fair test, Ravens Progressive Matrices, as on WAIS ¢ IQ scores have same utility for prediction regardless of race or socio-economic status.

Differential item functioning analysis ¢ ¢ In this approach to testing for bias, you first form groups for comparison which are equated on overall test score Implication: groups are equivalent in overall ability ¢ ¢ Then, you look for differences between groups on individual items Where difference is found, you conclude that the item is biased (since groups are not different on ability)

Differential item functioning analysis ¢ But removing such items does not eliminate group differences ¢ ¢ E. g. , people depicted in test items may typically be White & male But changing this has little effect (Mc. Carty, Noble, & Huntley, 1989)

Criterion-related sources of bias ¢ We evaluate criterion validity by looking at correlation between test scores and criterion scores ¢ E. g. , SAT scores vs. GPA after 4 years at university

Criterion-related sources of bias ¢ If correlation is good, we use test scores (e. g. , SAT) to predict criterion – and make selection decisions ¢ ¢ What do we do if the correlation is different for different groups? This would imply that test scores mean different things for different groups

Criterion-related sources of bias In this graph, Group B performs better than Group A but the correlation is the same for both Criterion ¢ Group B Group A Test score

¢ ¢ In this graph, the slopes of the lines are the same but the intercepts are different Equal slopes means equal correlations – that is, equally good predictions Criterion-related sources of bias Group B Group A Test score

Criterion-related sources of bias ¢ ¢ Here, the intercepts are different and the slopes are different, so predictions for Groups A and B would not be equally good Such cases are rare Group B Group A X 1 X 2

Criterion-related sources of bias ¢ Major tests, such as SAT and WISC-R, have equal criterion validity for various ethnic groups (e. g, African-American, White, Latino/Latina) ¢ Similar results have been found in other multi-ethnic countries, such as Israel

Other approaches to testing minority groups The Chitling Test ¢ The BITCH Test ¢ SOMPA ¢

The Chitling Test (Dove, 1968) ¢ ¢ Developed to make a point about testing for information a group is unlikely to have acquired Questions require a particular form of “street smarts” to answer correctly ¢ ¢ No validity data exist for this test If you want to predict college performance for minority students, this test won’t help

The BITCH test (Williams, 1974) ¢ Task: define 100 words drawn from the Afro-American Slang Dictionary and Williams' personal experience ¢ ¢ African-Americans score higher than Whites Williams argues that this test is analogous to the standard IQ tests, which are also culture-bound

The BITCH test (Williams, 1974) ¢ ¢ Problem: there is no reason to accept the claim that this is an intelligence test. There is no validity evidence – no prediction of any performance ¢ ¢ Does not test reasoning skills May have some value for testing familiarity with African-American culture

SOMPA (Mercer, 1979) ¢ ¢ System of Multicultural Pluralistic Assessment Based on idea that what constitutes knowledge is socially-constructed ¢ Mercer also suggested that IQ tests are a tool Whites use to keep minority groups “in their place”.

SOMPA (Mercer, 1979) ¢ Inspired originally in part by overrepresentation of minority group children in EMR classes in US schools ¢ Mercer: this overrepresentation resulted from both l l More medical problems Unfamiliar cultural references on tests

SOMPA (Mercer, 1979) ¢ Fundamental assumption: all cultural groups have the same potential on average ¢ On this view, if one cultural group does more poorly than another on a test, that is a fact about the test, not the groups.

SOMPA (Mercer, 1979) ¢ Combines 3 kinds of evaluation: ¢ Medical l ¢ Social l ¢ Health, vision, hearing, etc. Entire WISC-R Pluralistic l Compare WISC-R scores to those of same community

SOMPA (Mercer, 1979) ¢ Estimated Learning Potentials: WISC-R scores adjusted for socio-economic background ¢ ¢ But these ELPs don’t predict school performance as well as the original WISC-R scores Mercer: ELPs are intended to assess who should be in EMR classes

SOMPA (Mercer, 1979) ¢ A major problem, in my view, is that we don’t know what consequences arise for children who are removed from EMR classes on basis of ELPs ¢ ¢ Is what we call these children important? It is if the label has an effect, but data do not show that effect SOMPA used much less today than it used to be

Models of test Bias ¢ ¢ Regression Constant Ratio Cole/Darlington Quota

Regression ¢ Basis – unqualified individualism: l l Treat each person as an individual, not as a member of a group Select people with highest scores for job or college place ¢ ¢ Ignores sex, race, other group characteristics Leads to highest average performance on criterion

Constant Ratio ¢ Basis – choose so that selection ratio for groups = success ratio for groups ¢ Select the best candidate but give a boost to minority group members’ scores so that selection probability = success probability

Constant Ratio ¢ Adjust test scores for minority groups upwards by half the mean difference between groups ¢ Leads to somewhat lower average performance on criterion

Cole/Darlington ¢ Basis – If there is special value in selecting minority group members, then a minority score of Y on criterion is equal to a majority score of Y + k on criterion ¢ ¢ Separate regression equations used for different groups and adjustment made Leads to lower average performance on criterion

Cole/Darlington ¢ If a value is placed on selection of minority group members, and intercept is lower for that group, then we consider minority test score X 1 and majority test score X 2 equal k X 1 X 2

Quota ¢ ¢ Basis – idea that all groups should have equal outcomes Selection based on different regression equations for each group ¢ Produces lower average performance on criterion

Quota ¢ If 10% of population is Asian then 10% of student body should be Asian ¢ ¢ Another way to look at this: if 10% of population is Jewish then no more than 10% of professors should be Jewish. This puts the quota idea in a different light.