Computer Science and Decision Making Fred Roberts, Rutgers University 1
Computer Science and Decision Making • Many recent applications in CS involve issues/problems of long interest in decision theory: üsocial choice or consensus üpreference, utility üconflict and cooperation üallocation üincentives ümeasurement • Methods developed in decision theory, often in the social sciences, beginning to be used in CS 2
CS and DM • CS applications place great strain on SS-DM methods üSheer size of problems addressed üComputational power of agents an issue üLimitations on information possessed by players üSequential nature of repeated applications • Thus: Need for new generation of SS-DM methods • Also: These new methods will provide powerful 3 tools to social scientists/decision makers.
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 4
Consensus Rankings: Social Choice • Relevant social science problems: voting, group decision making • Goal: based on everyone’s opinions, reach a “consensus” • Typical opinions expressed as: ü “first choice” ü ranking of all alternatives ü scores ü classifications • Long history of research on such problems. 5
Consensus Rankings Background: Arrow’s Impossibility Theorem: • There is no “consensus method” that satisfies certain reasonable axioms about how societies should reach decisions. • Input to Arrow’s Theorem: rankings of alternatives (ties allowed). • Output: consensus ranking. Kenneth Arrow Nobel prize winner 6
Consensus Rankings • There are widely studied and widely used consensus methods (that violate one or more of Arrow’s conditions). • One well-known consensus method: “Kemeny-Snell medians”: Given set of rankings, find ranking minimizing sum of distances to other rankings. John Kemeny, pioneer in time sharing in CS • Kemeny-Snell medians are having surprising new applications in CS. 7
Consensus Rankings • Kemeny-Snell distance between rankings: twice the number of pairs of candidates i and j for which i is ranked above j in one ranking and below j in the other + the number of pairs that are ranked in one ranking and tied in another. a b x y-z y x z On {x, y}: +2 On {x, z}: +2 On {y, z}: +1 d(a, b) = 5. 8
Consensus Rankings • Kemeny-Snell median: Given rankings a 1, a 2, …, ap, find a ranking x so that d(a 1, x) + d(a 2, x) + … + d(ap, x) is minimized. • x can be a ranking other than a 1, a 2, …, ap. • Sometimes just called Kemeny median. 9
Consensus Rankings a 1 Fish Chicken Beef a 2 Fish Chicken Beef a 3 Chicken Fish Beef • Median = a 1. • If x = a 1: d(a 1, x) + d(a 2, x) + d(a 3, x) = 0 + 2 = 2 is minimized. • If x = a 3, the sum is 4. • For any other x, the sum is at least 1 + 1 = 3. 10
Consensus Rankings a 1 Fish Chicken Beef a 2 Chicken Beef Fish a 3 Beef Fish Chicken • Three medians = a 1, a 2, a 3. • This is the “voter’s paradox” situation. 11
Consensus Rankings a 1 Fish Chicken Beef a 2 Chicken Beef Fish a 3 Beef Fish Chicken • Note that sometimes we wish to minimize d(a 1, x)2 + d(a 2, x)2 + … + d(ap, x)2 • A ranking x that minimizes this is called a Kemeny-Snell mean. • In this example, there is one mean: the ranking declaring all three alternatives tied. 12
Consensus Rankings a 1 Fish Chicken Beef a 2 Chicken Beef Fish a 3 Beef Fish Chicken • If x is the ranking declaring Fish, Chicken and Beef tied, then d(a 1, x)2 + d(a 2, x)2 + … + d(ap, x)2 = 32 + 32 = 27. • Not hard to show this is minimum. 13
Consensus Rankings Theorem (Bartholdi, Tovey, and Trick, 1989; Wakabayashi, 1986): Computing the Kemeny median of a set of rankings is an NP-complete problem. 14
Consensus Rankings Okay, so what does this have to do with practical computer science questions? 15
Consensus Rankings I mean really practical computer science questions. 16
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 17
Meta-search and Collaborative Filtering Meta-search • A consensus problem • Combine page rankings from several search engines • Dwork, Kumar, Naor, Sivakumar (2000): Kemeny-Snell medians good in spam resistance in meta-search (spam by a page if it causes metasearch to rank it too highly) • Approximation methods make this 18 computationally tractable
Meta-search and Collaborative Filtering • Recommending books or movies • Combine book or movie ratings • Produce ordered list of books or movies to recommend • Freund, Iyer, Schapire, Singer (2003): “Boosting” algorithm for combining rankings. • Related topic: Recommender Systems 19
Meta-search and Collaborative Filtering A major difference from SS-DM applications: • In SS-DM applications, number of voters is large, number of candidates is small. • In CS applications, number of voters (search engines) is small, number of candidates (pages) is large. • This makes for major new complications and research challenges. 20
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 21 10. Sequential DM: Port of Entry Inspections
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • Successful group decision making requires efficient elicitation of information and efficient representation of the information elicited. • Old problems in the social sciences. • Computational aspects becoming a focal point because of need to deal with massive and complex information. 22
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • Example I: Preferences are key components in decision making applications. • “I prefer beef to fish” • Extracting and representing preferences is key in group decision making applications. 23
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • “Brute force” approach: For every pair of alternatives, ask which is preferred to the other. • Often computationally infeasible. 24
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • In many applications (e. g. , collaborative filtering), important to elicit preferences automatically. • CP-nets introduced as tool to represent preferences succinctly and provide ways to make inferences about preferences (Boutilier, Brafman, Doomshlak, Hoos, Poole 2004). 25
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • Example II: combinatorial auctions. • Auctions increasingly used in business and government. • Information technology allows complex auctions with huge number of bidders. • There are key decision making problems arising in auctions. 26
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • Bidding functions maximizing expected profit can be exceedingly difficult to compute. • Determining the winner of an auction can be extremely hard. (Rothkopf, Pekec, Harstad 1998) 27
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation Combinatorial Auctions • • Multiple goods auctioned off. Submit bids for combinations of goods. This leads to NP-complete allocation problems. Might not even be able to feasibly express all possible preferences for all subsets of goods. • Rothkopf, Pekec, Harstad (1998): determining winner is computationally tractable for many economically interesting kinds of combinations. 28
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation Combinatorial Auctions • Decision maker needs to elicit preferences from all agents for all plausible combinations of items in the auction. • Similar problem arises in optimal bundling of goods and services. • Elicitation requires exponentially many queries in general. 29
Computational Approaches to Information Management in Group Decision Making Representation and Elicitation • Challenge: Recognize situations in which efficient elicitation and representation is possible. • One result: Fishburn, Pekec, Reeds (2002) • Even more complicated: When objects in auction have complex structure. • Problem arises in: ü Legal reasoning, sequential decision making, automatic decision devices, collaborative filtering. 30
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 31
Large Databases and Inference • • Decision makers consult massive data sets. Real data often in form of sequences. Example: Bioinformatics. Gen. Bank has over 7 million sequences comprising 8. 6 billion bases. • The search for similarity or patterns has extended from pairs of sequences to finding patterns that appear in common in a large number of sequences or throughout the database: “consensus sequences” • Emerging field of “Bioconsensus”: applies SS 32 DM consensus methods to biological databases.
Large Databases and Inference Why look for such patterns? Similarities between sequences or parts of sequences lead to the discovery of shared phenomena. For example, it was discovered that the sequence for platelet derived factor, which causes growth in the body, is 87% identical to the sequence for v-sis, a cancer-causing gene. This led to the discovery that v-sis works by stimulating growth. 33
Large Databases and Inference Example Bacterial Promoter Sequences studied by Waterman (1989): RRNABP 1: TNAA: UVRBP 2: SFC: ACTCCCTATAATGCGCCA GAGTGTAATAATGTAGCC TTATCCAGTATAATTTGT AAGCGGTGTTATAATGCC Notice that if we are looking for patterns of length 4, each sequence has the pattern TAAT. 34
Large Databases and Inference Example Bacterial Promoter Sequences studied by Waterman (1989): RRNABP 1: TNAA: UVRBP 2: SFC: ACTCCCTATAATGCGCCA GAGTGTAATAATGTAGCC TTATCCAGTATAATTTGT AAGCGGTGTTATAATGCC Notice that if we are looking for patterns of length 4, each sequence has the pattern TAAT. 35
Large Databases and Inference Example However, suppose that we add another sequence: M 1 RNA: AACCCTCTATACTGCGCG The pattern TAAT does not appear here. However, it almost appears, since the pattern TACT appears, and this has only one mismatch from the pattern TAAT. 36
Large Databases and Inference Example However, suppose that we add another sequence: M 1 RNA: AACCCTCTATACTGCGCG The pattern TAAT does not appear here. However, it almost appears, since the pattern TACT appears, and this has only one mismatch from the pattern TAAT. So, in some sense, the pattern TAAT is a good consensus pattern. 37
Large Databases and Inference Example We make this precise using best mismatch distance. Consider two sequences a and b with b longer than a. Then d(a, b) is the smallest number of mismatches in all possible alignments of a as a consecutive subsequence of b. 38
Large Databases and Inference Example a = 0011, b = 111010 Possible Alignments: 111010 0011 The best-mismatch distance is 2, which is achieved in the third alignment. 39
Large Databases and Inference Smith-Waterman • Let be a finite alphabet of size at least 2 and be a finite collection of words of length L on . • Let F( ) be the set of words of length k 2 that are our consensus patterns. (Assume L k. ) • Let = {a 1, a 2, …, an}. • One way to define F( ) is as follows. • Let d(a, b) be the best-mismatch distance. • Consider nonnegative parameters sd that are monotone decreasing with d and let F(a 1, a 2, …, an) be all those words w of length k that maximize S (w) = isd(w, ai) 40
Large Databases and Inference • We call such an F a Smith-Waterman consensus. • In particular, Waterman and others use the parameters sd = (k-d)/k. Example: • An alphabet used frequently is the purine/pyrimidine alphabet {R, Y}, where R = A (adenine) or G (guanine) and Y = C (cytosine) or T (thymine). • For simplicity, it is easier to use the digits 0, 1 rather than the letters R, Y. • Thus, let = {0, 1}, let k = 2. Then the possible pattern words are 00, 01, 10, 11. 41
Large Databases and Inference • Suppose a 1 = 111010, a 2 = 111111. How do we find F(a 1, a 2)? • We have: d(00, a 1) = 1, d(00, a 2) = 2 d(01, a 1) = 0, d(01, a 2) = 1 d(10, a 1) = 0, d(10, a 2) = 1 d(11, a 1) = 0, d(11, a 2) = 0 S (00) = sd(00, ai) = s 1 + s 2, S (01) = sd(01, ai) = s 0 + s 1 S (10) = sd(10, ai) = s 0 + s 1 S (11) = sd(11, ai) = s 0 + s 0 • As long as s 0 > s 1 > s 2, it follows that 11 is the consensus 42 pattern, according to Smith-Waterman’s consensus.
Example: • Let ={0, 1}, k = 3, and consider F(a 1, a 2, a 3) where a 1 = 000000, a 2 = 100000, a 3 = 111110. The possible pattern words are: 000, 001, 010, 011, 100, 101, 110, 111. d(000, a 1) = 0, d(000, a 2) = 0, d(000, a 3) = 2, d(001, a 1) = 1, d(001, a 2) = 1, d(001, a 3) = 2, d(100, a 1) = 1, d(100, a 2) = 0, d(100, a 3) = 1, etc. S (000) = s 2 + 2 s 0, S (001) = s 2 + 2 s 1, S (100) = 2 s 1 + s 0, etc. • Now, s 0 > s 1 > s 2 implies that S (000) > S (001). • Similarly, one shows that the score is maximized by S (000) or S (100). • Monotonicity doesn’t say which of these is highest. 43
Large Databases and Inference The Special Case sd = (k-d)/k • Then it is easy to show that the words w that maximize s (w) are exactly the words w that minimize id(w, ai). • In other words: In this case, the Smith-Waterman consensus is exactly the median. Algorithms for computing consensus sequences such as Smith-Waterman are important in modern molecular biology. 44
Large Databases and Inference Other Topics in “Bioconsensus” • Alternative phylogenies (evolutionary trees) are produced using different methods and we need to choose a consensus tree. • Alternative taxonomies (classifications) are produced using different models and we need to choose a consensus taxonomy. • Alternative molecular sequences are produced using different criteria or different algorithms and we need to choose a consensus sequence. • Alternative sequence alignments are produced and we need to choose a consensus alignment. 45
Large Databases and Inference Other Topics in “Bioconsensus” • Several recent books on bioconsensus. • Day and Mc. Morris [2003] • Janowitz, et al. [2003] • Bibliography compiled by Bill Day: In molecular biology alone, hundreds of papers using consensus methods in biology. • Large database problems in CS are being approached using methods of “bioconsensus” having their origin in SS-DM. 46
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 47
Consensus Computing, Image Processing • Old SS-DM problem: Dynamic modeling of how individuals change opinions over time, eventually reaching consensus. • Often use dynamic models on graphs • Related to neural nets. • CS applications: distributed computing. • Values of processors in a network are updated until all have same value. 48
Consensus Computing, Image Processing • • CS application: Noise removal in digital images Does a pixel level represent noise? Compare neighboring pixels. If values beyond threshold, replace pixel value with mean or median of values of neighbors. • Related application in distributed computing. • Values of faulty processors are replaced by those of neighboring non-faulty ones. • Berman and Garay (1993) use “parliamentary procedure” called cloture 49
Consensus Computing, Image Processing • Side comment: same models are being applied in “computational and mathematical epidemiology”. • Modeling the spread of disease through large social networks. Measles SARS 50
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 51
Computational Intractability of Consensus Functions • How hard is it to compute the winner of an election? • We know counting votes can be difficult and time consuming. • However: • Bartholdi, Tovey and Trick (1989): There are voting schemes where it can be computationally intractable to determine who won an election. 52
Computational Intractability of Consensus Functions • So, is computational intractability necessarily bad? • Computational intractability can be a good thing in an election: Designing voting systems where it is computationally intractable to “manipulate” the outcome of an election by “insincere voting”: ü Adding voters ü Declaring voters ineligible ü Adding candidates ü Declaring candidates ineligible 53
Computational Intractability of Consensus Functions • Given a set A of all possible candidates and a set I of all possible voters. • Suppose we know voter i’s ranking of all candidates in A, for every voter i. • Given a subset of I of eligible voters, a particular candidate a in A, and a number k, is there a set of at most k ineligible voters who can be declared eligible so that candidate a is the winner? • Bartholdi, Tovey, Trick (1989): For some consensus functions (voting rules), this is an NP 54 complete problem.
Computational Intractability of Consensus Functions • Given a set A of all possible candidates and a set I of all possible voters. • Suppose we know voter i’s ranking of all candidates in A, for every voter i. • Given a subset of I of eligible voters, a particular candidate a in A, and a number k, is there a set of at most k eligible voters who can be declared ineligible so that candidate a is the winner? • Bartholdi, Tovey, Trick (1989): For some consensus functions (voting rules), this is an NP 55 complete problem.
Computational Intractability of Consensus Functions • Software agents may be more likely to manipulate than individuals (Conitzer and Sandholm 2002): ü Humans don’t think about manipulating ü Computation can be tedious. ü Software agents are good at running algorithms ü Software agents only need to have code for manipulation written once. ü All the more reason to develop computational barriers to manipulation. 56
Computational Intractability of Consensus Functions • Stopping those software agents: 57
Computational Intractability of Consensus Functions • Conitzer and Sandholm (2002): ü Earlier results of difficulty of manipulation depend on large number of candidates ü New results: manipulation possible with some voting methods if smaller number (bounded number) of candidates) ü In weighted voting, voters may have different numbers of votes (as in US presidential elections, where different states (= voters) have different numbers of votes). Here, manipulation is harder. ü Manipulation difficult when uncertainty about 58 others’ votes.
Computational Intractability of Consensus Functions • Conitzer and Sandholm (2006): ü Try to find voting rules for which manipulation is usually hard. ü Why is this difficult to do? ü One explanation: under one reasonable assumption, it is impossible to find such rules. 59
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 60
Aside: Electronic Voting • Issues: ü Correctness ü Anonymity ü Availability ü Security ü Privacy 61
Electronic Voting Security Risks in Electronic Voting • Threat of “denial of service attacks” • Threat of penetration attacks involving a delivery mechanism to transport a malicious payload to target host (thru Trojan horse or remote control program) • Private and correct counting of votes • Cryptographic challenges to keep votes private • Relevance of work on secure multiparty computation 62
Electronic Voting Other CS Challenges: • Resistance to “vote buying” • Development of user-friendly interfaces • Vulnerabilities of communication path between the voting client (where you vote) and the server (where votes are counted) • Reliability issues: random hardware and software failures 63
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 64
Software & Hardware Measurement • Theory of measurement developed by mathematical social scientists. • Measurement theory studies ways to combine scores obtained on different criteria. • A statement involving scales of measurement is considered meaningful if its truth or falsity is unchanged under acceptable transformations of all scales involved. • Example: It is meaningful to say that I weigh more than my daughter. • That is because if it is true in kilograms, then it is 65 also true in pounds, in grams, etc.
Software & Hardware Measurement • Measurement theory has studied what statements you can make after averaging scores. • Think of averaging as a consensus (DM) method. • One general principle: To say that the average score of one set of tests is greater than the average score of another set of tests is not meaningful (it is meaningless) under certain conditions. • This is often the case if the averaging procedure is to take the arithmetic mean: If s(xi) is score of xi, i = 1, 2, …, n, then arithmetic mean is is(xi)/n. • Long literature on what averaging methods lead to meaningful conclusions. 66
Software & Hardware Measurement A widely used method in hardware measurement: ü Score a computer system on different benchmarks. ü Normalize score relative to performance of one base system ü Average normalized scores ü Pick system with highest average. ü Fleming and Wallace (1986): Outcome can depend on choice of base system. ü Meaningless in sense of measurement theory ü Leads to theory of merging normalized scores 67
Software & Hardware Measurement P R R O C M E S S O Z R E 417 BENCHMARK F G H 83 66 39, 449 I 772 244 70 153 33, 527 368 134 70 135 66, 000 369 Data from Heath, Comput. Archit. News (1984) 68
Software & Hardware Measurement Normalize Relative to Processor R P R R O C M E S S O Z R E 417 1. 00 BENCHMARK F G H I 83 66 39, 449 772 1. 00 244. 59 70. 84 153 2. 32 33, 527. 85 368. 48 134. 32 70. 85 135 2. 05 66, 000 1. 67 369. 45 69
Software & Hardware Measurement Take Arithmetic Mean of Normalized Scores P R R O C M E S S O Z R E 417 1. 00 Arithmetic BENCHMARK Mean F G H I 83 66 39, 449 772 1. 00 244. 59 70. 84 153 2. 32 33, 527. 85 368. 48 1. 01 134. 32 70. 85 135 2. 05 66, 000 1. 67 369. 45 1. 07 70
Software & Hardware Measurement Take Arithmetic Mean of Normalized Scores P R R O C M E S S O Z R E 417 1. 00 Arithmetic BENCHMARK Mean F G H I 83 66 39, 449 772 1. 00 244. 59 70. 84 153 2. 32 33, 527. 85 368. 48 1. 01 134. 32 70. 85 135 2. 05 66, 000 1. 67 369. 45 1. 07 Conclude that machine Z is best 71
Software & Hardware Measurement Now Normalize Relative to Processor M P R R O C M E S S O Z R E 417 1. 71 BENCHMARK F G H I 83 66 39, 449 772 1. 19. 43 1. 18 2. 10 244 1. 00 70 1. 00 153 1. 00 33, 527 368 1. 00 134. 55 70 1. 00 135. 88 66, 000 369 1. 97 1. 00 72
Software & Hardware Measurement Take Arithmetic Mean of Normalized Scores P R R O C M E S S O Z R E 417 1. 71 Arithmetic BENCHMARK Mean F G H I 83 66 39, 449 772 1. 32 1. 19. 43 1. 18 2. 10 244 1. 00 70 1. 00 153 1. 00 33, 527 368 1. 00 134. 55 70 1. 00 135. 88 66, 000 369 1. 97 1. 00 1. 08 73
Software & Hardware Measurement Take Arithmetic Mean of Normalized Scores P R R O C M E S S O Z R E 417 1. 71 Arithmetic BENCHMARK Mean F G H I 83 66 39, 449 772 1. 32 1. 19. 43 1. 18 2. 10 244 1. 00 70 1. 00 153 1. 00 33, 527 368 1. 00 134. 55 70 1. 00 135. 88 66, 000 369 1. 97 1. 00 1. 08 Conclude that machine R is best 74
Software and Hardware Measurement • So, the conclusion that a given machine is best by taking arithmetic mean of normalized scores is meaningless in this case. • Above example from Fleming and Wallace (1986), data from Heath (1984) • Sometimes, geometric mean is helpful. • Geometric mean is n is(xi) 75
Software & Hardware Measurement Normalize Relative to Processor R P R R O C M E S S O Z R E 417 1. 00 Geometric BENCHMARK Mean F G H I 83 66 39, 449 772 1. 00 244. 59 70. 84 153 2. 32 33, 527. 85 368. 48 . 86 134. 32 70. 85 135 2. 05 66, 000 1. 67 369. 45 . 84 Conclude that machine R is best 76
Software & Hardware Measurement Now Normalize Relative to Processor M P R R O C M E S S O Z R E 417 1. 71 Geometric BENCHMARK Mean F G H I 83 66 39, 449 772 1. 17 1. 19. 43 1. 18 2. 10 244 1. 00 70 1. 00 153 1. 00 33, 527 368 1. 00 134. 55 70 1. 00 135. 88 66, 000 369 1. 97 1. 00 . 99 Still conclude that machine R is best 77
Software and Hardware Measurement • In this situation, it is easy to show that the conclusion that a given machine has highest geometric mean normalized score is a meaningful conclusion. • Even meaningful: A given machine has geometric mean normalized score 20% higher than another machine. • Fleming and Wallace give general conditions under which comparing geometric means of normalized scores is meaningful. • Research area: what averaging procedures make sense in what situations? Large literature. • Note: There are situations where comparing arithmetic means is meaningful but comparing geometric means is 78 not.
Technical Aside: How Should We Average Scores? Unknown function u = F(a 1, a 2, …, an) The values a 1, a 2, …, an are scores and F is some averaging function R. Duncan Luce's idea (“Principle of Theory Construction”): If you know the scale types of the ai and the scale type of u and you assume that an admissible transformation of each of the ai leads to an admissible transformation of u, you can derive the form of F. Admissible transformation: pounds into grams, Fahrenheit into Centigrade. Scale type determined by class of admissible transformations. 79 Ratio scales, interval scales, …
How Should We Average Scores? Example: a 1, a 2, …, an are independent ratio scales, u is a ratio scale. F: ( +)n + F(a 1, a 2, …, an) = u F( 1 a 1, 2 a 2, …, nan) = u, 1 > 0, 2 > 0, n > 0, depends on a 1, a 2, …, an. • Thus we get the functional equation: (*) F( 1 a 1, 2 a 2, …, nan) = A( 1, 2, …, n)F(a 1, a 2, …, an), A( 1, 2, …, n) > 0 80
How Should We Average Scores? (*) F( 1 a 1, 2 a 2, …, nan) = A( 1, 2, …, n)F(a 1, a 2, …, an), A( 1, 2, …, n) > 0 Theorem (Luce 1964): If F: ( +)n + is continuous and c c F (a 1; a 2; : : : ; an ) = ¸ a 11 a 22 : : : acn : satisfies (*), then there are > 0, c 1, c 2, …, cnn so that 81
How Should We Average Scores? (Aczél, Roberts, Rosenbaum 1986): It is easy to see that the assumption of continuity can be weakened to continuity at a point, monotonicity, or boundedness on an (arbitrarily small, open) n-dimensional interval or on a set of positive measure. Call any of these assumptions regularity. Janos Aczél “Mr Functional Equations” 82
How Should We Average Scores? Theorem (Aczél and Roberts 1989): If in addition F satisfies reflexivity and symmetry, then = 1 and c 1 = c 2 = … = cn = 1/n , so F is the geometric mean. Reflexivity: F(a, a, . . . , a) = a Symmetry: F(a 1, a 2, …, an) = F(a (1), a (2), …, a (n)) for all permutations of {1, 2, …, n} 83
How Should We Average Scores? Sometimes You Get the Arithmetic Mean Example: a 1, a 2, …, an interval scales with the same unit and independent zero points; u an interval scale. Functional Equation: (**) F( a 1+ 1, a 2+ 2, …, an+ n) = A( , 1, 2, …, n)F(a 1, a 2, …, an) + B( , 1, 2, …, n) A( , 1, 2, …, n) > 0 84
How Should We Average Scores? Functional Equation: (**) F( a 1+ 1, a 2+ 2, …, an+ n) = A( , 1, 2, …, n)F(a 1, a 2, …, an) + B( , 1, 2, …, n) A( , 1, 2, …, n) > 0 Solutions to (**) (Aczél, Roberts, and Rosenbaum 1986): Xn F (a ; : : : ; a ) = ¸ a +b 1 2 i i n i= 1 1, 2, …, n, b arbitrary constants (no continuity assumptions needed) 85
How Should We Average Scores? Theorem (Aczél and Roberts 1989): (1). If in addition F satisfies reflexivity, then Pn ¸ i = 1, b = 0: = i 1 (2). If in addition F satisfies reflexivity and symmetry, then i= 1/n for all i, and b = 0, i. e. , F is the arithmetic mean. 86
Software and Hardware Measurement • Message from measurement theory to computer science (and DM): Do not perform arithmetic operations on data without paying attention to whether the conclusions you get are meaningful. 87
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 88
Power of a Voter Shapley-Shubik Power Index • Think of a “voting game” • Shapley-Shubik index measures the power of each player in a multi-player game. • Consider a game in which some coalitions of players win and some lose, with no subset of a losing coalition winning. Martin Shubik Lloyd Shapley 89
Power of a Voter Shapley-Shubik Power Index • Consider a coalition forming at random, one player at a time. • A player i is pivotal if addition of i throws coalition from losing to winning. • Shapley-Shubik index of i = probability i is pivotal if an order of players is chosen at random. • Power measure applying to more general games than voting games is called Shapley Value. 90
Power of a Voter Example: Shareholders of Company Shareholder 1 holds 3 shares. Shareholders 2, 3, 4, 5, 6, 7 hold 1 share each. A majority of shares are needed to make a decision. Coalition {1, 4, 6} is winning. Coalition {2, 3, 4, 5, 6} is winning. Shareholder 1 is pivotal if he is 3 rd, 4 th, or 5 th. So shareholder 1’s Shapley value is 3/7. Sum of Shapley values is 1 (since they are probabilities) Thus, each other shareholder has Shapley value (4/7)/6 = 2/21 91
Power of a Voter Example: United Nations Security Council • 15 member nations • 5 permanent members üChina, France, Russia, UK, US • 10 non-permanent • Permanent members have veto power • Coalition wins iff it has all 5 permanent members and at least 4 of the 10 non-permanent members. 92
Power of a Voter Example: United Nations Security Council • What is the power of each Member of the Security Council? • Fix non-permanent member i. • i is pivotal in permutations in which all permanent members precede i and exactly 3 nonpermanent members do. • How many such permutations are there? 93
Power of a Voter Example: United Nations Security Council • Choose 3 non-permanent members preceding i. • Order all 8 members preceding i. • Order remaining 6 non-permanent members. • Thus the number of such permutations is: C(9, 3) x 8! x 6! = 9!/3!6! x 8! x 6! = 9!8!/3! • The probability that i is pivotal = power of nonpermanent member = 9!8!/3!15! =. 001865 • The power of a permanent member = [1 – 10 x. 001865]/5 =. 1963. • Permanent members have 100 times power of 94 non-permanent members.
Power of a Voter • There a variety of other power indices used in game theory and political science (Banzhaf index, Coleman index, …) • Need calculate them for huge games • Mostly computationally intractable 95
Power of a Voter: Allocation/Sharing Costs and Revenues • Shapley-Shubik power index and more general Shapley value have been used to allocate costs to different users in shared projects. ü Allocating runway fees in airports ü Allocating highway fees to trucks of different sizes ü Universities sharing library facilities ü Fair allocation of telephone calling charges among users sharing complex phone systems (Cornell’s experiment) 96
Power of a Voter: Allocating/Sharing Costs and Revenues Allocating Runway Fees at Airports • Some planes require longer runways. Charge them more for landings. How much more? • Divide runways into meter-long segments. • Each month, we know how many landings a plane has made. • Given a runway of length y meters, consider a game in which the players are landings and a coalition “wins” if the runway is not long enough for planes in the coalition. 97
Power of a Voter: Allocating/Sharing Costs and Revenues Allocating Runway Fees at Airports • A landing is pivotal if it is the first landing added that makes a coalition require a longer runway. • The Shapley value gives the cost of the yth meter of runway allocated to a given landing. • We then add up these costs over all runway lengths a plane requires and all landings it makes. 98
Power of a Voter: Allocating/Sharing Costs and Revenues Multicasting • Applications in multicasting. • Unicast routing: Each packet sent from a source is delivered to a single receiver. • Sending it to multiple sites: Send multiple copies and waste bandwidth. • In multicast routing: Use a directed tree connecting source to all receivers. • At branch points, a packet is duplicated as necessary. 99
Multicasting 100
Power of a Voter: Allocating/Sharing Costs and Revenues Multicasting • Multicast routing: Use a directed tree connecting source to all receivers. • At branch points, a packet is duplicated as necessary. • Bandwidth is not directly attributable to a single receiver. • How to distribute costs among receivers? • One idea: Use Shapley value. 101
Allocating/Sharing Costs and Revenues • Feigenbaum, Papadimitriou, Shenker (2001): no feasible implementation for Shapley value in multicasting. • Note: Shapley value is uniquely characterized by four simple axioms. • Sometimes we state axioms as general principles we want a solution concept to have. • Jain and Vazirani (1998): polynomial time computable cost-sharing algorithm ü Satisfying some important axioms ü Calculating cost of optimum multicast tree within factor of two of optimal. 102
CS and DM: Outline 1. Consensus Rankings 2. Meta-search and Collaborative Filtering 3. Computational Approaches to Information Management in Group Decision Making 4. Large Databases and Inference 5. Consensus Computing, Image Processing 6. Computational Intractability of Consensus Functions 7. Electronic Voting 8. Software and Hardware Measurement 9. Power of a Voter 10. Sequential DM: Port of Entry Inspections 103
Algorithms for Port of Entry Inspection for WMDs Joint work with Saket Anand, David Madigan, Richard Mammone, Saumitr Pathak, Philip Stroud 104
Port of Entry Inspection Algorithms • Goal: Find ways to intercept illicit nuclear materials and weapons destined for the U. S. via the maritime transportation system • Currently inspecting only small % of containers arriving at ports • Even inspecting 8% of containers in Port of NY/NJ might bring international trade to a halt (Larrabbee 2002) 105
Port of Entry Inspection Algorithms • Aim: Develop decision support algorithms that will help us to “optimally” intercept illicit materials and weapons subject to limits on delays, manpower, and equipment • Find inspection schemes that minimize total “cost” including “cost” of false positives and false negatives Mobile Vacis: truckmounted gamma ray imaging system 106
Sequential Decision Making Problem • Stream of containers arrives at a port • The Decision Maker’s Problem: • Which to inspect? • Which inspections next based on previous results? • Approach: –“decision logics” –combinatorial optimization methods –Builds on ideas of Stroud and Saeger at Los Alamos National Laboratory –Need for new models and methods 107
Sequential Diagnosis Problem • Such sequential diagnosis problems arise in many areas: –Communication networks (testing connectivity, paging cellular customers, sequencing tasks, …) –Manufacturing (testing machines, fault diagnosis, routing customer service calls, …) –Artificial intelligence/CS (optimal derivation strategies in knowledge bases, best-value satisficing search, coding decision trees, …) –Medicine (diagnosing patients, sequencing treatments, …) 108
Sequential Decision Making Problem • Containers arriving to be classified into categories. • Simple case: 0 = “ok”, 1 = “suspicious” • Inspection scheme: specifies which inspections are to be made based on previous observations 109
Sequential Decision Making Problem • Containers have attributes, each in a number of states • Sample attributes: –Levels of certain kinds of chemicals or biological materials –Whether or not there are items of a certain kind in the cargo list –Whether cargo was picked up in a certain port 110
Sequential Decision Making Problem • Currently used attributes: –Does ship’s manifest set off an “alarm”? –What is the neutron or Gamma emission count? Is it above threshold? –Does a radiograph image come up positive? –Does an induced fission test come up positive? Gamma ray detector 111
Sequential Decision Making Problem • We can imagine many other attributes • Concern with general algorithmic approaches. • Seek a methodology not tied to today’s technology. • Detectors are evolving quickly. 112
Sequential Decision Making Problem • Simplest Case: Attributes are in state 0 or 1 • Then: Container is a binary string like 011001 • So: Classification is a decision function F that assigns each binary string to a category. 011001 F(011001) If attributes 2, 3, and 6 are present and others are not, assign container to category F(011001). 113
Sequential Decision Making Problem • If there are two categories, 0 and 1, decision function F is a boolean function. Example: F(000) = F(111) = 1, F(abc) = 0 otherwise This classifies a container as positive iff it has none of the attributes or all of them. 1= 114
Sequential Decision Making Problem • Given a container, test its attributes until know enough to calculate the value of F. • An inspection scheme tells us in which order to test the attributes to minimize cost. • Even this simplified problem is hard computationally. 115
Sequential Decision Making Problem • This assumes F is known. • Simplifying assumption: Attributes are independent. • At any point we can stop inspecting and output the value of F based on outcomes of inspections so far. • Complications: May be precedence relations in the components (e. g. , can’t test attribute a 4 before testing a 6. • Or: cost may depend on attributes tested before. • F may depend on variables that cannot be directly 116 tested or for which tests are too costly.
Sequential Decision Making Problem • Such problems are hard computationally. • There are many possible boolean functions F. • Even if F is fixed, problem of finding a good classification scheme (to be defined precisely below) is NP-complete. • Several classes of functions F allow for efficient inspection schemes: –k-out-of-n systems –Certain series-parallel systems –Read-once systems –“regular” systems 117 –Horn systems
Sensors and Inspection Lanes • n types of sensors measure presence or absence of the n attributes. • Many copies of each sensor. • Complication: different characteristics of sensors. • Entities come for inspection. • Which sensor of a given type to use? • Think of inspection lanes and queues. • Besides efficient inspection schemes, could decrease costs by: –Buying more sensors –Change allocation of containers to sensor lanes. 118
Binary Decision Tree Approach • Sensors measure presence/absence of attributes. • Binary Decision Tree (BDT): –Nodes are sensors or categories (0 or 1) –Two arcs exit from each sensor node, labeled left and right. –Take the right arc when sensor says the attribute is present, left arc otherwise 119
Binary Decision Tree Approach • Reach category 1 from the root only through the path a 0 to a 1 to 1. • Container is classified in category 1 iff it has both attributes a 0 and a 1. • Corresponding boolean function F(11) = 1, F(10) = F(01) = F(00) = 0. Figure 1 120
Binary Decision Tree Approach • Reach category 1 from the root by: a 0 L to a 1 R a 2 R 1 or a 0 R a 2 R 1 • Container classified in category 1 iff it has a 1 and a 2 and not a 0 or a 0 and a 2 and possibly a 1. • Corresponding boolean function F(111) = F(101) = F(011) = 1, F(abc) = 0 otherwise. Figure 2 121
Binary Decision Tree Approach • This binary decision tree corresponds to the same boolean function F(111) = F(101) = F(011) = 1, F(abc) = 0 otherwise. However, it has one less observation node ai. So, it is more efficient if all observations are equally costly and equally likely. Figure 3 122
Binary Decision Tree Approach • Even if the boolean function F is fixed, the problem of finding the “optimal” binary decision tree for it is very hard (NP-complete). • For small n = number of attributes, can try to solve it by brute force enumeration. Port of Long Beach • Even for n = 4, not practical. • (n = 4 at Port of Long Beach-Los Angeles) 123
Binary Decision Tree Approach Promising Approaches: • Heuristic algorithms, approximations to optimal. • Special assumptions about the boolean function F. • For “monotone” boolean functions, integer programming formulations give promising heuristics. • Stroud and Saeger enumerate all “complete, ” monotone boolean functions and calculate the least expensive corresponding binary decision trees. • Their method practical for n up to 4, not n = 5. 124
Binary Decision Tree Approach Monotone Boolean Functions: • Given two strings x 1 x 2…xn, y 1 y 2…yn • Suppose that xi yi for all i implies that F(x 1 x 2…xn) F(y 1, y 2…yn). • Then we say that F is monotone. • Then 11… 1 has highest probability of being in category 1. 125
Binary Decision Tree Approach Incomplete Boolean Functions: • Boolean function F is incomplete if F can be calculated by finding at most n-1 attributes and knowing the value of the input string on those attributes • Example: F(111) = F(110) = F(101) = F(100) = 1, F(000) = F(001) = F(010) = F(011) = 0. • F(abc) is determined without knowing b (or c). • F is incomplete. 126
Binary Decision Tree Approach Complete, Monotone Boolean Functions: • Stroud and Saeger: “brute force” algorithm for enumerating binary decision trees implementing complete, monotone boolean functions and choosing least cost BDT. • Feasible to implement up to n = 4. • n = 2: –There are 6 monotone boolean functions. –Only 2 of them are complete, monotone –There are 4 binary decision trees for calculating these 2 complete, monotone boolean 127 functions.
Binary Decision Tree Approach Complete, Monotone Boolean Functions: • n = 3: – 9 complete, monotone boolean functions. – 60 distinct binary trees for calculating them • n = 4: – 114 complete, monotone boolean functions. – 11, 808 distinct binary decision trees for calculating them. –(Compare 1, 079, 779, 602 BDTs for all boolean functions) 128
Binary Decision Tree Approach Complete, Monotone Boolean Functions: • n = 5: – 6894 complete, monotone boolean functions – 263, 515, 920 corresponding binary decision trees. • Combinatorial explosion! • Need alternative approaches; enumeration not feasible! • (Even worse: compare 5 x 1018 BDTs corresponding to all boolean functions) 129
Cost Functions • Stroud-Saeger method applies to more sophisticated cost models, not just cost = number of sensors in the BDT. • Using a sensor has a cost: –Unit cost of inspecting one item with it –Fixed cost of purchasing and deploying it –Delay cost from queuing up at the sensor station • Preliminary problem: disregard fixed and delay 130 costs. Minimize unit costs.
Cost Functions: Delay Costs • Tradeoff between fixed costs and delay costs: Add more sensors cuts down on delays. • Stochastic process of containers arriving • Distribution of delay times for inspections • Use queuing theory to find average delay times under different models 131
Cost Functions: Unit Costs Tree Utilization • Complication: Assume cost depends on how many nodes of BDT are actually visited during an “average” container’s inspection. (This is sum of unit costs. ) • Depends on characteristics of population of entities being inspected. • I. e. , depends on “distribution” of containers. • In our early models, we assume we are given probability of sensor errors and probability of bomb in a container. • This allows us to calculate “expected” cost of 132 utilization of the tree Cutil.
Cost Functions • Cost of false positive: Cost of additional tests. –If it means opening the container, it’s very expensive. • Cost of false negative: –Complex issue. –What is cost of a bomb going off in Manhattan? 133
Cost Functions: Sensor Errors • One Approach to False Positives/Negatives: Assume there can be Sensor Errors • Simplest model: assume that all sensors checking for attribute ai have same fixed probability of saying ai is 0 if in fact it is 1, and similarly saying it is 1 if in fact it is 0. • More sophisticated analysis later describes a model for determining probabilities of sensor errors. • Notation: X = state of nature (bomb or no bomb) Y = outcome (of sensor or entire inspection process). 134
Probability of Error for The Entire Tree State of nature is zero (X = 0), absence of a bomb State of nature is one (X = 1), presence of a bomb A A C 0 B 0 C 1 1 Probability of false positive (P(Y=1|X=0)) for this tree is given by P(Y=1|X=0) = P(YA=1|X=0) * P(YB=1|X=0) + P(YA=1|X=0) *P(YB=0|X=0)* P(YC=1|X=0) Pfalsepositive 0 1 1 Probability of false negative (P(Y=0|X=1)) for this tree is given by P(Y=0|X=1) = P(YA=0|X=1) + P(YA=1|X=1) *P(YB=0|X=1)*P(YC=0|X=1) Pfalsenegative 135
Cost Function used for Evaluating the Decision Trees. CTot = CFalse. Positive *PFalse. Positive + CFalse. Negative *PFalse. Negative + Cutil CFalse. Positive is the cost of false positive (Type I error) CFalse. Negative is the cost of false negative (Type II error) PFalse. Positive is the probability of a false positive occurring PFalse. Negative is the probability of a false negative occurring Cutil is the expected cost of utilization of the tree. 136
Stroud Saeger Experiments • Stroud-Saeger ranked all trees formed from 3 or 4 sensors A, B, C and D according to increasing tree costs. • Used cost function defined above. • Values used in their experiments: – CA =. 25; P(YA=1|X=1) =. 90; P(YA=1|X=0) =. 10; – CB = 10; P(YC=1|X=1) =. 99; P(YB=1|X=0) =. 01; – CC = 30; P(YD=1|X=1) =. 999; P(YC=1|X=0) =. 001; – CD = 1; P(YD=1|X=1) =. 95; P(YD=1|X=0) =. 05; – Here, Ci = unit cost of utilization of sensor i. • Also fixed were: CFalse. Negative, CFalse. Positive, P(X=1) 137
Stroud Saeger Experiments: Our Sensitivity Analysis • We have explored sensitivity of the Stroud-Saeger conclusions to variations in values of these three parameters. • We estimated high and low values for these parameters. n = 3 (use sensors A, B, C) • We chose one of the values from the interval of values and then explored the highest ranked tree as the other two were chosen at random in the interval of values. 10, 000 experiments for each pair of fixed values. • We looked for the variation in the top-ranked tree and how the top-rank related to choice of parameter values. • Very surprising results. 138
Conclusions from Sensitivity Analysis • Considerable lack of sensitivity to modification in parameters for trees using 3 sensors. • Very few optimal trees. • Very few boolean functions arise among optimal and near-optimal trees. • Similar results for trees using 4 sensors. 139
Stroud Saeger Experiments: Our Sensitivity Analysis – CFalse. Negative was varied between 25 million and 10 billion dollars • Low and high estimates of direct and indirect costs incurred due to a false negative. – CFalse. Positive was varied between $180 and $720 • Cost incurred due to false positive (4 men * (3 -6 hrs) * (15 – 30 $/hr) – P(X=1) was varied between 1/10, 000 and 1/100, 000 140
Frequency of Top-ranked Trees when CFalse. Negative and CFalse. Positive are Varied • • 10, 000 randomized experiments (randomly selected values of CFalse. Negative and CFalse. Positive from the specified range of values) for the median value of P(X=1). The above graph has frequency counts of the number of experiments when a particular tree was ranked first or second, or third and so on. • Only three trees (7, 55 and 1) ever came first. 6 trees came second, 10 came third, 13 came fourth. 141
Frequency of Top-ranked Trees when CFalse. Negative and P(X=1) are Varied • 10, 000 randomized experiments for the median value of CFalse. Positive. • Only 2 trees (7 and 55) ever came first. 4 trees came second. 7 trees came third. 10 and 13 trees came 4 th and 5 th respectively. 142
Frequency of Top-ranked Trees when P(X=1) and CFalse. Positive are Varied • 10, 000 randomized experiments for the median value of CFalse. Negative. • Only 3 trees (7, 55 and 1) ever came first. 6 trees came second. 143 10 th and 5 th respectively. trees came third. 13 and 16 trees came 4
Values of CFalse. Negative and CFalse. Positive when Tree 7 was Ranked First • This is a graph of CFalse. Positive against CFalse. Negative values obtained from the randomized experiments. The black dots 144 represent points at which tree 7 scored first rank.
Values of CFalse. Negative and CFalse. Positive when Tree 55 was Ranked First • Tree 55 fills up the lower area in the range of CFalse. Negative 145 and CFalse. Positive values.
Values of CFalse. Negative and CFalse. Positive when Tree 1 was Ranked First • Tree 1 fills up the upper area in the range of CFalse. Negative 146 and CFalse. Positive.
Values of CFalse. Negative and CFalse. Positive for the Three First Ranked Trees • Trees 7, 55 and 1 fill up the entire area in the range of CFalse. Negative and CFalse. Positive among themselves. 147
Values of CFalse. Negative and P(X=1) when Tree 7 was Ranked First • Tree 7 again fills up the major area in the range of CFalse. Negative 148 and P(X=1).
Values of CFalse. Negative and P(X=1) when Tree 55 was Ranked First • Tree 55 fills up the rest of the area in the range of CFalse. Negative 149 and P(X=1).
Values of CFalse. Negative and P(X=1) for First Ranked Trees • Together trees 7 and 55 fill up the entire region of CFalse. Negative 150 and P(X=1).
Values of CFalse. Positive and P(X=1) When Tree 7 was Ranked First • Tree 7 fills up the major area in the range of CFalse. Positive and P(X=1). 151
Values of CFalse. Positive and P(X=1) when Tree 55 was Ranked First • Tree 55 fills up the upper area in the range of CFalse. Positive and 152 P(X=1).
Values of CFalse. Positive and P(X=1) when Tree 1 was Ranked First • Tree 1 fills up the lower area in the range of CFalse. Positive and P(X=1). 153
Values of CFalse. Positive and P(X=1) for First Ranked Trees • Trees 7, 55 and 1 fill up the entire area in the range of CFalse. Positive and P(X=1) among themselves. 154
Stroud Saeger Experiments: Our Sensitivity Analysis: 4 Sensors • Second set of computer experiments: n = 4 (use sensors, A, B, C, D). • Same values as before. • Experiment 1: Fix values of two of CFalse. Negative, CFalse. Positive, P(X=1) and vary the third through their interval of possible values. • Experiment 2: Fix a value of one of CFalse. Negative, CFalse. Positive, P(X=1) and vary the other two. • Do 10, 000 experiments each time. 155 • Look for the variation in the highest ranked tree.
Stroud Saeger Experiments: Our Sensitivity Analysis: 4 Sensors • Experiment 1: Fix values of two of CFalse. Negative, CFalse. Positive, P(X=1) and vary the third. 156
CTot vs CFalse. Negative for Ranked 1 Trees (Trees 11485(9651) and 10129(349)) Only two trees ever were ranked first, and one, tree 11485, was ranked first in 9651 out of 10, 000 runs. 157
CTot vs CFalse. Positive for Ranked 1 Trees (Tree no. 11485 (10000)) One tree, number 11485, was ranked first every time. 158
CTot vs P(X=1) for Ranked 1 Trees (Tree no. 11485(8372), 10129(488), 11521(1056)) Three trees dominated first place. Trees 10201(60), 10225(17) and 159 (7) also achieved first rank but with relatively low frequency. 10153
Stroud Saeger Experiments: Our Sensitivity Analysis: 4 Sensors • Experiment 2: Fix the values of one of CFalse. Negative, CFalse. Positive, P(X=1) and vary the others. 160
Frequency of First Ranked Trees when Two Parameters (CFalse. Negative and CFalse. Positive) were Varied Keeping P(X=1) Constant at Randomly Selected Values. § 10, 000 randomized experiments with randomly selected values of P(X=1) § The experiments were repeated for 20 different randomly selected values of 161 P(X=1)
Frequency of First Ranked Trees when Two Parameters (CFalse. Negative and P(X=1)) were Varied Keeping CFalse. Positive Constant at Randomly Selected Values. § 10, 000 randomized experiments with randomly selected values of CFalse. Positive §The experiments were repeated for 20 different randomly selected values of 162 CFalse. Positive
Frequency of First Ranked Trees when Two Parameters (P(X=1) and CFalse. Positive) were Varied Keeping CFalse. Negative Constant at Randomly Selected Values. § 10, 000 randomized experiments with randomly selected values of CFalse. Negative §The experiments were repeated for 20 different randomly selected values of 163 CFalse. Negative
Modeling Sensor Errors • One Approach to Sensor Errors: Modeling Sensor Operation • Threshold Model: –Sensors have different discriminating power –Many use counts (e. g. , Gamma radiation counts) –See if count exceeds threshold –If so, say attribute is present. 164
Modeling Sensor Errors Threshold Model: • Sensor i has discriminating power Ki, threshold Ti • Attribute present if counts exceed Ti • Calculate fraction of objects in each category whose readings exceed T • Seek threshold values that minimize all costs: inspection, false positive/negative • Assume readings of category 0 containers follow a Gaussian distribution and similarly category 1 containers 165 • Simulation approach
Probability of Error for Individual Sensors • For ith sensor, the type 1 (P(Yi=1|X=0)) and type 2 (P(Yi=0|X=1)) errors are modeled using Gaussian distributions. – State of nature X=0 represents absence of a bomb. – State of nature X=1 represents presence of a bomb. – i represents the outcome (count) of sensor i. – Σi is variance of the distributions – PD = prob. of detection, PF = prob. of false pos. Ki P( i|X=0) Ti P( i|X=1) Characteristics of a typical sensor i 166
Modeling Sensor Errors The probability of false positive for the ith sensor is computed as: P(Yi=1|X=0) = 0. 5 erfc[Ti/√ 2] The probability of detection for the ith sensor is computed as: P(Yi=1|X=1) = 0. 5 erfc[(Ti-Ki)/(Σ√ 2)] erfc = complementary error function erfc(x) = (1/2, x 2)/sqrt( ) The following experiments have been done using sensors A, B, C and using: KA = 4. 37; ΣA = 1 KB = 2. 9; ΣB = 1 KC = 4. 6; ΣC = 1 We then varied the individual sensor thresholds TA, TB and TC from -4. 0 to +4. 0 in steps of 0. 4. These values were chosen since they gave us an “ROC curve” for the individual sensors over a 167 complete range P(Yi=1|X=0) and P(Yi=1|X=1)
Frequency of First Ranked Trees for Variations in Sensor Thresholds • 68, 921 experiments were conducted, as each Ti was varied through its entire range. • The above graph has frequency counts of the number of experiments when a particular tree was ranked first. There are 15 such trees. Tree 37 had the highest frequency of attaining rank one. 168
Conclusions from Sensitivity Analysis: Recapitulation • Considerable lack of sensitivity to modification in parameters for trees using 3 or 4 sensors. • Very few optimal trees. • Very few boolean functions arise among optimal and near-optimal trees. 169
Some Complications • More complicated cost models; bringing in costs of delays • More than two values of an attribute -(present, absent, present with probability > 75%, absent with probability at least 75%) -(ok, not ok, ok with probability > 99%, ok with probability between 95% and 99%) • Inferring the boolean function from observations (partially defined boolean 170 functions)
Some Research Challenges • Explain why conclusions are so insensitive to variation in parameter values. • Explore the structure of the optimal trees and compare the different optimal trees. • Develop less brute force methods for finding optimal trees that might work if there are more than 4 attributes. • Develop methods for approximating the optimal tree. 171
Port of Entry Inspection: Closing Remark • Recall that the “cost” of inspection includes the cost of failure, including failure to foil a terrorist plot. • There are many ways to lower the total “cost” of inspection: -Use more efficient orders of inspection. -Find ways to inspect more containers. -Find ways to cut down 172 on delays at inspection lanes.
Concluding Comment • In recent years, interplay between CS and biology has transformed major parts of Bio into an information science. • Led to major scientific breakthroughs in biology such as sequencing of human genome. • Led to significant new developments in CS, such as database search. • The interplay between CS and SS-DM not nearly as far along. • Moreover: problems are spread over many disciplines. 173
Concluding Comment • However, CS/SS-DM interplay has already developed a unique momentum of its own. • One can expect many more exciting outcomes as partnerships between computer scientists and social scientists/decision theorists expand mature. 174
175
Скачать презентацию Computer Science and Decision Making Fred Roberts Rutgers
f97851be6af7d899ba3f3d017bd5d58e.ppt