f2936f84804faaff776315c1c42ee2c8.ppt

- Количество слайдов: 32

Fundamentals of Software Engineering Economics (IV) Software Engineering Economics Dealing with Uncertainties (Chapters 19 -20) Li. Guo Huang Computer Science and Engineering Southern Methodist University 1

Dealing With Uncertainties • • • TPS Example: Uncertain outcome Decision rules for complete uncertainty Utility functions Expected value of perfect information Statistical decision theory The value-of-information procedure 2

TPS Decision Problem 7 OS Development Options: • Option B-Conservative (BC) – Sure to work (cost=$600 K) – Performance = 4000 tr/sec • Option B-Bold (BB) – If successful, Cost=$600 K Perf. = 4667 tr/sec – If not, Cost=$100 K Perf. = 4000 tr/sec Which option should we choose? 3

Operating System Development Options Option BB (Bold) Successful Not Successful Option BC (Conservative) Option A Performance (tr/sec) 4667 4000 2400 Value ($0. 3 K per tr/sec) 1400 1200 720 Basic cost 600 600 170 Total cost 600 700 600 170 Net value NV 800 500 600 550 250 -50 50 0 NV relative to Option A 4

Decision-making under Complete Uncertainty • The outcome or payoff depends on which of several states of nature may hold. • Given any state of nature, the payoff for each alternative is known. • The probability that any given state of nature holds is unknown. 5

Payoff Matrix for OS Options BB, BC State of Nature Alternative Favorable Unfavorable BB (Bold) 250 -50 BC (Conservative) 50 50 6

Decision Rules for Complete Uncertainty (1) • Maximin Rule. Determine minimum payoff for each option. Choose option maximizing the minimum payoff. (Most Pessimistic) Alternative Favorable Unfavorable BB 1, 000 -50 BC 50 50 7

Decision Rules for Complete Uncertainty (2) • Maximax Rule. Determine max-payoff for each option. Choose option maximizing max-payoff. (Most optimistic) Alternative Favorable Unfavorable BB 51 -1, 000 BC 50 50 8

Decision Rules for Complete Uncertainty (3) • Laplace Rule. Assume all states of nature equally likely. Choose option with maximum expected value. BB = 0. 5(250) + 0. 5(-50) = 100 BC = 0. 5(50) + 0. 5(50) = 50 9

Difficulty with Laplace Rule Favorable U 1 U 2 BB 250 -50 Expected Value 50 BC 50 50 U 1 : performance failure U 2 : reliability failure 10

Breakeven Analysis • Treat uncertainty as a parameter P= Prob (Nature is favorable) • Compute expected values as f(P) EV (BB) = P(250) + (1 -P)(-50) = -50 + 300 P EV (BC) = P(50) + (1 -P)(50) = 50 • Determine breakeven point P such that EV (BB) = EV (BC): -50 + 300 P = 50, P=0. 333 If we feel that actually P>0. 333, Choose BB P<0. 333, Choose BC 11

Utility Functions Is a Guaranteed $50 K The same as A 1/3 chance for $250 K and A 2/3 chance for -$50 K? 12

Utility Function for Two Groups of Managers Utility 20 10 -3 -2 -1 1 2 3 4 5 Payoff ($M) -10 Group 1 -20 Group 2 13

Possible TPS Utility Function Utility 20 10 (250, 20) (50, 5) (170, 15) (100, 10) 100 -10 200 300 Payoff ($M) (-50, -20) -20 (-80, -25) 14

Utility Functions in S/W Engineering • Managers prefer loss-aversion – EV-approach unrealistic with losses • Managers’ U. F. ’S linear for positive payoffs – Can use EV-approach then • People’s U. F. ’S aren’t – Identical – Easy to predict – Constant 15

TPS Decision Problem 8 State of Nature Alternative Favorable Unfavorable BB (Bold) 250 -50 BC (Conservative) 50 50 • Available decision rules inadequate • Need better information Info. has economic value 16

Expected Value of Perfect Information (EVPI) Build a Prototype for $10 K – If prototype succeeds, choose BB Payoff: $250 K – 10 K = $240 K – If prototype fails, choose BC Payoff: $50 K – 10 K = $40 K If equally likely, EV = 0. 5 ($240 K) + 0. 5 ($40 K) = $140 K Could invest up to $50 K and do better than before – thus, EVPI = $50 K 17

However, Prototype Will Give Imperfect Information That is, P(IB|SF) = 0. 0 Investigation (Prototype) says choose bold state of nature: Bold will fail P(IB|SS) = 1. 0 Investigation (prototype) says choose bold state of nature: bold will succeed 18

Suppose we assess the prototype’s imperfections as P(IB|SF) = 0. 20, P(IB|SS) = 0. 90 And Suppose the states of nature are equally likely P(SF) = 0. 50 P(SS) = 0. 50 We would like to compute the expected value of using the prototype EV(IB, IC) = P(IB) (Payoff if use bold) +P(IC) (Payoff if use conservative) = P(IB) [ P(SS|IB) ($250 K) + P(SF|IB) (-$50 K) ] + P(IC) ($50 K) But these aren’t the probabilities we know 19

How to get the probabilities we need P(IB) = P(IB|SS) P(SS) + P(IB|SF) P(IC) = 1 – P(IB) P(SS|IB) = P(IB|SS) P(IB) (Bayes’ formula) P(SF|IB) = 1 – P (SS|IB) P(SS|IB) = Prob (we will choose Bold in a state of nature where it will succeed) Prob (we will choose Bold) 20

Net Expected Value of Prototype PROTO COST, $K P (IB|SF) P (IB|SS) NET EV, $K 60 0 EV, $K 0 0. 30 0. 80 69. 3 4. 3 10 0. 20 0. 90 78. 2 20 0. 10 0. 95 86. 8 30 0. 00 1. 00 90 0 NET EV, $K 5 8 4 0 10 20 PROTO COST, $K 30 21

Discussion: Expected Value of Perfect Information (EVPI) (1) Given a choice between m alternatives: A 1, A 2, …, Am • n possible states of nature: S 1, S 2, …, Sn • probability of occurrence P(S 1), P(S 2), …, P(Sn) • payoff value of choosing Ai in state of nature Sj S 1 S 2 … Sn A 1 V 12 … V 1 n A 2 V 21 V 22 … V 2 n . . . Am Vm 1 Vm 2 … . . Vmn 22

Discussion: Expected Value of Perfect Information (EVPI) (2) • Use subjective probability approach to compute the expected value of choosing each alternative Ai • If we have no information, pick the alternative providing the maximum expected value • If we have perfect information on which state of nature will occur, 23

Discussion: Expected Value of Perfect Information (EVPI) (3) • The expected value of acquiring the perfect information (EVPI) : • Example 24

Discussion: Expected Value of Imperfect Information , for j = 1, …, n Bayes’s formula: 25

Value-of-Information Procedure 1. Formulate a set of alternative IS development approaches A 1, A 2, …, Am 2. Determine the states of nature S 1, S 2, …, Sn 3. Determine the values Vij in the payoff matrix 4. Determine the probability P(Sj) 5. Compute (EV)no info, (EV)perfect info, and EVPI 6. If EVPI is negligible, choose the approach Ai which provides the best (EV)no info 7. If EVPI is not negligible, it provides a rough upper bound for the investigation effort. 8. For each type of investigation, estimate P(IAi|Sj) 9. Compute the expected value of information provided by investigation K, EV(Ik) 10. Compute the net value of each investigation NV(Ik)= EV(Ik)-Ck 26

Value-of-Information Procedure (Cont. ) Considering choosing a preferred investigation approach: • • The relative net values of investigations Whether or not any of the net values are positive Whether the results of the investigation will be available in time The relative magnitude of side benefits of the investigations 27

Using Value-of-Information Procedure in Software Engineering How much should we invest in further information gathering and analysis investigations before committing ourselves to a Course of action? 1. 2. 3. 4. How much should we invest in feasibility studies before committing to a particular concept for development? How much should we invest in COTS product analysis? How much should we invest in risk analysis? How much should we invest in V&V? 28

Conditions for Successful Prototyping (or Other Info-Buying) 1. 2. 3. 4. 5. There exist alternatives whose payoffs vary greatly depending on some states of nature. The critical states of nature have an appreciable probability of occurring. The prototype has a high probability of accurately identifying the critical states of nature. The required cost and schedule of the prototype does not overly curtail its net value. There exist significant side benefits derived from building the prototype. 29

Pitfalls Associated by Success Conditions 1. Always build a prototype or simulation – May not satisfy conditions 3, 4 2. Always build the software twice – May not satisfy conditions 1, 2 3. Build the software purely top-down – May not satisfy conditions 1, 2 4. Prove every piece of code correct – May not satisfy conditions 1, 2, 4 5. Nominal-case testing is sufficient – May need off-nominal testing to satisfy conditions 1, 2, 3 30

Statistical Decision Theory: Other S/W Engineering Applications How much should we invest in: – User information gathering – Make-or-buy information – Simulation – Testing – Program verification How much should our customers invest in: – MIS, query systems, traffic models, CAD, automatic test equipment, … 31

The Software Engineering Field Exists Because Processed Information Has Value 32