USC C S E University of Southern California

USC C S E University of Southern California Center for Software Engineering Ethics Barry Boehm Fall 2015 Presented by Jim Alstad 2015/12/02 © USC-CSSE 1

USC C S E University of Southern California Center for Software Engineering Outline • Definitions and context – Power to do public harm or good – ACM/IEEE Software Engineering Code of Ethics • Principles and examples – Rawls’ Theory of Justice – Relation to stakeholder win-win – Case study: Mercy Hospital • Integrating ethics into daily software engineering practices – VBSE/MBASE/Win Spiral Model 2015/12/02 © USC-CSSE 2

USC C S E University of Southern California Center for Software Engineering Definition of “Ethics” -Webster, 1993 • The discipline dealing with what is good and bad – And with moral duty and obligation • A theory, system, or set of moral principles or values • The principles of conduct governing an individual or group – Professional ethics 2015/12/02 © USC-CSSE 3

USC C S E University of Southern California Center for Software Engineering Context • Software engineers have increasing power to do public harm or good – Intellectual property, privacy, confidentiality, quality of work, fairness, liability, risk disclosure, conflict of interest, unauthorized access • Professional societies have developed codes of ethics • Hard to integrate value-based ethics into valueneutral software engineering practices • VBSE/MBASE/Win Spiral enable ethics integration 2015/12/02 © USC-CSSE 4

USC C S E University of Southern California Center for Software Engineering Power to Do Public Harm or Good – I • Intellectual Property: use without credit; use copyrighted material • Privacy: credit, health, personal information • Confidentiality: competitive information, political sensitivity • Quality of work: many dimensions; see table 2015/12/02 © USC-CSSE 5

USC C S E University of Southern California Center for Software Engineering Example: Confidentiality • Government agency hires company to support SW procurement – Provides data under nondisclosure agreement • Employee and company consultant prepare cost estimate – Employee: “ I don’t see how anyone can do all this for $8 M” • Consultant provides $8 M target cost to some bidders • Government agency angry with company for leak – Whose fault? How could it be avoided? 2015/12/02 © USC-CSSE 6

USC University of Southern California C S E Center for Software Engineering Information Consumers Mission - Protection critical Safety ** Security * ** ** Privacy ** * Robustness Reliability Availability Acquirers indirect Administrators 0 Insignificant or System Controllers *Significant Info Brokers Info Suppliers Stakeholder Classes System Dependents **Critical Developers, Maintainers Quality Concerns Vary by Stakeholders Role uncrit. ** ** ** * * Survivability ** ** Quality of Service Performance ** ** * * Accuracy, Consistency ** ** ** * Accessibility, ease of use; * ** ** * difficulty of misuse Evolvability ** ** ** ** Interoperability ** * ** Correctness ** ** Cost * ** Schedule * ** ** Reusability ** * * 2015/12/02 © USC-CSSE 7

USC C S E University of Southern California Center for Software Engineering Power to Do Public Harm or Good - II • Fairness: equality of opportunity/treatment; fair reward system • Liability: accountability; parity of authority and responsibility • Risk Disclosure: safety tests, COTS capabilities; schedule slips • Conflict of Interest: source selection; personnel or product reviews • Unauthorized Access: reading, copying, modifying; denial of service 2015/12/02 © USC-CSSE 8

USC University of Southern California C S E Center for Software Engineering Examples: Fairness • Enron software to schedule power outages, raise prices – Suppose you had been asked to develop it? • Urban fire dispatching system – Inefficient old system caused $700 M property loss – New-system spec. includes dispatching algorithm to minimize property loss • Any fairness issues? 2015/12/02 © USC-CSSE 9

USC C S E University of Southern California Center for Software Engineering Example: Safety Tests • Your company is delivering a drug prescription fulfillment system – Reusing software from a warehouse inventory system • You are the quality assurance manager – With company responsibility for certifying product safety • The software has passed all the contracted tests – But many off-nominal conditions untested – Some have shown unsafe outcomes – You feel more off-nominal testing is necessary • Company president says if you don’t certify safety by delivery date, company may go out of business – What should you do? 2015/12/02 © USC-CSSE 10

USC C S E University of Southern California Center for Software Engineering ACM/IEEE Software Engineering Code of Ethics -Table of Contents 1. Products: achievable goals, realistic estimates, high quality 2. Public: safety, respect of diversity, public interest first 3. Judgment: objectivity, no bribes or conflicts of interest 4. Client and Employer: no employer-adverse interests, surface problems 5. Management: fair, ethical work rules, due process for violations 6. Profession: support profession and ethics code, don’t misrepresent software 7. Colleagues: credit colleagues’ work, give colleagues a fair hearing 8. Self: improve your technical and ethical knowledge and 2015/12/02 © USC-CSSE 11 practices

USC University of Southern California C S E Center for Software Engineering Code of Ethics 2. Public 2. 01 2. 02 2. 03 2. 04 2. 05 2. 06 2. 07 2. 08 2. 10 Disclose any software-related dangers Approve only safe, well tested software Only sign documents in area of competence Cooperate on matters of public concern Produce software that respects diversity Be fair and truthful in all matters Always put the public’s interest first Donate professional skills to good causes Accept responsibility for your own work 2015/12/02 © USC-CSSE 12

USC University of Southern California C S E Center for Software Engineering Code of Ethics 4. Client and Employer 4. 01 4. 02 4. 03 4. 04 4. 05 4. 06 4. 07 4. 08 4. 09 Provide services only where competent Ensure resources are authentically approved Only use property as authorized by the owner Do not use illegally obtained software Honor confidentiality of information Raise matters of social concern Inform when a project becomes problematic Accept no detrimental outside work Represent no interests adverse to your employer 2015/12/02 © USC-CSSE 13

USC University of Southern California C S E Center for Software Engineering Outline • Definitions and context – Power to do public harm or good – ACM/IEEE Software Engineering Code of Ethics • Principles and examples – Rawls’ Theory of Justice – Relation to stakeholder win-win – Case study: Mercy Hospital • Integrating ethics into daily software engineering practices – VBSE/MBASE/Win Spiral Model – CS 577 ethics situations 2015/12/02 © USC-CSSE 14

USC C S E University of Southern California Center for Software Engineering Rawls’ Theory of Justice (1971) -Following Collins et al. , “How Good Is Good Enough? ” Comm. ACM, Jan. 1994 • Fair rules of conduct • Principles of justice • Participants and obligations – Provider (developer) – Buyer (acquirer) – User(s) – Penumbra (general public) • Negotiate mutually satisfactory (win-win) agreements 2015/12/02 © USC-CSSE 15

USC University of Southern California C S E Center for Software Engineering Rawls’ Theory of Justice - II • Fair rules of conduct – Negotiation among interested parties – Veil of ignorance (about what affects whom) – Rationality • Principles – Least Advantaged - don’t increase harm to them • Harm = probability x magnitude (~risk exposure) – Risking harm - don’t risk increasing harm • Don’t use “low-threat” software in “high-threat” context – Publicity test - defensible with honor before an informed public 2015/12/02 • Use for difficult cost-benefit tradeoffs © USC-CSSE 16

USC C S E University of Southern California Center for Software Engineering Obligations of the Software Provider 2015/12/02 © USC-CSSE 17

USC C S E University of Southern California Center for Software Engineering Obligations of the Software Buyer 2015/12/02 © USC-CSSE 18

USC University of Southern California C S E Center for Software Engineering Obligations of the Software User 2015/12/02 © USC-CSSE 19

USC C S E University of Southern California Center for Software Engineering Obligations of the Software Penumbra 2015/12/02 © USC-CSSE 20

USC C S E University of Southern California Center for Software Engineering Case Study: Mercy Hospital Pharmacy System -Collins et al. , 1994 • Growing hospital – Manual pharmacy information system reaching overload • Spec developed for PC-based information system – Rachel: VP, Records & Automation – George: Chief Pharmacist • System developed by consultants – – Hired by George Rachel: test procedures Based on mature warehouse inventory system Budgeted 50% more testing than other bidders • Installation & Training discovers problems – Helen: consultant in charge of installation & training – Ann: skeptical nurse cross-checking computer outputs 2015/12/02 © USC-CSSE 21

USC C S E University of Southern California Center for Software Engineering Mercy Hospital Pharmacy System: Problems • Dosage problems from data entry errors – 10 x dosage; wrong patient • Cross-checking incomplete; not trusted by some doctors • Heavier data-entry load – Formalizing automated procedures more info. needed – Pharmacy info > warehouse info • Helen: Should go back to old system during cleanup • George: - Is old system less risky? - How do we ensure cleanup will get it right? - How much will cleanup cost? • Future practice: How to anticipate, avoid similar problems? 2015/12/02 © USC-CSSE 22

USC University of Southern California C S E Center for Software Engineering Outline • Definitions and context – Power to do public harm or good – ACM/IEEE Software Engineering Code of Ethics • Principles and examples – Rawls’ Theory of Justice – Relation to stakeholder win-win – Case study: Mercy Hospital • Integrating ethics into daily software engineering practices – VBSE/MBASE/Win Spiral Model – CS 577 ethics situations 2015/12/02 © USC-CSSE 23

USC C S E University of Southern California Center for Software Engineering Mercy Hospital : Use of VBSE/MBASE/Win Spiral • Results chain – Add patient safety outcome, patient stakeholder representative – Rework-business-workflows initiative, including safety checks; add clerical-staff stakeholder • Stakeholder Win – Patient representative: safety criteria; parallel-operation phasein – Clerical staff: prototype GUI, including safety-check support • Business Case: includes added safety costs and benefits • Risk Management: assess warehouse package safety, effects of workflow changes. 2015/12/02 © USC-CSSE 24

USC C S E University of Southern California Center for Software Engineering Use of VBSE/MBASE/Win Spiral-II • Concurrent Engineering – Concurrently address business workflows, GUI prototypes, COTS alternatives, feature prioritization, cost/schedule/benefits analysis, other risks – Prepare to pass LCO, LCA, CCD, and IOC anchor point milestone reviews • Monitoring and Control: Use Balanced Scorecard to track progress with respect to plans; apply corrective actions as necessary • Change as Opportunity: Look for emerging COTS pharmacy-related fulfillment systems 2015/12/02 © USC-CSSE 25

USC C S E University of Southern California Center for Software Engineering Conclusions • Software engineers have increasing power to do public harm or good • Value-based codes of ethics are hard to integrate with value-neutral software engineering practices • Rawls’ Theory of Justice enables constructive approach for integrating ethics into daily software engineering practice – Stakeholder win-win with least-advantaged system dependents as success-critical stakeholders – Win Spiral Model/MBASE/VBSE provides dailypractice framework 2015/12/02 © USC-CSSE 26