the e-risks of e-commerce Professor Ken Birman Dept

the e-risks of e-commerce Professor Ken Birman Dept. of Computer Science Cornell University

Reliability w If it stops ticking when it takes a licking… your e-commerce company could tank w So you need to know that your technology base is reliable w It does what it should do, does it when needed, does it correctly, and is accessible to your customers.

A Quiz w Q: When and why did Sun Microsystems have a losing quarter? Ken Birman: Mr. Birman, Sun experienced a loss in Q 4 FY 89 (June 1989). This was the quarter in which we transitioned to a new manufacturing, order processing and inventory control systems. Andrew Casey Manager, Investor Relations Sun Microsystems, Inc. (650) 336 -0761 andrew. casey@corp. sun. com

Typical Web Session get http: //www. cs. cornell. edu/People/ken where firewall what

Typical Web Session DNS root resolve “www. cs. cornell. edu” DNS leaf caching proxy firewall DNS node DNS root DNS leaf IP address 128. 64. 31. 77 caching proxy DNS leaf load-balancing proxy web server

The Web’s dark side Netscape error: web server www. cs. cornell. edu. . . not responding. Server may have crashed or is overloaded. OK

Right URL, but the request times out. Why? The web server could be down Your network connection may have failed There could be a problem in the “DNS” There could be a network routing problem The Internet may be experiencing an overload Your web caching proxy may be down Your PC might have a problem, or your version of Netscape (or Explorer), or the file system you are using, or your LAN w The URL itself may be wrong w A router or network link may have failed and the Internet may not yet have rerouted around the problem w w w w

E-Trade computers crash again -- and again Edupage Editors <edupage@franklin. oit. unc. edu> Sun, 07 Feb 1999 10: 28: 30 -0500 The computer system of online security firm E-Trade crashed on Friday for the third consecutive day. "It was just a software glitch. I think we were all frustrated by it, " says an E-Trade executive. Industry analyst James Mark of Deutsche Bank commented “…it's the application on a large scale. As soon as E-Trade's volumes started spiking up, they had the same problems as others…. "

Reliable Distributed Computing: Increasingly urgent, yet unsolved w Distributed computing has swept the world n n n Impact has become revolutionary Vast wave of applications migrating to networks Already as critical a national infrastructure as water, electricity, or telephones w Yet distributed systems remain n Unreliable, prone to inexplicable outages Insecure, easily attacked Difficult (and costly) to program, bug-prone

A National Imperative w Potential for catastrophe cited by n n Presidential Commission on Critical Infrastructure Protection (PCCIP) National Academy of Sciences Study on Trust in Cyberspace w These experts warn that we need a quantum improvement in technologies w Meanwhile, your e-commerce venture is at grave risk of stumbling – just like many others

A Business Imperative

E-projects Often Fail w e-commerce revolves around computing w Even business and marketing people are at the mercy of these systems w When your company’s computing systems aren’t running, you’re out of business

Big and Little Pictures w It is too easy to understand “reliability” as a narrow technical issue w In fact, many systems and companies stumble by building n n Unreliable technologies, because of A mixture of poor management and poor technical judgment w Reliable systems demand a balance between good management and good technology

A Glimpse of Some “Unreliable Systems” w Quick review of some failed projects w These were characterized by poor reliability of the final product w But the issues were not really technical w As future managers you need to understand this phenomenon!

Tales from the Software Crypt NYC Control of 10, 000 Traffic Lights Univac, based on experience in Baltimore and Toronto Started in late 1960’s Scrapped 2 -3 years later Spent: ? Second system effect: New radio control system New software, algorithms Earlier systems were 100 x smaller Incommensurate scaling California Dept. of Motor Vehicles Vehicle registration and drivers licenses Started in 1987 Scrapped 1994 Spent: $44 M Underestimated cost by a factor of 3 Slower than 1965 system Governor fired the whistleblower DMV blames Tandem, Tandem blames DMV United Airlines/UNIVAC Automated reservations, ticketing, flight schedules, fuel delivery, kitchens, and general administration Started in late 1960’s Scrapped early 1970’s Spent: $50 M Second system effect: Hilton, Marriot, Budget, American Airlines Hotel res. , links to Wizard and Sabre Started: 1988 Scrapped: 1992 Spent: $125 M Second system Very dull tools (machine language) Bad-news diode See CACM October 1994 for details CONFIRM Source: Jerry Saltzer, Keynote address, SOSP 1999 Tried to automate everything, including kitchen sink Ditto: Burroughs/TWA. Delta currently planning to build something similar But they will use the web. “Magic bullet” concept… Today uses web, works well

Tales from the Software Crypt SACSS (California) State-wide system for automated child support tracking Started 1991 ($99 M) “On hold” 1997 Spent: $300 M Lockheed and HWDC disagree on what the system contains and which part of it isn’t working Taurus (British Stock Exchange) Replacement for British Stock Exchange Started 1980’s Scrapped 1993 Spent $600 M “massive complexity of the back-end settlement systems…” delays and cost overruns IBM Workplace OS for PC Mach 3. 0 + binary compatibility with Pink, AIX, DOS, OS/400 + new clock mgt. + new RPC + new I/O + new CPU Started in 1991 Scrapped 1996 Spent: $2 B 400 staff on kernel, 1500 elsewhere “sheer complexity of the class structure proved to be overwhelming” Replacement for “in route” air traffic control system Started 1982 Scrapped 1994 Spent more than $6 B Management misestimated size and length of project Project goals constantly changed Advanced Automation System (AAS) Source: Jerry Saltzer, Keynote address, SOSP 1999 “Departments shouldn’t deploy a system to additional users if it is not working” Even question of how to represent numbers wasn’t settled Early design choices and compatibility decisions doomed the project Poor technology choices Run by gov’t. bureaucrats

1995 Standish Group Study Over budget Missed schedule Lacks functions “Success” 20% On time On budget On function “Challenged” 50% “Impaired” 30% Scrapped 2 x budget 2 x competion time 2/3 planned functionality Source: Jerry Saltzer, Keynote address, SOSP 1999

A strange picture w Many technology projects fail n For lots of reasons w But some succeed n n Today we do web-based hotel reservations all the time, yet “Confirm” failed French air traffic project was a success yet US project lost $6 billion w Is there a pattern?

Recurring Problems Incommensurate scaling Too many ideas Mythical man-month Bad ideas included Modularity is hard Bad-news diode Best people are far more productive than average employees w New is better, not-even-available yet is best w Magic bullet syndrome w w w w Source: Jerry Saltzer, Keynote address, SOSP 1999

1995 Study of Tandem Computer Systems w 77% of failures that are software problems. w Software fault-tolerance techniques can overcome about 75% of detected faults. w Loose coupling between primary and backup is important for software fault tolerance. w Over two-thirds (72%) of measured software failures are recurrences of previously reported faults. Source: Jerry Saltzer, Keynote address, SOSP 1999

A Buggy Aside w Q: What are the two main categories of software bugs called? w A: Bohrbugs and Heisenbugs w Q: Why?

Bohr Model of Atom w Bohr argued that the nucleus was a little ball

Bohr Model of Atom w Bohr argued that the nucleus was a little ball w Bohr bug is a nasty but well defined thing

Bohr Model of Atom w Bohr argued that the nucleus was a little ball w Bohr bug is a nasty but well defined thing w Your technical people can reproduce it, so they can nail it

Heisenbug w Heisenberg modeled atom as a cloud of electroms and a cloud-like nucleus w The closer you look, the more it wiggled w A Heisenbug moves when your people try and pin it down. They won’t find it easy to fix.

Why? w Bohrbugs tend to be deterministic errors – outright mistakes in the code w Once you understand what triggers them they are easy to search for and fix w Heisenbugs are often delayed side-effects of an old error. Like a bad tank of gas, effect may happen long after the bug first “occurs”. Hard to fix because at the time the mistake happened, nothing obvious went wrong

Why Systems fail w Mostly, because something crashes w Usually, software or a human error w Mean time to failure improves with age but software problems remain prevalent w Every kind of software system is prone to failures. Failure to plan for failures is the most common way for e-systems to fail.

E-reliability w We want e-commerce solutions to be reliable… but what should this mean? n n Fault-tolerant? Secure? Fast enough? Accessible to customers? w Deliver critical services when needed, where needed, in a correct, timely manner

Costs of a Failure

Minimizing Downtime w Idea is to design critical parts of your system to survive failures w Two basic approaches n n Recoverable systems are designed to restart without human intervention – but may wait until outage is repaired Highly available systems are designed to keep running during failure

Recoverability w The technology is called “transactions” n We’ll discuss this next time, but… w Main issue is time needed to restart the service w For a large database, half an hour or more is not at all unusual w Faster restart requires a “warm standby”

High Availability w Idea is to have a way to keep the system running even while some parts are crashed w For example, a backup that takes over if primary fails w Backup is kept “warm” n n This involves replicating information As changes occur, backup may lag behind

Complexity w The looming threat to your e-commerce solution, no matter what it may be w Even simple systems are hard to make reliable w Complex systems are almost impossible to make reliable w Yet innovative e-commerce projects often require fairly complex technologies!

Two Side-by-Side Case Studies w American Advanced Automation System n n n Intended as replacement for air traffic control system Needed because Pres. Reagan fired many controllers in 1981 But project was a fiasco, lost $6 B w French Phidias System n n Similar goals, slightly less ambitious But rolled out, on time and on budget, in 1999

Background w Air traffic control systems are using 1970’s technology w Extremely costly to maintain and impossible to upgrade w Meanwhile, load on controllers is rising steadily w Can’t easily reduce load

Air Traffic Control system (one site) Onboard Radar Team of Controllers X. 500 Directory Air Traffic Database (flight plans, etc)

Politics w Government wanted to upgrade the whole thing, solve a nagging problem w Controllers demanded various simplifications and powerful new tools w Everyone assumed that what you use at home can be adapted to the demands of an air traffic control center

Technology w IBM bid the project, proposed to use its own workstations w These aren’t super reliable, so they proposed to adapt a new approach to “fault-tolerance” w Idea is to plan for failure n n Detect failures when they occur Automatically switch to backups

Core Technical Issue? w Problem revolves around high availability w Waiting for restart not seen as an option: goal is 10 sec downtime in 10 years w So IBM proposed a replication scheme much like the “load balancing” approach w IBM had primary and backup simply do the same work, keeping them in the same state

Technology radar find tracks Identify flight Lookup record Plan actions Human action Conceptual flow of system IBM’s fault-tolerant process pair concept radar find tracks Identify Lookup flight record Plan actions Human action

Why is this Hard? w The system has many “real-time” constraints on it n n Actions need to occur promptly Even if something fails, we want the human controller to continue to see updates w IBM’s technology n n Based on a research paper by Flaviu Cristian But had never been used except for proof of concept purposes, on a small scale in the laboratory

Politics w IBM’s proposal sounded good… n n … and they were the second lowest bidder … and they had the most aggressive schedule w So the FAA selected them over alternatives w IBM took on the whole thing all at once

Disaster Strikes w Immediate confusion: all parts of the system seemed interdependent n To design part A I need to know how part B, also being designed, will work w Controllers didn’t like early proposals and insisted on major changes to design w Fault-tolerance idea was one of the reasons IBM was picked, but made the system so complex that it went on the back burner

Summary of Simplifications w Focus on some core components w Postpone worry about fault-tolerance until later w Try and build a simple version that can be fleshed out later … but the simplification wasn’t enough. Too many players kept intruding with requirements

Crash and Burn w The technical guys saw it coming n n n Probably as early as one year into the effort But they kept it secret (“bad news diode”) Anyhow, management wasn’t listening (“they’ve heard it all before – whining engineers!”) w The fault-tolerance scheme didn’t work n Many technical issues unresolved w The FAA kept out of the technical issues n But a mixture of changing specifications and serious technical issues were at the root of the problems

What came out? w In the USA, nothing. w The entire system was useless – the technology was of an all-or-nothing style and nothing was ready to deploy w British later rolled out a very limited version of a similar technology, late, with many bugs, but it does work…

Contrast with French w They took a very incremental approach n n n Early design sought to cut back as much as possible If it isn’t “mandatory” don’t do it yet Focus was on console cluster architecture and fault-tolerance w They insisted on using off-the-shelf technology

Contrast with French w Managers intervened in technology choices n n n For example, the vendor wanted to do a home-brew fault-tolerance technology French insisted on a specific existing technology and refused to bid out the work until vendors accepted A critical “good call” as it worked out

Learning by Doing w To gain experience with technology n n n They tested, and tested Designed simple prototypes and played with them Discovered that large cluster would perform poorly But found a “sweet spot” and worked within it This forced project to cut back on some goals

Testing w 9/10 th of time and expense on any system is in n Testing Debugging Integration w Many projects overlook this w French planned conservatively

Software Bugs Figure 1/10 lines in new code But as many as 1/250 lines in old code Bugs show up under stress Trick is to run a system in an unstressed mode French identified “stress points” and designed to steer far from them w Their design also assumed that components would fail and automated the restart w w w

All of this worked! w Take-aways from French project? n n n Complex technical issues at the core of the system But they managed to break big poject into pieces Do the critical core first, separately, and focus exclusively on it Test, test Don’t build anything you can possibly buy Management was technically sophisticated enough to make some critical “calls”

Your Problem w e-commerce systems are at e-risk w These e-risks take many forms: n n n System complexity Failure to plan for failures Poor project management w Ignore this at our peril, as we’ve seen w But how can we learn to do better?

Keys to Reliability w Know the basic technologies w Realize that software is buggy and failures will happen. n n Design to treat failure as a mundane event Failure to plan for failure is the biggest e-risk! w Complexity is a huge threat. Use your naiveté as an advantage: if you can’t understand it, why assume that “they” can understand it?

E-commerce Technologies w The network and associated services w Databases w Web servers w “Scripts” – the glue your people use to tie it all together

Next Lecture w Look at some realistic e-commerce systems w Ask ourselves where to start first, if we need to convince ourselves that the system will be reliable enough w Trick is to balance between system complexity and adequate risk coverage