Скачать презентацию The Failure of the London Ambulance Service Michael Скачать презентацию The Failure of the London Ambulance Service Michael

4e9d7f66251963d972630f5f9c9e1fe6.ppt

  • Количество слайдов: 42

The Failure of the London Ambulance Service Michael Mc. Dougall CIS 573 November 16 The Failure of the London Ambulance Service Michael Mc. Dougall CIS 573 November 16 th, 1999

The Accident On October 26 th 1992 the London Ambulance System failed. Phones rang The Accident On October 26 th 1992 the London Ambulance System failed. Phones rang for up to 10 minutes Ambulance response times were delayed Some calls were lost On November 2 nd the system crashed completely. Software was a major cause of the failures.

Outline London Ambulance Service Computer Aided Despatch (CAD) system Background Planning the system Developing Outline London Ambulance Service Computer Aided Despatch (CAD) system Background Planning the system Developing the system How it failed ISO 12207 – Software Development Standard LAS failure w. r. t. ISO standard

Background The London ambulance service (LAS) is was the largest ambulance service in the Background The London ambulance service (LAS) is was the largest ambulance service in the world. 6. 8 million residents – much higher during daytime. Services 5000 patients a day. Handles between 2000 and 2500 calls a day (more than 1 per minute). Employs 2700 full-time staff.

Background In 1990 the LAS was not meeting the U. K. standards for ambulance Background In 1990 the LAS was not meeting the U. K. standards for ambulance response times. Other parts of the U. K. National Health Service had undergone reforms throughout the 80’s but the LAS had not changed much since 1980. Staff/Management relations were low.

Despatch system The despatch system was responsible for: Taking emergency calls Deciding which ambulance Despatch system The despatch system was responsible for: Taking emergency calls Deciding which ambulance to send Sending information to ambulances Managing allocation of ambulances

Despatch system Take Call RA Paper RA Collection Point Paper Regional Allocator Voice Despatcher Despatch system Take Call RA Paper RA Collection Point Paper Regional Allocator Voice Despatcher Paper Ambulance

Despatch system The UK national standard required that this take less than 3 minutes. Despatch system The UK national standard required that this take less than 3 minutes. The LAS system in 1990 had a number of inefficiencies which made it impossible to meet the standard.

Inefficiencies Take Call Finding the location of an accident was often difficult and time Inefficiencies Take Call Finding the location of an accident was often difficult and time consuming.

Inefficiencies Paper Moving pieces of paper took unnecessary time Inefficiencies Paper Moving pieces of paper took unnecessary time

Inefficiencies Identifying duplicate calls relied on human memory and was therefore slow and error Inefficiencies Identifying duplicate calls relied on human memory and was therefore slow and error prone. Collection Point

Inefficiencies Voice Allocator Despatcher Paper Ambulance Voice communication was slow Allocating ambulances was done Inefficiencies Voice Allocator Despatcher Paper Ambulance Voice communication was slow Allocating ambulances was done by hand. Relied on memory of allocator. Regional

Improving the system The LAS was under pressure from their superiors, MPs, the public Improving the system The LAS was under pressure from their superiors, MPs, the public and the media to improve performance. LAS management decided that a Computer Aided Despatch system was the fastest way to improve service.

The Plan LAS wanted to radically change the despatch system. In Autumn 1991 they The Plan LAS wanted to radically change the despatch system. In Autumn 1991 they began to write the system requirements for the new system.

CAD system goals Take Call Finding the location of an accident was often difficult CAD system goals Take Call Finding the location of an accident was often difficult and time consuming. Software connected to public telephones will locate incidents automatically

CAD system goals Paper Information will move through a network between workstations. Paper Moving CAD system goals Paper Information will move through a network between workstations. Paper Moving pieces of paper took unnecessary time

CAD system goals Identifying duplicate calls relied on human memory and was therefore slow CAD system goals Identifying duplicate calls relied on human memory and was therefore slow and error prone. Collection Point AI will try to identify duplicate calls.

CAD system goals Allocating ambulances was done by hand. Relied on memory of allocator. CAD system goals Allocating ambulances was done by hand. Relied on memory of allocator. Regional Allocation of nearest ambulance will be done by computer in most cases. Allocator

CAD system goals Ambulance Voice communication was slow Voice Digital communication to and from CAD system goals Ambulance Voice communication was slow Voice Digital communication to and from ambulances Despatcher

LAS ambitions The new system was intended to mobilize an ambulance in less than LAS ambitions The new system was intended to mobilize an ambulance in less than 1 minute. The system would be the most ambitious of its time. A much more modest system had been planned for the LAS, but this was abandoned when it failed load-testing. No independent audit of the system requirements was carried out.

CAD requirements LAS wanted a one-phase delivery LAS decided that the system should cost CAD requirements LAS wanted a one-phase delivery LAS decided that the system should cost £ 1, 500, 000 LAS decided that the system would take 6 months to implement (though a project of this scale would usually take 18 months) These requirements were not based on any analysis of the design. They appear to be arbitrary.

Asking for tenders In early 1991, LAS publicized the requirements and asked for bids Asking for tenders In early 1991, LAS publicized the requirements and asked for bids Many potential suppliers expressed doubts that the project could be finished on time with the required budget LAS replied that the timetable was not negotiable

Bids Many potential suppliers submitted bids for the project Most of the bids required Bids Many potential suppliers submitted bids for the project Most of the bids required more time and/or money The bids were evaluated by LAS staff who had no experience with information technology

Selecting a contractor Only one bid was under £ 1, 500, 000 and promised Selecting a contractor Only one bid was under £ 1, 500, 000 and promised an implementation system in 6 months. This bid was selected. The winning bid was from Systems Options Ltd (SO), a small software house with no experience in safetycritical software. SO had never managed a large project

The Contract LAS signed a contract with SO in September 1991. The system was The Contract LAS signed a contract with SO in September 1991. The system was supposed to go on-line on January 8 th, 1992. The contract did not specify who would act as project manager or who would be responsible for quality assurance. No acceptance criteria was defined

Developing the system Suppliers failed to meet deadlines SO initially handled the project management, Developing the system Suppliers failed to meet deadlines SO initially handled the project management, but this shifted to LAS as the project proceeded No independent QA or audit was performed; LAS intended to save money by leaving QA to the suppliers

Problem tracking There was a formal procedure for reporting, analyzing and fixing bugs but… Problem tracking There was a formal procedure for reporting, analyzing and fixing bugs but… this was often skipped so that the software could be changed quickly to satisfy users

Training problems Users were trained long before the system was on-line. The training was Training problems Users were trained long before the system was on-line. The training was often out of date or forgotten by the time the system was available Users were only trained for their part of the system

Partial deployment The complete system was not ready by Jan 8; systems was deployed Partial deployment The complete system was not ready by Jan 8; systems was deployed in pieces Bugs encountered System needed perfect vehicle information Every 53 rd vehicle was unavailable Workstations froze often (Windows 3. 0) Vehicle allocation could not be overridden Sending the wrong vehicle

Expected to fail Interacting with the system was often awkward and frustrating The LAS Expected to fail Interacting with the system was often awkward and frustrating The LAS Staff had little confidence in the system

No testing of the full system was ever done Nobody ever tested to see No testing of the full system was ever done Nobody ever tested to see if radio system could handle traffic Management did not know what resources were required to maintain service; the CAD system was supposed to give this information

Failure 1 On October 26 th the LAS management decided to switch to the Failure 1 On October 26 th the LAS management decided to switch to the full CAD system. This decision was made even though the system was never tested there were outstanding bugs which were considered ‘severe’

Failure 1 Initially the system worked; there were some errors but the staff were Failure 1 Initially the system worked; there were some errors but the staff were able to correct them As the load increased the system response time decreased and the ambulance location data became less and less reliable

Feedback problems Crew frustration Longer waits for ambulance Bad data Bad allocation Fewer available Feedback problems Crew frustration Longer waits for ambulance Bad data Bad allocation Fewer available vehicles More calls

Design errors Some of the design decisions made it harder to recover from errors Design errors Some of the design decisions made it harder to recover from errors Allocators could only get info on ambulances by reserving an ambulance Control room layout made it hard for operators to communicate System could not handle operators overriding computer decisions

Consequences At the height of the accident emergency calls were ringing for 10 minutes Consequences At the height of the accident emergency calls were ringing for 10 minutes before being answered Some calls were lost because the list of calls was too big for the terminals 80% of ambulances took more than 15 minutes to respond. (Average was 67%).

Consequences cont. The media reported that patients died because of the failure. A coroner Consequences cont. The media reported that patients died because of the failure. A coroner later concluded that this was false.

Failure 2 After the first failure LAS went back to the semi-automated system in Failure 2 After the first failure LAS went back to the semi-automated system in use before October 26 th. On November 4 th the system froze The cause was a server that had run out of memory

Memory leak The server software had been changed 3 weeks before. This change introduced Memory leak The server software had been changed 3 weeks before. This change introduced a small memory leak. The server had been running out of memory ever since

Backup system There was a backup server, but it was only designed to work Backup system There was a backup server, but it was only designed to work in the full CAD system

Consequences At the time of the 2 nd failure the load was light enough Consequences At the time of the 2 nd failure the load was light enough that the staff recovered all the information lost in the crash. No calls were missed. LAS went back to the original paper system

Next class ISO 12207 - Software life cycle processes Would standards have prevented the Next class ISO 12207 - Software life cycle processes Would standards have prevented the LAS failure? Are standards worth it?