The Therac-25 case is complex and multi-layered enough to require more than a simple once over to understand. There are multiple actors, some of them representing the same entity at different times. There are closely interwoven networks of action and reaction guided by multiple and mixed motives, where the real state of the information available to an actor at any one time is unclear. Show This is not, however, simply the uniqueness of the Therac-25 case, it is a property of all cases if they are studied closely enough. Finally, it is a property of the real life of technology in use. We provide here some exercises to help students grapple with the complexity of these situations. But first a comment on simple answers. We recommend you read the section on pitfalls before teaching this case. It outlines ways to approach this case that bring only a shallow level of understanding to the complexities. In the Therac-25 case, one of these pitfalls (single causation) leads to the tendency to fix each error one discovers with a local "patch." This usually increases the complexity of the system, provides false confidence in its safety, and does not address the design issues that led to the existence of the error in the first place. This is clearly the kind of thinking that AECL indulged in during its initial reactions to the early accidents. We recommend you help your students avoid it as they approach this case. Bill Frey and Jose Cruz provide many other exercises associated with cases from computingcases.org. Exercises - Table of Contents1. Analyzing Therac-25 2.Computer Control Choices Exercise 3. Tracing the Coding Errors to the Hazards 4. Software Safety Myths 5. Designing a Reporting System 6. Role Playing the Case Analyzing Therac-25This exercise uses a modified version of Robert Collins and Keith Miller's ParaMedic Ethics procedure. Collins and Miller recommend a procedure to use in evaluating a decision. We are not here evaluating any particular decision, but we can use their method to help us understand the obligations, rights, costs, and benefits for each of the parties in the system. This exercise will require students to read the case on the website with some knowledge of the method they will be using, so they can take relevant notes as they read. Thus, the best approach to this exercise requires introducing the modified paramedic ethics procedure in one class, assigning the exercise and case as homework, and then spending the next class period discussing student’s conclusions. Alternatively, students might be assigned the case to read for homework and then introduced to the method of analysis in the subsequent class. If this approach is taken, be sure to have the case available in class (either on a computer with a projector or in printouts for each student) to aid recall. There are several approaches to having students read the case for this exercise. You might have them read all the case section but exclude the accident reports. Once students have gone through the paramedic procedure based on their knowledge, you might then introduce them to one or more of the accidents. Does this new information change their assessments of the case? You might give some students partial information (e.g. just the background sections) and others more extensive information. This too is likely to produce differences in their analyses of the case. Alternatively, you might use small sections of the case (e.g. just the background) early in a course and add information about the case as the course progresses. Each of these approaches are likely to produce differences in the way the case is analyzed by students. These differences help make it clear how important a comprehensive view of a case is. Our modified paramedic ethic procedure consists of 4 phases. The basic analysis consists of phases 1 and 2, in which the basic relationships among the important stakeholders in the case are outlined. The phases that construct and judge the various alternative scenarios can be done as many times as you wish for each set of actions you think are important. To make this go faster, you might assign groups to construct and present their analysis of the duties and rights of each of the main stakeholders presented in the case: AECL, FDA, hospitals, operators, and patients. Gather data
Analyze the data
Construct an Alternative Scenario
Judge the Alternative
Reference
Computer Control Choices ExerciseEXERCISE: Use the range of human-computer control possibilities (on p. 448in Leveson) to locate Therac-25 control levels. Recommend and argue for a change in level. What would be required to move a level up? Down? Choosing the Level of Computer Control In her book Safeware: System Safety and Computers, Nancy Leveson lists nine different levels of computer control (taken from Sheridan’s analysis):
After students have explored the case, have them decide at what level the Therac-25 system is targeted. This may initially cause some confusion, since one way of looking at the system is to think that the operator tells the computer what to do and then the computer does it. Point out to them that this is true in the larger sense, but that the computer clearly has sensors and information available to it to allow it to give error messages. What do we know about the level in this control hierarchy at which those error messages are resolved? What levels of computer control is the system using when:
Once you have established levels of computer control the machine is using, ask for suggestions about how one might increase the amount of computer control. What safety issue does this bring up? One of the best ways to analyze the effects of changes in computer control is to have already completed the basic steps in the case analysis (determining stakeholders, duties and rights, opportunities and vulnerabilities). References
Tracing the Coding Errors to the HazardsThe Leveson excerpts section of the resources reprints explanations from Nancy Leveson about each of the two identified coding errors in the system that resulted in overdoses to patients. Have students trace each coding error from the problematic variable or operation (e.g. a comparison) to how this resulted in an overdose.
This exercise might be done as an in-class exercise or as individual homework and then discussed in the class. Software Safety MythsIn her book Safeware: System Safety and Computers (p. 26) Nancy Leveson lists seven myths regarding the safety of software.
After having the class explore the Therac 25 case, ask students to evaluate the truth of each of these statements as they pertain to the case. This can be done either as part of a homework assignment, with class discussion after papers are turned in, or as a class discussion followed by individual papers. Alternatively, you might combine these two approaches and have students turn in a paper and then revise it (or write a short postscript) based on class discussion. Reference
Designing a Reporting SystemA life cycle approach to software requires some way to gather reports in the field of the operation of the software and feed those reports back into maintenance and updating of the software. One of the clear difficulties in the Therac-25 case was the process of getting the right information back from the field to the AECL home office and to other sites and then getting resolutions of the problems communicated back to the sites. In some cases AECL was only notified by lawsuit months after an incident. In other cases, information languished at the home office that might have been useful to sites where the machine was being used. In this exercise, you will ask your class to design a reporting system and to evaluate its impact on the various stakeholders in the case. In her book Safeware: System Safety and Computers (p. 88), Nancy Leveson lists four requirements of a successful reporting system:
Use these requirements to design a reporting system that might help to reduce the risk to patients. Make sure to address all four points requirements in a successful system. This exercise might be done as an in-class exercise or as individual homework and then discussed in the class. A more time consuming but interesting alternative is to have teams from representing various stakeholders (AECL, the hospitals, the patients, the FDA) design their preferred reporting system as homework and then have these systems presented in class on the same day. Class discussion after these presentations might be a general comparison or some sort of a negotiation among the various parties. References
Role Playing the CaseHave students read the case, including the background materials. Do not allow student to read any of the accident reports. Assign particular groups to prepare to defend the viewpoint of each of the participants in the case (AECL, FDA, Hospital, Operator). In class, give each group the description of the two Tyler incidents. Also give to them the explanation of the Tyler code and why it produced Malfunction 54. This is what each participant knew shortly after the Tyler accidents. Allow each group 15 minutes to produce a proposal regarding what should be done. Keep this part of the assignment vague enough to allow them to propose a wide variety of remedies if they desire. Allow each group 3 minutes to propose its remedy and each group 3 minutes to comment after hearing all the proposals. Class discussion can initially center on which proposals are better. Use your knowledge of the case to present the Yakima accidents and ask them which of their proposals would have helped prevent that case. This will allow you to point out the larger issues involved in designing for safety: safety is a system property and not just a property of the software itself. |