Computer-Based System Safety Essential Reading List

Here is a quick start resource guide for computer-based system safety literacy. If you work on computer-based system safety and you aren't familiar with the below case studies, you really need to read them.  (Not just safety engineers -- everyone!)

Patriot Missile [Wikipedia]


Essential Case Studies: Because those who have not read history are doomed to repeat it.
Additional Case Studies:
Recommended Supplemental Materials
Other Case Studies: (Still important, and should be read by anyone digging deep into safety. But less specifically related to computer-based system risks.)
Other Related Topics:
Resources for deeper engagement:
  • Systems Engineering Body of Knowledge on Safety Engineering (SEBoK)
  • NASA Safety library (index | Safety Guidebook)
  • NASA Real System Failure story collection by Kevin Driscoll  (Home | slides)
  • FAA System Safety Handbook (FAA)
  • USAF System Safety Handbook (USAF)
  • List of NHTSA software-related automotive recalls (Blog)
  • Safety of Work podcast (Rae & Provan) (Podcast)

Advanced Specialty Topics/Research:
    • Computer System Diversity, Independence, and Bootstrapping Safety (Lorenzo Strigini)
    • Radiation-induced upsets. (YouTube, OK for background on the phenomenon, but has an inaccurate summary of Toyota UA findings)
    • Knight, "Safety Standards -- a New Approach" about the equity issues with safety standards that need to be resolved: https://scsc.uk/r126/1:1

    NOTE: While Wikipedia is not always an authoritative source, for these sorts of events it tends to present useful summary descriptions.

    If you think something important is missing, let me know!
    Last update 9/10/2023

    Comments

    1. Steve BranamApril 23, 2019 at 11:19 AM

      Leveson, "Engineering A Safer World", PDF download from https://mitpress.mit.edu/books/engineering-safer-world.

      Neumann, "Computer-Related Risks" (based on the Risks Digest archives as of 1994).

      ReplyDelete
    2. UnknownJanuary 2, 2022 at 10:38 AM

      Excellent list. While I've read most of these, seeing this list there's a few I want to add to my reading list.

      ReplyDelete
    3. Ulrik EJanuary 3, 2022 at 4:41 AM

      The Eschede train accident should be included under Other Mishap Case Studies, not only because it highlights the importance of proper maintenance procedures, but also the legal aftermath where officials and engineers were charged with manslaughter.
      https://en.wikipedia.org/wiki/Eschede_derailment

      ReplyDelete

    Post a Comment

    All comments are moderated by a human. While it is always nice to see "I like this" comments, only comments that contribute substantively to the discussion will be approved for posting.

    Popular posts from this blog

    Book: How Safe is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety

    Image
    How Safe Is Safe Enough for Autonomous Vehicles?  The Book The most pressing question regarding autonomous vehicles is: will they be safe enough? The usual metric of "at least as safe as a human driver" is more complex than it might seem. Which human driver, under what conditions? And are fewer total fatalities OK even if it means more pedestrians die? Who gets to decide what safe enough really means when billions of dollars are on the line? And how will anyone really know the outcome will be as safe as it needs to be when the technology initially deploys without a safety driver? This book is written by an internationally known expert with more than 25 years of experience in self-driving car safety. It covers terminology, autonomous vehicle (AV) safety challenges, risk acceptance frameworks, what people mean by "safe," setting an acceptable safety goal, measuring safety, safety cases, safety performance indicators, deciding when to deploy, and ethical AV deployment.
    Read more

    The UL 4600 Guidebook

    Image
    The UL 4600 Guidebook: What to Include in an Autonomous Vehicle Safety Case ANSI/UL 4600 is the most comprehensive standard for highly automated vehicle safety, applying to any vehicle in which a human driver can take their eyes off the road. It provides a way to check the completeness and correctness of a safety case that spans a broad range of concerns related to safety, including design, deployment, and lifecycle support. There is a special emphasis on computer hardware and software, as well as operational concepts and interaction with other road users. While other relevant standards can and should be used as well, UL 4600 provides an umbrella to make sure things don’t get missed for assuring safety. This book, written by the author of the original UL 4600 standard proposal, serves as a high-level guided tour. Early chapters provide historical context, a description of the distinctive UL 4600 prompt element approach, a discussion of key terms, and how a safety case works in the cont
    Read more

    A Liability Approach for Automated Vehicles in Three Parts

    Image
    I'm delighted that months of collaboration with co-author and law professor William Widen have resulted in a trio of papers that together provide a framework for resolving the vast majority of automated vehicle legal questions. Product liability will still be a thing, but that should be reserved for its more usual role, and not be the sole means of recourse for everyday Computer Driver road mishaps that will displace the everyday Human Driver road mishaps. A tort law approach based on negligence is a far better fit and will require far less disruption to existing legal and regulatory systems while providing a fair basis for compensation for anyone harmed by this novel technology. The three parts are in three separate SSRN papers intended to be used as a set, although each paper is self-contained. Below are very simplified summaries to give an overview. Slides for a talk summarizing these ideas are here:  https://users.ece.cmu.edu/~koopman/lectures/L134_SharedResponsibilitySafety.pd
    Read more