Sunday, January 8, 2023

The Tesla Autopilot Crashes Just Keep Coming

Picture from a Tesla AP-related crash just before impact into a disabled vehicle:


(Video here on twitter) Tesla autopilot crashes are still happening when drivers (apparently) succumb to automation complacency. It seems they've just stopped being news.

The above picture is from a Tesla camera a fraction of a second before impact. (Somehow it seems there was no injury.) The Tesla was said to have initiated AEB (and disabled AP) about two seconds before impact. The video shows clear sightline to the disabled vehicle for at least 5 seconds, but the driver apparently did not react.

Tesla fans can blame the driver all they want -- but that won't stop the next similar crash from happening. Pontificating about personal responsibility and that the driver should have known better won't change things either. And we're far, far past the point where "education" is going to move the needle on this issue.

It's time to get serious about:
- Requiring effective driver monitoring
- Addressing the very real #autonowashing problem that has so many users of these features thinking their cars really drive themselves.
- Requiring vehicle automation features to account for reasonably foreseeable misuse (you might fix the vehicle, or you might fix the driver, or more likely fix both, but casting blame accomplishes nothing)

The deeper issue here is pretending that autopilot-type systems involve humans who think they are driving. The car is driving and the humans are along for the ride, no matter what disclaimers are in the driver manual and/or warnings -- unless the vehicle designers can show they have a driver monitoring system and engagement model that provide real-world results.

The reality is that these are not "driver assistance" systems. They are automated vehicles with a highly problematic approach to safety. This goes for all companies. Tesla is simply the most egregious due to poor driver monitoring quality and scale of deployed fleet. As human-supervised automated driving gets more functionality the safety problem will just keep getting worse.

Source on twitter: https://twitter.com/greentheonly/status/1607475055713214464?ref_src=twsrc%5Etfw from Dec 26th: contains video of impact. No injury apparent to the person in the video, but it was a very close thing. Also a screenshot of the vehicle log showing AEB engaged 2 seconds before impact. https://twitter.com/greentheonly/status/1609271955383029763?ref_src=twsrc%5Etfw
For those saying "but Teslas are safer overall" that statement does not stem from any credible data I've ever seen: https://safeautonomy.blogspot.com/2022/12/take-tesla-safety-claims-with-about.html
on No comments:
Labels:

Monday, December 26, 2022

Computer-Based System Safety Essential Reading List

 Don't miss my permanent page on this blog! 

Here is the link:  Computer-Based System Safety Essential Reading List

on No comments:

Wednesday, December 21, 2022

Holiday AV Safety Video Viewing

Daily video & reading suggestions for the holiday season for those into autonomous vehicle safety. Primarily based on new materials from 2022 that you might have missed.

on No comments:
Labels:

Friday, December 16, 2022

I have to get out NOW from my autonomous vehicle: urgent egress and passenger overrides

What if you need to get out of the vehicle RIGHT NOW in a robotaxi -- is that allowed? What are the implications?

Woman looking out a car window.


In any automated system there will be times when an occupant wants to override the automation, and especially when they want to exit a moving automated vehicle. Reasons might include: wanting to re-open transit vehicle doors if a passenger was unable to exit in time at their stop; an attack of claustrophobia; wanting to get away from another passenger due to personal safety concerns; or even needing to escape a cabin fire. Some egress requests might constitute misuse or abuse, such as stopping a vehicle to intentionally block traffic, or intentionally accessing an off-limits area such as a bridge with no pedestrian infrastructure.

Creating a complete list of all possible motivations is difficult, and weighing the merits of all such egress attempts in advance seems intractable. Nonetheless, there are times when a passenger desire to exit a moving vehicle should be honored, although the vehicle should likely at least stop before permitting an exit.

In still other situations passengers might want to force an otherwise stopped vehicle to move. One reason might be fear for personal safety if threatened by malicious actors while stopped at a traffic light. Another reason might be overriding a police stop if the vehicle occupant suspects a stopping officer is instead a criminal imposter, at least until legitimacy of the police stop can be confirmed via contact with an emergency dispatcher.[1]

Another special situation is one in which a passenger has a compelling reason to order an AV to operate outside its ODD or with degraded equipment in an emergency, even if doing so will result in a reduced safety margin. For example, an AV might be programmed not to drive through heavy smoke, but doing so might be required to escape a burning town in a wildfire situation. The AV occupant might want to take the chance of driving rather than remaining in the burning town.[2]

Human drivers have the authority to deal with these situations so long as they are willing to accept the responsibility. Do you start driving when someone is trying to forcibly break into your car at a traffic light even if you might injure that malicious actor by doing so? The choice – and responsibility for consequences – falls upon the person driving a manually driven car.

The question is: to what degree should an AV support operator overrides of safety-relevant behaviors? A complication is that there might not be a responsible individual in a vehicle to exercise control. What if a passenger is allowed to override some behaviors of the vehicle, but that passenger is impaired, or not capable of exercising mature judgment? Should an 8 year old riding solo be able to command vehicle safety overrides?[3]

Answers as to how much control a passenger should have over AV operation will depend on how stringent qualifications are for a passenger to be capable of mature decision making. It is easy to say there must be one qualified driver if there are any passengers in an AV and that manual controls must be available if needed. However, requiring a qualified driver undermines the potential benefits that AVs might provide for those who are not capable of driving or should not be driving at a particular time.

If other than unimpaired licensed drivers are permitted to override AV behaviors, there will be difficult tradeoffs as to what overrides might be permitted. Likely an 8 year old child should not be permitted to exit a school vehicle in the middle of a highway to avoid going to school. On the other hand, a 14 year old[4] might be considered mature enough to demand an emergency stop if the vehicle tries to drive into flood waters, or initiate an emergency exit with their younger sibling if the cabin fills with smoke from a vehicle battery fire.

Even if a passenger is an adult licensed to drive, should that adult be permitted to override vehicle behavior if drunk or otherwise impaired? If not, should the vehicle disable override capability if the passenger is drunk? Or should it be illegal to enter an AV with override capability when drunk?

While it can be an interesting exercise to conjure extreme situations, the issue of passenger overrides and egress can also be as simple as a passenger saying “I want to get out now” when the vehicle is stopped at a red traffic light but not at the end of the scheduled trip. Should the passenger be able to unlock doors and exit? Or should the passenger be kept locked inside the vehicle until the end of the trip? Should there be a workaround available such as changing the destination? If so, should the passenger have permission to do this if some authority figure such as a parent input the original destination? Where should the threshold be drawn at which such a passenger request is denied both in context (speeding down a highway vs. stopped) or passenger maturity (a passenger one day before turning 18 years old but with no driver’s license vs. grade school age child)?

There is the possibility that remote operators will need to mediate requests for overriding AV behavior either routinely or if there is doubt as to the competence of passengers to make reasonable decisions. However, any such remote operators can be expensive, will have problems scaling, and might result in wait times long enough to impair safety by delaying decisions in urgent situations.[5]

For AVs to be deployed at scale, designers will need to decide how much authority passengers have to override vehicle behavior, and whether emergency manual vehicle controls will be required even in vehicles that are intended to be completely automated. There will be no perfect policy choice, but not setting a consistent policy is also a policy choice.

[1] Yes, this is a thing. Report of an accused police imposter pulling over a van full of legitimate police detectives: https://www.youtube.com/watch?v=ogGBwrrkKY4

[2] This consideration has become especially relevant for residents of California. See: https://www.insideedition.com/how-drive-through-fire-48422

[3] One might say that no 8 year old should ride in an AV solo. But if that is the case, what exactly is the cut-off age? Some public high school systems rely on public mass transit instead of dedicated school buses, so any AV public transit vehicle will have under-age ridership. Is training or perhaps even a “rider license” required to ride in an AV and use the override controls? This topic gets complex quickly.

[4] Some states issue driver licenses to 14 year-olds in special cases. Would such a driver license be required in this case? See: https://www.thedrive.com/news/39184/americas-rarest-drivers-license-lets-14-year-olds-hit-the-road-legally

[5] The usual solution proposed is remote customer service operators that intervene when needed. Those proposing that passengers need have no control because remote operators can solve all safety problems need to spend more time waiting in customer service phone waiting queues. An additional consideration is the likely disruption to emergency response services during a natural disaster that will also require simultaneous attention to numerous AV passenger distress situations. New Year’s Eve screening of requests from potentially drunk passengers will also be challenging.

Read more »
on No comments:
Labels:

Wednesday, December 7, 2022

SCSC Talk: Bootstrapping Safety Assurance

Bootstrapping Safety Assurance

Abstract:
The expense and general impracticability of doing enough real-world testing to demonstrate safety for autonomous systems motivates finding some sort of shortcut. A bootstrapped testing approach is often proposed, using evidence from initial mishap-free testing to argue that continued testing is safe enough. In this talk I'll explain why pure bootstrapping based on testing exposure as well as arguments involving "probably perfect" bootstrapping expose public road users to undue risk. Moreover, phased deployments often used to argue safe update release have the same problem. An approach that bootstraps on the safety case rather than on vehicle testing is proposed as a potentially better alternative. While the examples given involve autonomous ground vehicles, the principles involved apply to any argument that safety will be demonstrated via a bootstrap testing process.

This talk was recorded as part of the SCSC Future of Testing for Safety-Critical Systems seminar on Dec. 1, 2022.
Talks and videos are available here (access with paid annual club membership):  https://scsc.uk/e966prog

Free public-access copy of slides here: 

https://users.ece.cmu.edu/~koopman/lectures/L133_SCSC_Bootstrapping.pdf




on No comments:
Labels: , ,

Friday, December 2, 2022

Blaming the autonomous vehicle computer as a regulatory strategy

The AV industry has been successfully pursuing state regulations to blame the computer for any crashes by saying that the Automated Driving System (the computer) is considered to be the driver of any AV operating on public roads. That way there is no person at fault for any harm to road users. Yes, really, that is what is going on.[1]

Person pointing a finger at a computer

The general AV industry tactic when lobbying for such rules is to argue that when fully automated driving is engaged the “driver” is the driving computer (the ADS). Any remote safety supervisor is just there to lend a hand. In some states a remote human support team member need not have an appropriate driver license, because it is said that the ADS that is the driver. Superficially this seems to make sense. After all, if you are a passenger who has paid for a retail robotaxi ride and the AV breaks a traffic law due to some flaw in the design, you as the passenger should not be the one to receive a ticket or go to jail.

But the tricky bit is that ADS computers are not afforded the legal status of being a “person” – nor should they be.[2] Corporations are held to be fictitious people in some legal circumstances, but a piece of equipment itself is not even a fictitious person.[3]

If a software defect or improper machine learning training procedures result in AV behavior that would count as criminally reckless driving if a human were driving, what happens for an AV? Perhaps nothing. If the ADS is the “driver” then there is nobody to put on trial or throw into jail. If you take away the driver’s license for the ADS, does it get its license back with the next software update?[4] Where are the repercussions for an ADS being a bad actor? Where are the consequences?

Blaming the ADS computer for a bad outcome removes a substantial amount of deterrence due to negative consequences because the ADS does not fear being harmed, destroyed, locked up in jail, fined, or having its driver’s license revoked. It does not feel anything at all.

A related tactic is to blame the “operator” or “owner” for any crash. In the early days of AV technology these roles tended to be either the technology developer or a support contractor, but that will change over time. Contractors perform testing operations for AV developers. Individual vehicle owners are operators for some AV technology road tests. Other AV operators might work through a transportation network service. Someone might buy an AV in the manner of a rental condo and let it run as a robotaxi while they sleep.

Imagine an arrangement in which an investor buys a share in a group of robotaxis as might be done for a timeshare condo. A coordinator lines up independent contractors to manage investment money, negotiate vehicle purchases, arrange maintenance contracts, and participate in a ride-hailing network. Each AV is the sole asset of a series LLC to act as a liability firewall between vehicles. The initial investor later sells their partial ownership shares to an investment bank. The investment bank puts those shares into a basket of AV ownership shares. Various municipal retirement funds buy shares of the basket. At this point, who owns the AV has gotten pretty complicated, and there is no substantive accountability link between the AV “owner” and its operation beyond the value of the shares.

Then a change to the underlying vehicle (which was not sold as an AV platform originally, but rather was adapted by an upfitter contractor) impairs functionality of the aftermarket add-on ADS manufactured by a company that is no longer in business. If there is a crash who is the “operator?” Who is the “owner?” Who should pay compensation for any harm done by the AV? If the resultant ADS behavior qualifies as criminally negligent reckless driving, who should go to jail? If the answer is that nobody goes to jail and that only the state minimum insurance of, say, $25K pays out, what is the incentive to ensure that such an arrangement is acceptably safe so long as the insurance is affordable compared to the profits being made?

While the usual reply to concerns about accountability is that insurance will take care of things, recall that we have taken some passes at discussing insurance and risk management can be insufficient incentive to ensure acceptable safety, especially when it only meets a low state minimum insurance requirement[5] originally set for human drivers that have skin in the game for any crashes.

[1] For a compilation of US state laws and legislative hearing materials see:        https://safeautonomy.blogspot.com/2022/02/kansas-av-regulation-bill-hearings.html

[2] Despite occasional hype to the contrary, machine learning-based systems are nowhere near achieving sentience, let alone being reasonably qualified to be a “person.”

[3] I am not a lawyer (IANAL/TINLA), so this is a lay understanding of the rules that apply and nothing in this should be considered as legal advice.

[4] In several states an ADS is automatically granted a driver’s license even though it is not a person. It might not even be possible to take that license away.

[5] IIHS/HLDI keeps a list of autonomous vehicle laws including required insurance minimums. The $1M to $5M numbers fall short of the $12M statistical value of human life, and are typically per incident (so multiple victims split that maximum). In other states the normal state insurance requirement can apply, which can be something like a maximum of $50,000 per incident and might permit self-insurance by the AV company, such as is the case in Kansas: https://insurance.kansas.gov/auto-insurance/ This insurance maximum payout requirement is less than the cost of a typical AV. In practice it might be the case that victims are limited to recovering insurance plus the scrap value of whatever is left of the AV after a crash, with everyone else being judgement-proof.

Read more »
on No comments:
Labels: ,

Wednesday, November 30, 2022

The UL 4600 Guidebook


The UL 4600 Guidebook:
What to Include in an Autonomous Vehicle Safety Case

Book cover

ANSI/UL 4600 is the most comprehensive standard for highly automated vehicle safety, applying to any vehicle in which a human driver can take their eyes off the road. It provides a way to check the completeness and correctness of a safety case that spans a broad range of concerns related to safety, including design, deployment, and lifecycle support. There is a special emphasis on computer hardware and software, as well as operational concepts and interaction with other road users. While other relevant standards can and should be used as well, UL 4600 provides an umbrella to make sure things don’t get missed for assuring safety.

This book, written by the author of the original UL 4600 standard proposal, serves as a high-level guided tour. Early chapters provide historical context, a description of the distinctive UL 4600 prompt element approach, a discussion of key terms, and how a safety case works in the context of the standard. Then comes a chapter-by-chapter tour of UL 4600, explaining overall concepts and how all the pieces fit together for each area covered by the standard, from safety cases to hazard analysis to assessment. This book will help technical readers prepare for diving into the nitty gritty of the standard, as well as provide a more accessible discussion for those who want to understand what UL 4600 covers at a higher level. The last chapter provides pointers to further information, including how you can view the current version of UL 4600 for free.

This is a comparatively short (about 100 pages of main content) trade paperback (6"x9") discussion of a much longer, fairly complex standard. So think of it as a tour guidebook and not a textbook.

Currently available for purchase from Amazon, with international distribution via their print-on-demand network. (See country-specific distribution list below.)

eBook available from Smashwords: https://www.smashwords.com/profile/view/pkoopman

Available from Barnes & Noble and some US and UK book distributors: https://www.barnesandnoble.com/s/philip%20koopman

Media coverage and bonus content:

Chapters:

  1. Introduction
  2. Overview and applicability of UL 4600
  3. Requirements and prompt elements
  4. Terminology
  5. The safety case
  6. Hazards and risks
  7. Interaction with people and road users
  8. Autonomy functions and support
  9. Software & system engineering process
  10. Dependability
  11. Data and networking
  12. Verification, validation, and test
  13. Tools, COTS, and legacy qualification
  14. Lifecycle concerns
  15. Maintenance
  16. Safety Performance Indicators
  17. Assessment
  18. Wrap-up
138 pages.

Koopman, P., The UL 4600 Guidebook: What to Includes in an Autonomous Vehicle Safety Case, November 2022.
ISBN: 9798365303065  Trade Paperback
ISBN: 9798365303249  Hardcover   (available only in marketplaces supported by Amazon)
ASIN: B0BNLVC22J  Kindle ebook


For those asking about distribution -- it is served by the Amazon publishing network. Expanded distribution is selected, so other distributors might pick it up in 6-8 weeks to serve additional countries (e.g., India) or non-Amazon booksellers, especially in US and UK. How that goes is beyond my control, but in principle a bookstore anywhere should be able to order it by about mid-January 2023. Alternately, you can order it direct from Amazon in the closest one of these countries for international delivery: US, UK, DE, FR, ES, IT, NL, PL, SE, JP, CA, AU.


Your local bookstore should also be able to order it through their US or UK distributor. starting in mid-January.

If you are not in a listed country:
  • For printed books you can probably order it from a nearby country for international shipment.
  • For Kindle ebook what matters is what country your kindle is registered for, which is not necessarily your physical location.

Read more »
on No comments:
Labels: ,

Friday, November 18, 2022

The effect of AV company business models on safety

The business model and exit plan for an AV company can powerfully incentivize behavior that is at odds with public safety and transparency. This is probably not news regarding any private company, but it is especially a problem for AV safety.

Business meeting at table with laptops

An AV developer with a plan to develop, deploy, and long-term sustain their technology should be incentivized to reach at least some level of safety subject to all the ethical issues discussed already in this chapter. If they do not, they will probably not have a viable long-term business. Arguments for a light regulatory touch often make this argument that companies will act in their own long-term best interest. But what if the business incentive model is optimized for something shorter than the “long-term” outcomes?

Short-term aspects of the business objectives and the business structure itself can add pressure that might tend to erode any commitment to acceptable safety. Factors include at least the following, several of which can interact with each other:

·        Accepting money from traditional venture capital sources can commit a company to a five-year timeline to produce products. Thus far we have seen that five-year timelines are far too aggressive to develop and deploy an AV at scale. Re-planning and raising more funding can lengthen the timeline, but there remains risk that funding incentivizes aggressive milestones to show increased functionality and, in particular, remove safety drivers rather than core efforts on safety. Some companies will likely be better at resisting this pressure than others.

·        A business exit plan of an Initial Public Offering (IPO), going public via a Special Purpose Acquisition Company (SPAC), or being bought out by a competitor historically emphasize perceived progress on functionality rather than safety. If the exit plan is to make safety someone else’s problem post-exit, it is more difficult to justify spending resources on safety rather than functionality until the company goes public.[1]

·        The AV industry as a whole takes an aggressively non-regulatory posture, with that policy approach historically enabled by US DOT.[2] This situation forces little, if any, accountability for safety until crashes happen on public roads. There is a tendency for at least some companies seem to treat safety more as a public relations and risk management function than a substantive safety engineering function. Short-term incentives can align with a dysfunctional approach.

·        Founders of AV companies with a primarily research, consumer software, or other non-automotive background might not appreciate what is involved in safety at scale for such systems. They might earnestly – but incorrectly – believe that when bugs are removed that automatically bestows safety, or otherwise have a limited view of the different factors of safety discussed in chapter 4. They might also earnestly believe some of the incorrect talking points discussed in section 4.10 regarding safety myths promoted by the AV industry.

·        The mind-boggling amount of money at stake and potential winnings for participants in this industry would make it difficult for anyone to stay the course in ensuring safety in the face of rich rewards for expediency and ethical compromise. No matter how pure of spirit and well intentioned.

It is impossible to know the motivations, ethical framework, and sincerity of every important player in the AV industry. Many participants, especially rank and file engineers, are sincere in their desire to build AVs and believe they are helping to build a better, safer world. Regardless of that sincerity, it is important to have checks and balances in place to ensure that those good intentions translate into good outcomes for society.

One has to assume that outcomes will align with incentives. Without checks and balances, dangerous incentives can be expected to lead to dangerous outcomes. Checks and balances need to be a combination of internal corporate controls and government regulatory oversight. A profit incentive is insufficient to ensure acceptable safety, especially if it is associated with a relatively short-term business plan.

[1] Safety theater money spent to impart an aura of safety is a different matter, and spending on this area can bring good return on investment. But we are talking about real safety here. In the absence of a commitment to conform to industry safety standards it can be difficult to tell the difference without a deep dive into company practices and culture.

[2] The official US Department of Transportation policy still in effect at the time of this writing states: “In this document, NHTSA offers a nonregulatory approach to automated vehicle technology safety.” See page ii of:            https://www.nhtsa.gov/document/automated-driving-systems-20-voluntary-guidance

Read more »
on No comments:
Labels: , ,

Sunday, November 13, 2022

Book: How Safe is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety

How Safe Is Safe Enough for Autonomous Vehicles? 
The Book


The most pressing question regarding autonomous vehicles is: will they be safe enough? The usual metric of "at least as safe as a human driver" is more complex than it might seem. Which human driver, under what conditions? And are fewer total fatalities OK even if it means more pedestrians die? Who gets to decide what safe enough really means when billions of dollars are on the line? And how will anyone really know the outcome will be as safe as it needs to be when the technology initially deploys without a safety driver?

This book is written by an internationally known expert with more than 25 years of experience in self-driving car safety. It covers terminology, autonomous vehicle (AV) safety challenges, risk acceptance frameworks, what people mean by "safe," setting an acceptable safety goal, measuring safety, safety cases, safety performance indicators, deciding when to deploy, and ethical AV deployment. The emphasis is not on how to build machine learning based systems, but rather on how to measure whether the result will be acceptably safe for real-world deployment. Written for engineers, policy stakeholders, and technology enthusiasts, this book tells you how to figure out what "safe enough" really means, and provides a framework for knowing that an autonomous vehicle is ready to deploy safely.

Currently available for purchase from Amazon, with international distribution via their print-on-demand network. (See country-specific distribution list below.)

See bottom of this post for e-book information, from sources other than Amazon, as well as other distributors for the printed book.

Media coverage and bonus content:

Chapters:

  1. Introduction
  2. Terminology and challenges
  3. Risk Acceptance Frameworks
  4. What people mean by "safe"
  5. Setting an acceptable safety goal
  6. Measuring safety
  7. Safety cases
  8. Applying SPIs in practice
  9. Deciding when to deploy
  10. Ethical AV deployment
  11. Conclusions
368 pages.
635 footnotes.
On-line clickable link list for the footnotes here: https://users.ece.cmu.edu/~koopman/SafeEnough/

Koopman, P., How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety, September 2022.
ISBN: 9798846251243 Trade Paperback
ISBN: 9798848273397 Hardcover   (available only in marketplaces supported by Amazon)

Also see my other recent book: The UL 4600 Guidebook

For those asking about distribution -- it is served by the Amazon publishing network. Expanded distribution is selected, so other distributors might pick it up in 6-8 weeks to serve additional countries (e.g., India) or non-Amazon booksellers, especially in US and UK. How that goes is beyond my control, but in principle a bookstore anywhere should be able to order it by about mid-November 2022. Alternately, you can order it direct from Amazon in the closest one of these countries for international delivery: US, UK, DE, FR, ES, IT, NL, PL, SE, JP, CA, AU.


You can also buy it from some Amazon country web sites via distributors. A notable example is:

Your local bookstore should also be able to order it through their US or UK distributor.

E-book available from distributors as they pick it up over time: 

Read more »
on No comments:
Labels: , , , , , ,

Friday, November 11, 2022

Shiny vs. Critical Software

Coverage of Lucid software problems (cars bricked; wrong direction of travel) might be written off to growing pains for a new company. But I think this is just yet another story about a deeper industry-wide problem. (The article notes other more established companies have problems too.) This weeks story: https://www.businessinsider.com/electric-vehicle-startup-lucid-struggling-production-reveal-insiders-owners-2022-11

abstract oil and water photo
Shiny and critical software and developer skills
mix as oil and water.

All software is not created equal. For cars I am seeing three types:

  • Shiny Infotainment and other software that provides shiny customer features might be less reliable and still sell cars. There are limits to tolerance for problems, but more forgiveness if the features are shiny enough. Apparently "coding" is enough to build valuable companies, as is slapping "beta" on the label as a pretext for further reducing quality expectations for a product sold retail. Maximizing lines of code per day has made lots of founders rich.
  • Critical deeply embedded control firmware has to be rock solid or you're going to get serious malfunctions. "Coding" without software engineering invariably leads to deeper problems, some of which feature in harm to people. Maximizing lines of code per day at the cost of impaired quality and skipped safety engineering practices has made customers dead.
  • "AI" software based on machine learning, which is being asked to do safety critical work but often without using the foundational skills and processes of the critical software experts.
(You can argue that cloud services are yet another type, but in my experience that divides up into shiny services software and critical infrastructure support software.)

Journalists should differentiate when reporting. If you must call a shiny software problem a "glitch" so be it. But critical software failures are due to *defects* reflective of an important lapse in engineering, not glitches due to being in a hurry to deploy the new hotness.


Companies get in trouble, sometimes very seriously, via several mechanisms:

  • Treating all software the same. Software needs to be thought of as either shiny or critical. They are as oil and water. (Maybe this could be different, but in the world we live in this is the only pragmatic approach.)
  • Treating "software staff" as fungible. Developers for shiny vs. critical software are deeply different. The skill sets, mindset, and work flows are quite different. So is the training required. While some can do both, few can do both well. (This is not about being "smart." It is about being different.) To a first approximation, anyone talking about "coding" is in the shiny software business, especially if they indicate that knowing how to code is equivalent to being a software engineer.
  • Mixed components and features. We see an endless parade of NHTSA recalls for malfunctioning backup cameras. The usual story is that a critical function (backup camera; by definition safety critical per FMVSS) is hosted on a platform optimized for shiny (infotainment display and OS). The only surprise is that car companies persist in thinking that plan will work out well.
  • We're still sorting out how to fit AI software into the mix. A lot of that will end up forcing a choice between the shiny bin or the critical bin, with a human/machine interface suitable to the choice. Making AI software shiny can be a reasonable choice -- but only if we don't pretend shiny AI it is fit for purpose as critical software. (Critical machine learning might be done, but there is a significant gap to overcome in skills, work flows, etc. that the industry is only beginning to wrestle with.)
Cost is pushing companies to mix shiny with critical more than they should. That pressure will continue to generate news stories. Instead of just pretending they mix well, companies should be rethinking their software architectures to maintain separation of technical aspects, staff skills, and cultural aspects in a way that is harmonious. Pretending these differences don't exist will continue to lead to bad outcomes.


on No comments:
Labels: ,

The AV Blame Game

Assigning blame does not make roads safer. Rather, blaming is most commonly used to evade responsibility for mitigating a safety problem.

Two robots arguing after a car crash

The blame game is played by AV companies when they find some reason – any reason will do – for an AV crash that is not the fault of the AV itself. Candidates for blame include the safety driver, drivers of other vehicles, jaywalking pedestrians, and possibly unexpected conditions. A cousin of the blame game is claiming that the AV acted in a lawful manner even if doing so was clearly inappropriate for the situation. At a deeper level, the blame game is an extension of the tactic of blaming human drivers for being imperfect to deflect attention away from operational flaws with AVs.

The reality is that placing blame does not make streets safer. Driving involves a continual stream of social interactions with other drivers in which, hopefully, most drivers follow most of the rules most of the time. Importantly, drivers are expected to compensate for mistakes and any lack of rule following by other drivers to the degree they can.[1]

For every AV crash in which the AV design team insists some other party should be blamed, an essential follow-up question is whether the AV could have done something to avoid the crash, even if that something is not strictly required by the rules of the road. Any generally useful response that might have avoided the crash should be added to the AV behavioral repertoire even if not strictly required by law.

As a hypothetical example, when encountering a wrong-way driver it is likely better for an AV to pull to the side of the road than to continue driving in-lane until impact. This is the case even though the AV has right of way, and might be fully justified by the rules of the road in continuing to drive in its lane right into the impending crash. At worst, pulling to the side of the road reduces the relative impact speed. At best an impact is avoided as the other vehicle continues driving the wrong way in the travel lane. And who knows – it is possible that the AV itself was the vehicle going in the wrong direction due to a mapping error or other issue.[2] Blaming the other vehicle for wrong-way driving post-crash provides cold comfort to the families of the victims.

At a higher level, blame is irrelevant for determining AV safety. The crash rate is what it is, regardless of blame. Consider an AV that has twice as many crashes as human-driven vehicles, but would theoretically be able to prove in a court of law that every single crash was someone else’s fault. Such a perfectly blameless vehicle would nonetheless have a track record of being twice as dangerous as a human-driven vehicle. That type of approach should not be how AV designers claim that they are safe.

[1] As an example, pedestrians are not supposed to cross mid-block, but if they do so vehicles have an obligation to make best efforts to stop to avoid a collision. In states with this rule an AV that does not make a reasonable attempt to stop to avoid hitting a jaywalking pedestrian is failing to abide by the rules of the road.

[2] Yes, AV tests traveling the wrong way is a thing. See: https://qz.com/798092/a-self-driving-uber-car-went-the-wrong-way-on-a-one-way-street-in-pittsburgh/

Also, see a related video here: https://youtu.be/Ao2qssbXDXo

Read more »
on No comments:
Labels:

Saturday, November 5, 2022

Why Car-to-Bike Safety Communications Won't Solve Safety

This nicely thought out article by David Zipper breaks down the problems with C-V2X: in this case using direct wireless messaging connectivity between cars and bikes to warn of potential collisions. https://www.fastcompany.com/90801870/audis-new-technology-is-designed-to-keep-bikers-safe-it-wont


Might the technology help? Sure it might help some, but it will just as surely not be a complete solution. In a complex world it is possible the net outcome will be worse than other approaches (not because of the technology per se, but because of the complexities).

While it is great to see innovation to improve safety we need to be mindful of these pitfalls:

  • Equity issues: Not everyone can afford or will want to burden themselves with a C-V2X transponder. (Not everyone has a high-end cell phone, and not everyone wants one even if they can afford it.) This burdens other road users rather than owners of expensive cars.
  • Potential for risk homeostasis: If the transponders get reliable, will that teach drivers not to look for vulnerable road users if they don't hear a transponder warning?
  • Victim blaming: What if someone says a kid walking to school deserved to die because they should have known to not let the battery run out on their cell phone/transponder? Or that parents should have bought their 7-year-old a smart phone to keep them safe.
  • Allocation of societal resources: There is only so much attention and funding available. Arguably it would be a better net outcome to make roads inherently safer rather than spend those resources on C-V2X.
In short, this sounds like a high-tech version of "pedestrians should wear bright colors so they don't get hit by drivers who are not paying close enough attention." Which often shows up as an "education" message that displaces solving deeper systemic safety issues. (Might wearing a bright yellow rain coat help? Sure, that's what I do. But proposing that to deflect attention from deeper systemic issues is a problem.)

It is important to avoid running afoul of evergreen research advice by Prof. Mary Shaw, which I'll paraphrase as: if your idea only works if everyone in the whole industry adopts it, then you need a different idea.

on No comments:
Labels: , ,
Subscribe to: Posts (Atom)