Summary: Uber's reports indicate that they are taking improving their safety culture seriously. Their new approach to public road testing seems reasonable in light of current practices. Whether they can achieve an appropriate level of system safety and software quality for the final production vehicles remains an open question -- just as it does for the self-driving car industry in general.
Uber ATG has released a set of materials regarding their
in-development self-driving car technology and testing, including a
NHTSA-style safety report, as well as reports of a safety review in
light of the tragic death in Tempe AZ earlier this year. (See:
https://www.uber.com/info/atg/safety/)
Generally I have refrained from critical analysis of other company
safety reports because this is still something everyone is sorting out.
Anyone putting out a safety report is automatically in the top 10% for
transparency (because about 90% of the companies haven't even released
one yet). So by this metric Uber looks good. In fact, their report has a
lot more detail than we've been seeing in general, so kudos to them for improving transparency. The other companies who haven't even published a report at all should get with the program.
But, Uber has also had a fatality as a result of their on-road test
program. If any company should be put under increased scrutiny for
safety it should be them. I fully acknowledge that many of the critique
points apply to other companies as well, so this is not about whether
they are ahead or behind, but rather how they stand on their own
merits. (And, if anyone at Uber thinks I got something wrong, please
let me know.)
Overall
It seems that Uber's development and deployment plan is generally what
we're seeing
from other companies. They plan to operate on public roads to build a
library of surprises and teach their system how to handle each one
they encounter. They plan to have safety drivers (Mission Specialists)
intervene when the vehicle encounters something it can't handle. As a
result of the fatal mishap they plan to improve safety culture, improve
safety drivers, and do more pre-testing simulation. There is every
reason to believe that at least some other companies were already doing
those things, so this generally puts Uber on a par with where all
companies doing road testing should be.
Clearly the tragic death in Tempe got Uber's attention, as it
should have. Let's hope that other companies pay attention to the lessons learned before there is another fatality.
Doing the math, there should be no fatalities in any reasonable
pre-deployment road test program. That's because there simply won't be enough
miles accumulated in road testing with a small fleet to reach a level at which an average human driver
would be likely to have experienced a fatal accident. (It is not zero
risk, just as everyday driving is not risk free. But a fatality should
be unlikely.)
The Good
- This is perhaps the most thorough set of safety reports yet. We've
been seeing a trend that more recent reports often include areas not
touched on by earlier reports. I hope this results in a competitive
dynamic in which each company wants to raise the bar for safety
transparency. We'll see how this turns out. Uber is certainly doing
their part.
- The materials place significant emphasis on improving safety
culture, including excellent recommendations of good practices from the
external report. Safety culture is essential. I'm glad to see this.
- There are detailed discussions about Mission Specialist roles,
responsibilities, and training. This is important. Supervising
autonomy is a difficult, demanding role, and gets more difficult as the
autonomy gets better. Again, I'm glad to see this.
- There is quite a bit about hardware quality for both computer
hardware and vehicle hardware. It is hard to tell how far down the ISO
26262 hardware safety path they are for life critical functions such as
disengaging autonomy for Mission Specialist takeover. They mention some
external technical safety reviews, but none recently. This is a good
start, but more work required here. They say they plan more external reviews, which is good.
- They state concrete goals for each of their five safety principles. This is also good.
Jury Still Out on Fully Autonomous Operation System Safety:
- The section on system safety is for the most part aspirational. Even
the section that is not forward looking is mostly about plans, not
current capabilities. This is consistent with currently using Mission
Specialists to ensure testing safety. In other words, assuming the Mission Specialist can avoid mishaps, and the vehicle always responds to human driver takeover, this isn't a problem yet. So we'll have to wait to see how this turns out.
- The external review document concentrated on safety culture
and road testing supervision. That would be consistent with a conclusion
that the fatality root causes were poor safety culture and ineffective
road testing supervision. (Certainly it would be no surprise if this hypothetical analysis were true, but
we'll see what the final NTSB report says to know for sure.)
- In general, we have no idea how they plan to prove that their vehicles
are safe to deploy other than by driving around until they feel they
have sufficiently infrequent road failures. Simulation will improve
quality before they do road testing, but they are counting on road
testing to find the problems. To be clear, road testing and that type of
simulation can help, but I don't believe they are enough. (This is the
same as for many other developers, so this is not a particular criticism
of Uber.)
- Uber says they are working on a safety case, perhaps using a GSN-based approach. This is an excellent idea. But I
don't see anything that looks like a formal safety case in these
documents. Hopefully we'll get to see something like that down the road.
Software Quality and Software Safety:
- The software development process shown on page 47 of the report
emphasizes fixing bugs found in testing. I don't know of any application
domain in which that alone will actually get you acceptably safe
life-critical software. Again, for now their primary safety
strategy for testing is Mission Specialists, so this is an issue for the
future. Maybe we'll find out if they are doing more in a later edition
of this report.
- The information on software quality and software development process
description is a bit skimpy in general. It is difficult to tell if that
is a reflection of their process or they just didn't want to talk about
it. For example, there is a box in their process diagrams on page 47
that says "peer review" with no description as to which of the many
well-known review techniques they are using, whether they review
everything, etc. There are no boxes in their software process for
requirements, architecture, and design. There isn't an SQA function described (Software Quality Assurance, which deals with development process quality monitoring). For Agile fans, there
are plenty of boxes missing from whichever methodology you like. The
point is that this is an incomplete software process model compared to what I'd expect to see for life-critical software. The question
is whether the pieces are there and not drawn. Again, there is no
other industry in which the approach shown would be sufficient or acceptable for
creating life-critical software. It is possible there is more to their
process than they are revealing, or they have some plan to address this before they remove their Mission Specialists from the vehicles.
- Perception (having the system recognize objects, obstacles, and so on) is notoriously difficult to get right, and probably
the hardest problem out of many difficult problems to make self
driving cars safe. They talk about how they use perception, but not how they
plan to validate it, other than via road testing and possibly some aspects of simulating scenarios observed during road testing. But then again, other
developers don't say much about this either.
- It's easy to believe that at least some other organizations are
following similar software approaches and will face the same
challenges. Again, because they currently have safety drivers these are
forward-looking issues that are not the primary concerns for Uber road
testing safety in the near term.
- It is worth noting that they plan to have a secondary fail-over
computer that they say will be developed at least taking into account
software quality and safety standards such as ISO 26262 and MISRA C. (Safety Report Page 35.) But they don't seem to say if this is what they are doing for their basic control software that controls normal operation. Again, perhaps there is more to this they haven't revealed.
Is It Enough?
Overall the reports seem to put them on a par with other
developers in terms of road testing safety. Whether they operate safely
on public roads will largely depend upon maintaining their safety
culture and Mission Specialist proficiency. I'd suggest an independent
monitor within the organization to make sure that happens.
What I'd Like to See
There are a number of things I'd like to see from Uber to help in regaining public trust. (These same recommendations go for all the other companies doing public road testing.)
- Uber should issue periodic report cards in which they tell us about adopting the
recommendations in their various reports and their safety plans in general. Are they staying on track? Did the safety culture initiative really work? Are they following their Mission Specialist procedures?
- I'd like to see metrics that track the effectiveness of Mission Specialists. Nobody is perfect, but I'd be happier having data about how often they get distracted to see whether the break schedule and monitoring are working as intended. This should be something all companies do, since in the end they are putting the public at risk. The effectiveness of Mission Specialists who have been assigned a difficult job is their stated way to mitigate that risk -- but we have no insight as to whether that approach is really working until a crash is in the news.
- They have promised safety metrics that are better than disengagements
and miles driven. That's a great idea. We'll have to see how that turns
out. (They sponsored a RAND report on this topic that was recently
released. That will have to be the topic of another post.)
- We should track whether they establish their external safety advisory board -- and whether it has appropriate autonomy and software safety technical expertise as well as areas such as safety culture and human/machine interface.
- They should also have an independent internal monitor making sure their safety-relevant operational and design processes are being followed. This seems in line with their plans.
- They need a much stronger story about how they plan to ensure
software safety and system safety when they remove their Mission Specialists from the
vehicle downstream. Hopefully they'll make public a high level version
of the safety case and have it externally evaluated.
- I hope that they work with PennDOT to comply with PA AV testing
safety guidelines before resuming operation in Pittsburgh, where I live. From the materials I've seen that should be straightforward,
but they should still do it. As of right now, they'd only be the second company to do so.
Dr. Philip Koopman is a faculty member at Carnegie Mellon University. He is an internationally recognized expert in the area of self-driving car safety. He is also Co-Founder of Edge Case Research, which provides products and services relating to autonomy safety.
koopman@cmu.edu
