Road Testing Safety Metrics (Metrics Episode 3)

To build trust, self-driving car companies should be transparent about operational safety metrics for road testing, such as how often the safety driver fails to react to an issue. This won't be perfect, but it ought to be at least as safe as normal human drivers on public roads.

Right now when you see what looks like a self driving car on the road, it’s not really a production self driving car -- it’s a self driving car technology test platform. That usually means that some human someplace in the car remotely, whatever, is keeping an eye on it and making sure things are safe. In turn if you asked the question, “How safe is that car,” you’re not actually asking about the safety of the self driving technology pretty much at all. What you’re asking is whether or not that human is able to properly supervise the safety and do the right thing when something goes wrong. So if you care about the safety of this public on road testing of this maturing technology what you care about is the human safety driver’s performance. To understand that let’s talk about the types of things that safety driver has to do.

The safety driver has to build and maintain situational awareness and know what’s supposed to happen next. The safety driver has to notice something is going wrong and intervene at exactly the right time. While this is happening the safety driver is under some pressure to balance getting operational data with maintaining safety. After all, sitting in the garage with the car turned off doesn’t get the data that they’re out in the public roads to get. As they’re operating the safety driver has to figure out how to react when other drivers do things that are weird, illegal, dangerous, or just plain crazy.  And sometimes it’s the car itself that misbehaves due to a software defect in the self driving technology. When something bad happens, the safety driver has to execute a takeover maneuver and make sure that they don’t make things worse.

Now, that sounds like a lot to do but the fact is that when things are going well it’s an exceptionally boring job. The car just keeps doing the thing it’s supposed to be doing and the driver sits there watching and waiting. Every once in a while something will go wrong and the safety driver has to make sure that not only do they have situational awareness, and that they have an idea of what to do next, but also that they’re not startled into doing the wrong thing. Rolling all that up, supervising an autonomous test platform is exceptionally difficult work and requires the highest of driving skills. And yes, we actually do get to the part where the car has to behave properly, but it has nothing to do with the self driving technology -- it has to do with the emergency override. If the human safety driver wants to override the vehicle then they have to be able to turn off the self driving feature and get control of the vehicle.

It turns out getting that mostly right is straight forward but getting it 100% right takes a lot of care and attention. Taking all this into account, you start realizing that disengagements, which is the measure we use now, is actually the wrong metric. In fact it’s exactly the wrong metric. That's because disengagements aren’t what make you dangerous, the thing that makes you the most dangerous when you’re testing technology is non disengagements: a failure to disengage when you should have. In other words, the hazards that you did not catch with the human safety driver that’s what matters.

Now, measuring something that doesn’t happen is not straightforward but in fact if you want metrics for safety that’s what you have to go after. The starting point isn’t really quite metrics at all but rather processes and what you hear is the more thoughtful companies are starting to emphasize things like good driver qualifications, they have training, they have good reaction times. Driver testing, go out on a test track, make sure the driver can handle surprises and make sure the driver can maintain situational awareness. Yes, the disengagement mechanism has to work. The proverbial big red button needs to work all the time not just most of the time. Fortunately there’s a safety standard ISO 26262 which tells you exactly how to make that kind of system so these companies should be following that standard with their disengagement mechanism.

Now there’s an issue about whether or not training once is enough.  It’s not. You need refreshers at least, and so refresher training is also a good idea. Many of the companies stop with that. But the problem with doing only training is that you think you’re good enough but you don’t know you’re actually good enough unless you take operational data. 

Sure, the driver can pay attention on test day and maintain engagement. But can a safety driver maintain that same peak performance for six months of day in, day out testing that’s really boring because mostly things work pretty well? Humans aren’t perfect, and that means even really good safety drivers aren’t perfect.  That’s just the way it is. So what you really want is not just training but also operational metrics to make sure that the safety drivers are able to be as good as they need to be to reach your safety goal.

Here are some metrics that could work internally for an engineering effort:

• You could measure the attention of the driver throughout a shift. Maybe you’ll find out, as might be expected, that after many hours the driver has trouble maintaining attention. That wouldn’t be a surprise. Some companies are doing two hour shifts. While for active driving that might be a good shift size, there’s lots of data for several decades suggesting that for supervising autonomy you need shorter shifts, maybe only 30 minutes. So do you have data showing that your two hour shifts are okay? Or maybe you need a shorter shift?  Without data you won’t know that.

• You might also want to measure the intervention accuracy. Did the driver actually know if their blind spot was clear when they made a lane change, or did they just get lucky when they did a takeover and a lane change? Knowing the fraction of time that drivers do the right thing is important. It’s not going to be 100%, but you just need to make sure it’s good enough to achieve your safety goal. 

• And every once in a while if the driver tries to do a takeover and the vehicle doesn’t respond you really need to know that that happened.

Now those metrics are somewhat detailed. It might not be the kind of thing that a government agency or the public would really be in a position to process well. So you want some roll up metrics as well. But these are mostly things that have to do with non disengagements rather than disengagements. 

To really deal with road testing safety, the number one concern has got to be times when the vehicle did something dangerous and the safety driver did not do the right compensating action. Your drivers aren’t going to be perfect. Are you planning on just blaming them when there’s a mishap or are you actually taking measurements to find out if you’re hitting performance targets?  So if you have a target for how much the driver has to pay attention and how much is okay to lose attention for a second or two, then you need to know if you’re hitting that target and it’s not going to be perfection.

You probably also want some metrics for rule violations. 

• For example, running a stop sign even though nothing bad happened, is still a bad thing. The number of times that happens is not going to be zero, but it should be so low that it’s dramatically better than normal human drivers for example.
• You probably want to measure how often you unreasonably encroach on a bike lane even if there’s no bicyclists there.
• You might want to measure how often you fail to yield to a pedestrian. Most pedestrians aren’t going to jump out in front of a car to get hit, but the fact that they had to back off and let you pass means you weren’t behaving the way you were supposed to. 
• Probably the most important thing to measure is near misses (or as some people call them near hits) where the margin was just too small even if you got lucky and nothing bad actually happened. For example, maybe you’re supposed to leave three feet of clearance to a bicyclist. (And at high speeds I would hope you do leave at least that, but if you’re down at two or three miles an hour crawling through a dense urban area and you're two feet away from a bicyclist, maybe that’s okay.) You need a definition of near misses that responds to the particular situation but ultimately you want to know how often you had a near miss, meaning you were in a situation more dangerous than it was supposed to be. If your safety driver catches it, then great. But if your safety driver does not, that means you’re operating taking chances that you didn’t intend to take, and knowing that is super important.

No safety program and no safety metric is going to be perfect. It’s unreasonable to expect self driving car testing to be absolutely perfect with not even a fender bender. Surely mishaps will happen especially when interacting with human drivers. But there has to be some sort of strategy. 

A viable strategy might be that you use above average safety drivers who have to deal with the extra complexity of the safety platforms, but they come out overall to be at least as good as normal human drivers. You’re making a big ask to have these safety drivers be better than normal human drivers and compensate for all the difficulties with this technology. Probably it can be done, but you really need to know you got there instead of training the drivers, sending them out on roads and simply hoping that you’re safe enough. You really want some metrics. 

To build trust companies should be exposing at least the high level rollup metrics to the public. They have nothing to do with the secret sauce behind the self driving car but they have everything to do with the safety of the general public as this technology is being tested on public roads.

Disengagements as a progress metric is a bad idea (Metrics Episode 2)

 We should be worried about road testing safety metrics, not disengagements.

A disengagement happens when the autonomy in a self driving car detects an internal problem, or a human test driver takes over control of a self driving car test platform because of safety concerns. Self driving car developers have to report these disengagements, for example, to California. The apparent rationale for requiring these reports is that all things being equal, disengagements per mile might decrease over time as technology matures. Along those lines, eventually when disengagements reached zero, you might think it’s time to deploy the vehicle without a human test driver. The problem is that this model is much too simplistic and more importantly, not all things are equal.

Let’s start with some basics. Not all miles are equal. If you wanted to game disengagements, you could do so by driving around an empty block in beautiful weather at 4:00 AM with no traffic, no pedestrians, nothing on the road, around and around in circles. You get a lot of miles. You wouldn’t learn much, but you get a lot of miles. That’s not at all the same as, for example, trying to drive across all 446 bridges in Pittsburgh during a blizzard. Those miles are just not the same. Another potential problem is that not all safety drivers are equal. Some safety drivers will be more prone to be cautious and others less cautious. Hopefully there is rigorous driver screening so that the test drivers safety drivers are the right amount of cautious, but in fact, this is still an area the industry’s working on. So even with the best intentions, all disengagements might not be equal.

Now what happens next? After this disengagement data is collected, the metrics get published and that leads the media to trend those published metrics into the great disengagement metric horse race. Pundits opine about which company is in the lead and companies who are ahead say, "yeah, look at our low disengagement rate" and so on. Now it’s hard to blame people for doing this because the developers operate in such great secrecy. That’s really the only progress metric out there, but it’s not a good metric. In fact, it’s probably a harmful metric. A big concern is that using disengagements as a metric provide strong incentives for behavior that make things worse instead of better, especially if you’re being judged in it for progress and maybe your next funding round depends on your disengagement metric.

Here’s a problem: the disengagement metric penalizes companies who tell their safety drivers to be extra safe by being extra quick to disengage. So, that means there’s an incentive to tell drivers to give the vehicles a little more slack, which might or might not be as safe as it should be. Now, I’m not saying that people necessarily do this intentionally, but in a very competitive environment, there’s going to be natural pressure to say, well, if it’s on the borderline, let it go to make our numbers look better and probably it’s safe enough. And people might convince themselves of that, even though they’re operating unsafely. 

Another problem is the metric penalizes companies who are working on difficult operational design domains and incentivizes them to chase easy miles. Now again, I’m not saying companies are doing this on purpose, but certainly the incentive is there. In fact, there are good reasons why a company making excellent progress would actually see their disengagements increase rather than decrease. Maybe the company’s expanded its operational design domain to handle more challenging situations. The week that they decide to start operating in rain, I’d imagine the disengagement rate would go up instead of down.

Another reason is maybe safety driver training has been improved and the policies have been changed to improve road test safety at the expense of increased disengagements. I’d love to see that kind of outcome, but it makes the metric look bad. 

Some companies filter the disengagement to say, okay, well we’re only going to report the disengagements that count and the problem is that’s a two edge sword. Sure, it makes sense not to report planned disengagements. If it’s the end of the testing day and you’re going to take the car back to the garage and you turn off autonomy, surely that disengagement should not count.

But because companies are being judged on disengagements, there’s also incentive to gain them a bit. For example, the driver might take over control and maybe it was a dangerous situation, maybe it wasn’t, but because of the pressure of metrics, the company decides to round down and attributed to something else when in fact probably the car should have been doing better and the disengagement should have at least partially counted. You might end up with under-reporting disengagements and that should be a cause for concern.

Let me give a couple of hypothetical examples of the kind of situations which could lead to this kind of bad outcome. For example, let’s say a company only reports disengagement if an after the fact simulation says yes, they would hit something. And then the car goes by a pedestrian and the safety driver disengages because it looks like it’s going to be kind of close to the pedestrian, they don’t want to take a chance. So far, so good. Now let’s say you would’ve missed the pedestrian by 10 to 20 feet. Okay, fine. That disengagement probably should not count, but what if you’re only going to miss the pedestrian by one inch? Well, you didn’t hit the pedestrian. You could say, well, that one doesn’t count because we didn’t hit anything. But I’m going to say missing a pedestrian by one inch, that one ought to count.

And so without details about how exactly this reporting has been done, we don’t really know what the numbers mean. 

Here’s another hypothetical example. Let’s say a test vehicle runs a red light, but it’s late at night and no one’s around. The driver looks around, the driver’s been told, unfortunately, to make sure the disengagement numbers look good. There’s no cross traffic. The driver says, you know what? I’m just going to let it run the red light because no harm will be done. There’s a situation where the disengagement doesn’t happen, the number looks good, but in that hypothetical scenario, the driver’s been incentivized to do unsafe things. That made up example brings to a head the real issue here. 

Disengagements might be useful input to some parts of an engineering process, but in a hyper competitive market, they provide all the wrong incentives for road test safety.  Really are we worried about progress? 

Do we really want the Departments of Transportation measuring progress of companies? Their job really has to be keeping people safe on the road. And so if the publicly reported data actually provides incentives to do road testing unsafely to make progress look good, that’s a problem. Historically, those kinds of incentives, they lead to a story that ends badly for everyone. 

Well, it’s interesting to know what progress might be made by the industry. But if you really care about safety, the thing you ought to be worried about right now is road testing safety. Now, some of the reporting data actually does help with that. For example, crashes being reported. Sure, that actually directly measures road testing safety. But all the disengagement metrics and all the buzz really doesn’t help road testing safety -- and in fact might be hurting it. It might be putting pressure on the companies to undermine road testing safety just to make their numbers look better.

I would recommend that California and any other government that’s doing this should stop forcing disengagement reporting and instead encourage the reporting of more productive metrics that are about road testing safety, not about the horse race to get a self driving car deployed. This is not a simple ask. This is actually a hard thing to do. But the industry should step up and propose metrics that have to do with road testing safety. That’s going to help them build trust with the public and it’s going to help the government agencies fulfill their responsibility to ensure that the road testing and eventual deployment is done in a safe, responsible manner.

To learn more, we recommend a paper our team published for SAE World Congress titled “Safety Argument Considerations For Road Testing of Autonomous Vehicles.” This paper gives guidance on a safety case for human supervision of road testing.

Number of miles as a self-driving car progress and safety metric (Metrics Episode 1)

Even if you have the best possible safety drivers, every test mile adds some risk. Make sure every mile of road testing is actually doing something important.

You hear people saying that testing for lots and lots of miles must mean some self-driving car company is better than the rest, or at least in the lead in the so-called race to autonomy. But not so fast, there’s more to it than that.

Some companies have millions of miles of road testing experience, and to be sure that’s an impressive accomplishment. If they have that many miles, certainly they have incentive to boast about it in saying, “Look how many miles we have.” And the press often says, “All right, these guys have lots and lots of miles, somehow that must be in they’re ahead.” But miles really doesn’t tell you who’s safer or even necessarily ahead. Miles is a mostly reflection about the resources they have available to deploy a test fleet.

If you have lots of money, you can buy a lot of cars, hire lots of people, and put them out there to rack up the miles. Sure, having lots of resources makes it easier to make progress, but it doesn’t tell you who’s better. In fact, some companies are taking pride in reducing their testing road miles and instead putting those resources into simulation and other sorts of engineering activities other than road testing. It’s hard to believe that if you don’t have any miles or you only have a few miles that you’re really ready to deploy, I get that. And it’s hard to believe that a company without lots of data collection miles is actually seeing all the things that are going to need to deal with when they pick an operational design domain. But nobody will ever get to the billions of representative miles necessary to say anything compelling about expected safety until after they actually deploy their fleet.

Even if you have a lot of miles, not all miles are created equal. Okay, so we have the billion miles -- are they in simulation around the same block? Are they all in sunny weather -- or do they include rain and hail and ice and all those other types of weather conditions you care about? Are they in a place with wide roads and no pedestrians or are they in a chaotic urban center? Oh, was that chaotic urban center at 5:00 AM? Was it during rush hour? Did you include things like construction zones or Halloween costumes or all sorts of things that don’t happen that often, but that you have to handle the right way? If somebody wants to talk about miles, they should also talk about how those miles show they’ve covered the entirety of their operational design domain.

There’s a potential problem with using miles as a measure of progress because they can motivate the wrong behavior. If you’re judged solely on how many miles you’ve racked up, then you’re going to optimize the easy miles perhaps, but worse, every mile on public roads is a chance to make a mistake. Even if you have the best possible safety drivers, every test mile adds some risk. Hopefully that risk is no worse than human driver vehicle risks, but that’s a different discussion. So every mile costs not only money, but also puts you at some risk of some adverse news or some unfortunate event. So you should think carefully about racking up a lot of miles for the sake of miles; you should make every mile earn its keep. Make sure every mile of road testing is actually doing something important.

Now you have to have miles. For sure you won’t know if your system is done until you’ve done some road testing. But the early miles don’t have to be about testing at all, they can be about data collection. You don’t need to run an autonomous system to collect sensor data. You can just collect sensor data with a human driver and have risk no different than someone just driving around normally. That means the road test miles should not be used for primary data collection, but rather as a way to confirm your design is solid. So don’t think of miles on the road is as debugging problems with your system while you’re on public roads. Rather think about road testing miles as a way of making sure you didn’t overlook anything in a system you think is just about ready to go. That means instead of going for road test miles, companies should instead be trying to get data collection miles for most of the miles and making sure they cover the ODD.

Then they can feed that information to a simulation, make sure the system handles all the miles properly, and then after they’re pretty sure they got it right, they should be doing road testing miles simply to confirm that all the engineering effort they put in resulted in a system that behaves the way they expected it to. In other words, road testing miles should be the boring part of tying a ribbon around and putting a bow on a great design. Road testing miles should not be the pointy end of the spear for development.

Going back up to why people talk about miles as a progress metric, sure, having no miles on public roads probably means you’re not ready to deploy because you haven’t checked to make sure it works. But having a ton of miles doesn’t mean you’re ahead, it just means you’re well-funded and you’re out there operating. If you really want to know how someone’s doing, it isn’t just the miles but rather it’s which miles, and it’s how those miles go together with all the other engineering activities to make sure that they’re taking a solid engineering approach to designing a safe self driving car.

Positive Trust Balance for Self Driving Car Deployment

By Philip Koopman and Michael Wagner, Edge Case Research

Self-driving cars promise improved road safety. But every publicized incident chips away at confidence in the industry’s ability to deliver on this promise, with zero-crash nirvana nowhere in sight. We need a way to balance long term promise vs. near term risk when deciding that this technology is ready for deployment. A “positive trust balance” approach provides a framework for making a responsible deployment decision by combining testing, engineering rigor, operational feedback, and transparent safety culture.

MILES AND DISENGAGEMENTS AREN’T ENOUGH

Too often, discussions about why the public should believe a particular self-driving car platform is well designed center around number of miles driven. Simply measuring the number of miles driven has a host of problems, such as distinguishing “easy” from “hard” miles and ensuring that miles driven are representative of real world operations. That aside, accumulating billions of road miles to demonstrate approximate parity to human drivers is an infeasible testing goal. Simulation helps, but still leaves unresolved questions about including enough edge cases that pose problems for deployment at scale.

By the time a self-driving car design is ready to deploy, the rate of potentially dangerous disengagements and incidents seen in on-road testing should approach zero. But that isn’t enough to prove safety. For example, a hypothetical ten million on-road test miles with no substantive incidents would still be a hundred times too little to prove that a vehicle is as safe as a typical human driver. So getting to a point that dangerous events are too rare to measure is only a first step.

In fact, competent human drivers are so good that there is no practical way to measure that a newly developed self-driving car has a suitably low fatality rate. This should not be news. We don’t fly new aircraft designs for billions of hours before deployment to measure the crash rate. Instead, we count on a combination of thorough testing, good engineering, and safety culture. Self-driving cars typically rely on machine learning to sense the world around them, so we will also need to add significant feedback from vehicles operating in the field to plug inevitable gaps in training data.

POSITIVE TRUST BALANCE

The self-driving car industry is invested in achieving a “positive risk balance” of being safer than a human driver. And years from now actuarial data will tell us if we succeeded. But there will be significant uncertainty about risk when it’s time to deploy. So we’ll need to trust development and deployment organizations to be doing the right things to minimize and manage that risk.

To be sure, developers already do better than brute force mileage accumulation. Simulations backed up by comprehensive scenario catalogs ensure that common cases are covered. Human copilots and data triage pipelines flag questionable self-driving behavior, providing additional feedback. But those approaches have their limits.

Rather than relying solely on testing, other industries use safety standards to ensure appropriate engineering rigor. While traditional safety standards were never intended to address self-driving aspects of these vehicles, new standards such as Underwriters Laboratories 4600 and ISO/PAS 21448 are emerging to set the bar on engineering rigor and best practices for self-driving car technology.

The bad news is that nobody knows how to prove that machine learning based technology will actually be safe. Although we are developing best practices, when deploying a self-driving car we’ll only know whether it is apparently safe, and not whether it is actually as safe as a human driver. Going past that requires real world experience at scale.

Deploying novel self-driving car technology without undue public risk will involve being able to explain why it is socially responsible to operate these systems in specific operational design domains. This requires addressing all of the following points:

Is the technology as safe as we can measure? This doesn’t mean it will be perfect when deployed. Rather, at some point we will have reached the limits of viable simulation and testing.

Has sufficient engineering rigor been applied? This doesn’t mean perfection. Nonetheless, some objective process such as establishing conformance to sufficiently rigorous engineering standards that go beyond testing is essential.

Is a robust feedback mechanism used to learn from real world experience? There must be proactive, continual risk management over the life of each vehicle based on extensive field data collection and analysis.

Is there a transparent safety culture? Transparency is required in evolving robust engineering standards, evaluating that best practices are followed, and ensuring that field feedback actually improves safety. A proactive, robust safety culture is essential. So is building trust with the public over time.

Applying these principles will potentially change how we engineer, regulate, and litigate automotive safety. Nonetheless, the industry will be in a much better place when the next adverse news event occurs if their figurative public trust account has a positive balance.

Philip Koopman is the CTO of Edge Case Research and an expert in autonomous vehicle safety. Including his role as a faculty member at Carnegie Mellon University, Koopman has been helping government, commercial and academic self-driving developers improve safety for over 20 years. He is a principal contributor to the Underwriters Laboratories 4600 safety standard.

Michael Wagner is the CEO of Edge Case Research. He started working on autonomy at Carnegie Mellon over 20 years ago.

(Original post here:  https://medium.com/@pr_97195/positive-trust-balance-for-self-driving-car-deployment-ff3f04a7ef93)

The Lesson Learned from the Tempe Arizona Autonomous Driving System Testing Fatality NTSB Report

Now that the press flurry over the NTSB's report on the Autonomous Driving System (ADS) fatality in Tempe has subsided, it's important to reflect on the lessons to be learned. Hats off to the NTSB for absolutely nailing this. Cheers to the Press who got the messaging right. But not everyone did. The goal of this essay is to help focus on the right lessons to learn, clarify publicly stated misconceptions, and emphasize the most important take-aways.

I encourage everyone in the AV industry to watch the first 5 and a half minutes of the NTSB board meeting video ( Youtube: Link // NTSB Link). Safety leadership should watch the whole thing. Probably twice. Then present a summary at your company's lunch & learn.

Pay particular attention to this part from Chairman Sumwalt: "If your company tests automated driving systems on public roads, this crash -- it was about you.  If you use roads where automated driving systems were being tested, this crash -- it was about you."

I live in Pittsburgh and these public road tests happen near my place of work and my home. I take the lessons from this crash personally. In principle, every time I cross a street I'm potentially placed at risk by any company that might be cutting corners on safety. (I hope that's none. All the companies testing here have voluntarily submitted compliance reports for the well-drafted PennDOT testing guidelines. But not every state has those, and those guidelines were developed largely in response to the fatality we’re discussing.)

I also have long time friends who have invested their careers in this technology. They have brought a vibrant and promising industry to Pittsburgh and other cities.  Negative publicity resulting from a major mishap can threaten the jobs of those employed by those companies.

So it is essential for all of us to get safety right.

The first step: for anyone in charge of testing who doesn't know what a Safety Management System (SMS) is: (A) Watch that NSTB hearing intro. (B) Pause testing on public roads until your company makes a good start down that path. (Again, the PennDOT guidelines are a reasonable first step accepted by a number of companies. LINK)  You’ll sleep better having dramatically improved your company’s safety culture before anyone gets hurt unnecessarily.

A major implication of the NTSB report is that if there is even a single company road testing without an adequate SMS -- that is everyone's problem. 

Clearing up some misconceptions

I’ve seen some articles and commentary that missed the point of all of this. Large segments of coverage emphasized technical shortcomings of the system - That's not the point. Other coverage highlighted test driver distraction - That's not the point either.  Some blame that it was too dark, which is simply untrue -- NTSB estimates the pedestrian was visible at the full line of sight range of 637 feet. The fatal mishap involved technical shortcomings, and the test driver was not paying adequate attention. Both contributed to the mishap, and both were bad things.

But the lesson to learn is that solid safety culture is without a doubt necessary to prevent avoidable fatalities like these. That is the Point.

To make the most of this teachable moment let's break things down further. These discussions are not really about the particular test platform that was involved. The NTSB report gave that company credit for significant improvement. Rather, the objective is to make sure everyone is focused on ensuring they have learned the most important lesson so we don’t suffer another avoidable ADS testing fatality.

A self-driving car killed someone - NOT THE POINT

This was not a self-driving car. It was a test platform for Automated Driving System (ADS) technology. The difference is night and day.  Any argument that this vehicle was safe to operate on public roads hinged on a human driver not only taking complete responsibility for operational safety, but also being able to intervene when the test vehicle inevitably made a mistake. It's not a fully automated self-driving car if a driver is required to hover with hands above the steering wheel and foot above the brake pedal the entire time the vehicle is operating.

It's a test vehicle. The correct statement is: a test vehicle for developing ADS technology killed someone.

The pedestrian was initially said to jump out of the dark in front of the car - NOT THE POINT

I still hear this sometimes based on the initial video clip that was released immediately after the mishap. The pedestrian walked across almost 4 lanes of road in view of the test vehicle before being struck, with sufficient lighting to have been seen by the driver. The test vehicle detected the pedestrian 5.6 seconds before the crash. That was plenty of time to avoid the crash, and plenty of time to track the pedestrian crossing the street to predict that a crash would occur. Attempting to claim that this crash was unavoidable is incorrect, and won't prevent the next ADS testing fatality.

It's the pedestrian's fault for jaywalking - NOT THE POINT

Jaywalking is what people do when it is 125 yards to the nearest intersection and there is a paved walkway on the median. Even if there is a sign saying not to cross.  Tearing up the paved walkway might help a little on this particular stretch of road, but that's not going to prevent jaywalking as a potential cause of the next ADS testing fatality.

Victim's apparent drug use - NOT THE POINT

It was unlikely that the victim was a fully functional, alert pedestrian. But much of the population isn't in this category for other reasons. Children, distracted walkers, and others with less than perfect capabilities and attention cross the street every day, and we expect drivers to do their best to avoid hitting them.

There is no indication that the victim’s medical condition substantively caused the fatality. (We're back to the fact that the pedestrian did not jump in front of the car.) It would be unreasonable to insist that the public has the responsibility to be fully alert and ready to jump out of the way of an errant ADS test platform at all times they are outside their homes.

Tracking and classification failure - NOT THE POINT

The ADS system on the test vehicle suffered some technical issues that prevented predicting where the pedestrian would be when the test vehicle got there, or even recognizing the object it was sensing was a pedestrian walking a bicycle. However, the point of operating the test vehicle was to find and fix defects.

Defects were expected, and should be expected on other ADS test vehicles. That's why there is a human safety driver. Forbidding public road testing of imperfect ADS systems basically outlaws road testing at this stage. Blaming the technology won't prevent the next ADS testing fatality, but it could hurt the industry for no reason.

It's the technology's fault for ignoring jaywalkers - NOT THE POINT

This idea has been circulating, but apparently this isn't quite true. Jaywalkers aren't ignored, but rather according to the information presented by the NTSB a pedestrian isn't expected to cross the street at first. Once the pedestrian moves for a while a track is built up that could indicate street crossing, but until then movement into the street is considered unexpected if the pedestrian is not at a designated crossing location. A deployment-ready ADS could potentially use a more sophisticated approach to predict when a pedestrian would enter the roadway.

Regardless of implementation, this did not contribute to the fatality because the system never actually classified the victim as a pedestrian. Again, improving this or other ADS technical features won't prevent the next ADS testing fatality. That’s because testing safety is about the safety driver, not which ADS prototype functions happen to be active on any particular test run.

ADS emergency braking behavior - NOT THE POINT

The ADS emergency braking function had behaviors that could hinder its ability to provide backup support to the safety driver. Perhaps another design could have done better for this particular mishap. However, it wasn't the job of the ADS emergency braking to avoid hitting a pedestrian. That was the safety driver's job. Improving ADS emergency braking capabilities might reduce the probability of an ADS testing fatality, but won't entirely prevent the next fatality from happening sooner than it should.

Native emergency braking disabled - NOT THE POINT

It looks bad to have disabled the built-in emergency braking system on the passenger vehicle used as the test platform. The purpose of such systems is to help out after the driver makes a mistake. In this case there is a good, but not perfect, chance it would have helped. But as with the ADS emergency braking function, this simply improves the odds. Any safety expert is going to say your odds are better with both belt and suspenders, but enabling this function alone won't entirely prevent the next ADS testing fatality from happening before it should.

Inattentive safety driver - NOT THE POINT

There is no doubt that an inattentive safety driver is dangerous when supervising an ADS test vehicle. And yet, driver complacency is the expected outcome of asking a human to supervise an automated system that works most of the time. That’s why it’s important to ensure that driver monitoring is done continually and used to provide feedback. (In this case a form of driver monitoring equipment was installed, but data was apparently not used in a way that assured effective driver alertness.)

While enhanced training and stringent driver selection can help, effective analysis and action taken upon monitoring data is required to ensure that drivers are actually paying attention in practice. Simply firing this driver without changing anything else won't prevent the next ADS testing fatality from happening to some other driver who has slipped into bad operational habits.

A fatality is regrettable, but human drivers killed about 100 people that same day with minimal news attention - NOT THE POINT

Some commentators point out the ratio of fatalities caused by test vehicles vs. general automotive fatality rates. They then generally argue that a few deaths in comparison to the ongoing carnage of regular cars is a necessary and appropriate price to pay for progress. However, this argument is not statistically valid.

Consider a reasonable goal that ADS testing (with highly qualified, alert drivers presumed) should be no more dangerous than the risk presented by normal cars. For normal US cars that's ballpark 500 million road miles per pedestrian fatality. This includes mishaps caused by drunk, distracted, and speeding drivers. Due to the far smaller number of miles being driven by current test platform fleet sizes, the "budget" for fatal accidents due to ADS road testing phase should, at this early stage, still be zero.

The fatality somehow “proves” that self-driving car technology isn't viable - NOT THE POINT

Some have tried to draw conclusions about the viability of ADS technology from the fact that there was a testing fatality. However, the issues with ADS technical performance only prove what we already knew. The technology is still maturing, and a human needs to intervene to keep things safe. This crash wasn't about the maturity of the technology; it was about whether the ADS public road testing itself was safe.

Concentrating on technology maturity (for example, via disclosing disengagement rates) serves to focus attention on a long term future of system performance without a safety driver. But the long term isn’t what’s at issue.

The more pressing issue is ensuring that the road testing going on right now is sufficiently safe. At worst, continued use of disengagement rates as the primary metric of ADS performance could hurt safety rather than help. This is because disengagements, if gamed, could incentivize safety drivers to take chances by avoiding disengagements in uncertain situations to make the numbers look better. (Some companies no doubt have strategies to mitigate this risk. But those are probably the companies with an SMS, which is back to the point that matters.)

THE POINT: The safety culture was broken

Safety culture issues were the enabler for this particular crash. Given the limited number of miles that can be accumulated by any current test fleet, we should see no fatalities occur during ADS testing. (Perhaps a truly unavoidable fatality will occur. This is possible, but given the numbers it is unlikely if ADS testing is reasonably safe. So our goal should be set to zero.) Safety culture is critical to ensure this.

The NTSB rightly pushes hard for a safety management system (SMS). But be careful to note that they simply say that this is a part of safety culture, not all of it. Safety culture means, among other things, taking responsibility for ensuring that their safety drivers are actually safe despite the considerable difficulty of accomplishing that. Human safety drivers will make mistakes, but a strong safety culture accounts for such mistakes in ensuring overall safety.

It is important to note that the urgent point here is not regulating self-driving car safety, but rather achieving safe ADS road testing. They are (almost) two entirely different things. Testing safety is about whether the company can consistently put an alert, able-to-react safety driver on the road. On the other hand, ADS safety is about the technology. We need to get to the technology safety part over time, but ADS road testing is the main risk to manage right now.

Perhaps dealing with ADS safety would be easier if the discussions of testing safety and deployment safety were more cleanly separated.

THE TAKEAWAYS:

Chairman Sumwalt summed it up nicely in the intro. (You did watch that 5 and half minute intro, right?)  But to make sure it hits home, this is my take:

One company's crash is every company's crash.  You'll note I didn't name the company involved, because really that's irrelevant to preventing there from being a next fatality and the potential damage it could do to the industry’s reputation.

The bigger point is every company can and should institute good safety culture before further fatalities take place if they have not done so already. The NTSB credited the company at issue with significant change in the right direction.  But it only takes one company who hasn’t gotten the message to be a problem for everyone. We can reasonably expect fatalities involving ADS technology in the future even if these systems are many times safer than human drivers. But there simply aren’t that many vehicles on the road yet for a truly unavoidable mishap to be likely to occur. It’s far too early.

If your company is testing (or plans to test) autonomous vehicles, get a Safety Management System in place before you do public road testing. At least conform to the details in the PennDOT testing guidelines, even if you’re not testing in Pennsylvania. If you are already testing on public roads without an SMS, you should stand down until you get one in place.

Once you have an SMS, consider it a down-payment on a continuing safety culture journey.

The NTSB full report can be accessed here: https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf

Prof. Philip Koopman, Carnegie Mellon University

Author Note: The author and his company work with a variety of customers helping to improve safety. He has been involved with self-driving car safety since the late 1990s. These opinions are his own, and this piece was not sponsored.

Autonomous Vehicle Testing Safety Needs More Transparency

Last week there were two injuries involving human-supervised autonomous test shuttles on different continents, with no apparent connection other than random chance.  (For example: Link) As deployment of this work-in-progress technology scales up in public, we know that we can expect more high-profile accidents. Fortunately this time nobody was killed or suffered life-altering injuries. (But we still need to find out what actually happened.) And to be sure, human-driven vehicles are far from accident-free. But what about next time?

The bigger issue for the industry is: will the next autonomous vehicle testing mishap be due to rare and random problems that are within the bounds of reasonable risk? Or will they be due to safety issues that could and should have been addressed beforehand?

Public trust in autonomous vehicle technology has already eroded in the past year. Each new mishap has the unfortunate potential to make that situation worse, regardless of the technical root cause. While no mode of transportation is perfectly safe, it's important that the testing of experimental self driving car technology not expose the public to reasonably avoidable risk. And it's equally important that the public's perception matches the actual risk.

Historically the autonomous vehicle industry has operated under a cloak of secrecy. As we've seen, that can lead to boom and bust cycles of public perception, with booms of optimism followed by backlash after each publicized accident. But in fairness, if there is no information about public testing risk other than hype about an accident-free far-flung future, what is the public supposed to think? Self-driving cars won't be perfect. The goal is to make them better than the current situation. One hopes that along the way things won't actually get worse.

Some progress in public safety disclosure has been made, albeit with low participation rates. One of the two vehicles involved in injuries this past week has a public safety report available. The other does not. In fact, a significant majority of testing organizations have not taken the basic step of making a Voluntary Safety Self-Assessment report available to NHTSA. And to be clear, that disclosure process is more about explaining progress toward production maturity rather than the specific topic of public testing safety.

The industry needs to do better at providing transparent, credible safety information while testing this still-experimental technology. Long term public education and explanation are important. But the more pressing need revolves around what's happening on our roads right now during testing operations. That is what is making news headlines, and is the source of any current risk.

At some point either autonomous vehicle testers are actually doing safe, responsible public operations or they aren't. If they aren't, that is bound to catch up with them as operations scale up. From the point of view of a tester:

- It's a problem if you can't explain to yourself why you are acceptably safe in a methodical, rigorous way. (In that case, probably you're unsafe.) 
- It's a problem if you expect human safety drivers to perform with superhuman ability. They can't.
- It's a problem if you aren't ready to explain to authorities why are still acceptably safe after a mishap.
- It's a problem if you can't explain to a jury that you used reasonable care to ensure safety. 
- It's a problem if your company's testing operations get sidelined by an accident investigation.
- It's a problem for mishap victims if accidents occur that were reasonably avoidable, especially involving vulnerable road users. 
- It's a problem for the whole industry if people lose trust in the technology's ability to operate safely in public areas. 

Therefore: 
- It's a problem if you can't explain to the public -- with technical credibility -- why they should believe you are safe. Preferably before you begin testing operation on public roads. 

Some companies are working on transparent safety more aggressively than others. Some are working on safety cases that contain detailed chains of reasoning and evidence to ensure that they have all the bases covered for public road testing. Others might not be.  But really, we don't know. 

And that is an essential part of the problem -- we really have no idea who is being diligent about safety. Eventually the truth will out, but bad news is all too likely to come in the form of deaths and injuries. We as an industry need to keep that from happening.

It only takes one company to have a severe mishap that, potentially, dramatically hurts the entire industry. While bad luck can happen to anyone, it's more likely to happen to a company that might be cutting corners on safety to get to market faster. 

The days of "trust us, we're smart" are over for the autonomous vehicle industry. Trust has to be earned. Transparency backed with technical credibility is a crucial first step to earning trust. The industry has been given significant latitude to operate on public roads, but that comes with great responsibility and a need for transparency regarding public safety.

Safety should truly come first. To that end, every company testing on public roads should immediately make a transparent, technically credible statement about their road testing safety practices. A simple "we have safety drivers" isn't enough. These disclosures can form a basis for uniform practices across the industry to make sure that this technology survives its adolescence and has a chance to reach maturity and the benefits that promises.

Author:
Dr. Philip Koopman is co-founder and CTO of Edge Case Research, which helps companies make their autonomous systems safer. He is also a professor at Carnegie Mellon University. He is a principle technical author of the UL 4600 draft standard for autonomous system safety, and has been working on self-driving car safety for more than 20 years. Contact: pkoopman@ecr.ai

Autonomous Vehicle Fault and Failure Management

When you build an autonomous vehicle you can't count on a human driver to notice when something's wrong and "do the right thing." Here is a list of faults, system limitations, and fault responses AVs will need to get right. Did you think of these?

System Limitations:

Sometimes the issue isn't that something is broken, but rather simply that all vehicles have limitations. You have to know your system's limitations.

• Current capabilities of sensors and actuators, which can depend upon the operational state space.
• Detecting and handling a vehicle excursion outside the operational state space for which it was validated, including all aspects of {ODD, OEDR, Maneuver, Fault} tuples.
• Desired availability despite fault states, including any graceful degradation plan, and any limits placed upon the degraded operational state space.
• Capability variation based on payload characteristics (e.g. passenger vehicle overloaded with cargo, uneven weight distribution, truck loaded with gravel, tanker half filled with liquid) and autonomous payload modification (e.g. trailer connect/disconnect).
• Capability variation based on functional modes (e.g. pivot vs. Ackerman vs. crab steering, rear wheel steering, ABS or 4WD engaged/disengaged).
• Capability variation based on ad-hoc teaming (e.g. V2V, V2I) and planned teaming (e.g. leader-follower or platooning vehicle pairing).
• Incompleteness, incorrectness, corruption or unavailability of external information (V2V, V2I).

System Faults:

• Perception failure, including transient and permanent faults in classification and pose of objects.
• Planning failures, including those leading to collision, unsafe trajectories (e.g., rollover risk), and dangerous paths (e.g., roadway departure).
• Vehicle equipment operational faults (e.g., blown tire, engine stall, brake failure, steering failure, lighting system failure, transmission failure, uncommanded engine power, autonomy equipment failure, electrical system failure, vehicle diagnostic trouble codes).
• Vehicle equipment maintenance faults (e.g., improper tire pressure, bald tires, misaligned wheels, empty sensor cleaning fluid reservoir, depleted fuel/battery).
• Operational degradation of sensors and actuators including temporary (e.g., accumulation of mud, dirt, dust, heat, water, ice, salt spray, smashed insects) and permanent (e.g., manufacturing imperfections, scratches, scouring, aging, wear-out, blockage, impact damage).
• Equipment damage including detecting and mitigating catastrophic loss (e.g., vehicle collisions, lighting strikes, roadway departure), minor losses (e.g., sensor knocked off, actuator failures), and temporary losses (e.g., misalignment due to bent support bracket, loss of calibration).
• Incorrect, missing, stale, and inaccurate map data.
• Training data incompleteness, incorrectness, known bias, or unknown bias.

Fault Responses:

Some of the faults and limitations fall within the purview of safety standards that apply to non-autonomous functions. However, a unified system-level view of fault detection and mitigation can be useful to ensure that no faults are left unaddressed. More importantly, to the degree that credit has been taken for a human driver participating in fault mitigation by safety standards, that places fault mitigation obligations upon the autonomy.

• How the system behaves when encountering an exceptional operational state space, experiencing a fault, or reaching a system limitation.
• Diagnostic gaps (e.g., latent faults, undetected faults, undetected faulty redundancy).
• How the system re-integrates failed components, including recovery from transient faults and recovery from repaired permanent faults during operation and/or after maintenance.
• Response and policies for prioritizing or otherwise determining actions in inherently risky or certain-loss situations.
• Withstanding an attack (system security, compromised infrastructure, compromised other vehicles), and deterring inappropriate use (e.g., malicious commands, inappropriately dangerous cargo, dangerous passenger behavior).
• How the system is updated to correct functional defects, security defects, safety defects, and addition of new or improved capabilities.

Is there anything we missed?

(This is an excerpt of Koopman, P. & Fratrik, F., "How many operational design domains, objects, and events?" SafeAI 2019, AAAI, Jan 27, 2019.)

Event Detection Considerations for Autonomous Vehicles (OEDR -- part 2)

Object and Event Detection and Recognition (OEDR) also involves making predictions about what might happen next. Is that pedestrian waiting for a bus? or about to walk out into the crosswalk right in front of my car? Did you think of all of these aspects?

The infamous Pittsburgh Left. The first vehicle at a red light
will (sometimes) turn left when the light turns to green.

Some factors to consider when deciding what events and behaviors your system needs to predict include:

• Determining expected behaviors of other objects, which might involve a probability distribution and is likely to be based on object classification.
• Normal or reasonably expected movements by objects in the environment.
• Unexpected, incorrect, or exceptional movement of other vehicles, obstacles, people, or other objects in the environment.
• Failure to move by other objects which are reasonably expected to move.
• Operator interactions prior to, during, and post autonomy engagement including: supervising driver alertness monitoring, informing occupants, interaction with local or remote operator locations, mode selection and enablement, operator takeover, operator cancellation or redirect, operator status feedback, operator intervention latency, single operator supervision of multiple systems (multi-tasking), operator handoff, loss of operator ability to interact with vehicle.
• Human interactions including: human commands (civilians performing traffic direction, police pull-over, passenger distress), normal human interactions (pedestrian crossing, passenger entry/egress), common human rule-breaking (crossing mid-block when far from an intersection, speeding, rubbernecking, use of parking chairs, distracted walking), abnormal human interactions (defiant jaywalking, attacks on vehicle, attempted carjacking), and humans who are not able to follow rules (children, impaired adults).
• Non-human interactions including: animal interaction (flocks/herds, pets, dangerous wildlife, protected wildlife) and delivery robots.

Is there anything we missed?   (Previous post had the "objects" part of OEDR.)

(This is an excerpt of Koopman, P. & Fratrik, F., "How many operational design domains, objects, and events?" SafeAI 2019, AAAI, Jan 27, 2019.)

Object Detection Considerations for Autonomous Vehicles (OEDR -- part 1)

Object and Event Detection and Recognition (OEDR) involves having an autonomous vehicle detect and classify various types of objects so that it can plan a response. Detection is only the first step; you need to also be able to classify the obstacle to predict what might happen next. Pedestrians tend to walk into the roadway. Bushes, not so much. Did you think of all of these aspects?

Q: Why did the Mr. Rogers-saurus cross the street?
A: Trick question; he doesn't actually move because he is part of the Pittsburgh Dinosaur Parade.

Some factors to consider when deciding what objects your system needs to detect and recognize include:

• Ability to detect and identify (e.g. classify) all relevant objects in the environment.
• Processing and thresholding of sensor data to avoid both false positives (e.g., bouncing drink can, steel bridge joint, steel road construction cover plate, roadside sign, dust cloud, falling leaves) and false negatives (e.g., highly publicized partially automated vehicle collisions with stationary vehicles)
• Characterizing the likely operational parameters of other road users (e.g., braking capability of leading and following vehicle, or whether another vehicle is behaving erratically enough that there is a likely control fault.)
• Permanent obstacles such as structures, curbs, median dividers, guard rails, trees, bridges, tunnels, berms, ditches, roadside and overhanging signage.
• Temporary obstacles such as transient keep-out zones, spills, floods, water-filled potholes, landslides, washed out bridges, overhanging vegetation, and downed power lines. (For practical purposes, “temporary” might mean obstacles not included on maps, with some vehicle having to be the first vehicle to detect an obstacle for placement even on a dynamic map.)
• People, including cooperative people, uncooperative people, malicious behaviors, and people who are unaware of the operation of the autonomous system.
• At-risk populations which might be unable, incapable, or exempt from following established rules and norms, such as children as well as injured, ability-impaired, or under-the-influence people.
• Other cooperative and uncooperative human-driven and autonomous vehicles.
• Other road users including special purpose vehicles, temporary structures, street dining, street festivals, parades, motorcades, funeral processions, farm equipment, construction crews, draft animals, farm animals, and endangered species.
• Other non-stationary objects including uncontrolled moving objects, falling objects, wind-blown objects, in-traffic cargo spills, and low-flying aircraft.

Is there anything we missed?   (Next post will have the "events" part of OEDR.)

(This is an excerpt of Koopman, P. & Fratrik, F., "How many operational design domains, objects, and events?" SafeAI 2019, AAAI, Jan 27, 2019.)

Operational Design Domain (ODD) for Autonomous Systems

The Operational Design Domain (ODD) is the set of environmental conditions that an autonomous system is designed to work in. Typically an ODD is thought of as some sort of geo-fencing plus a obvious weather conditions (rain, snow, sun). But, it's a lot more than that. Did you think of all of these?

Canton Avenue, the unofficial steepest street in the world, is less than 4 miles from downtown Pittsburgh.
Note cobblestone on the top half and the sidewalk stairs.  Cars slide (sometimes backwards) down the street in winter.
Geo-fencing is more complicated than drawing a circle around a city center.
[Wikipedia]

Characterizing the system operational environment should include at least the following:

• Operational terrain, and associated location-dependent characteristics (e.g., slope, camber, curvature, banking, coefficient of friction, road roughness, air density) including immediate vehicle surroundings and projected vehicle path. It is important to note that dramatic changes can occur in relatively short distances.
• Environmental and weather conditions such as surface temperature, air temperature, wind, visibility, precipitation, icing, lighting, glare, electromagnetic interference, clutter, vibration, and other types of sensor noise.
• Operational infrastructure, such as availability and placement of operational surfacing, navigation aids (e.g., beacons, lane markings, augmented signage), traffic management devices (e.g., traffic lights, right of way signage, vehicle running lights), keep-out zones, special road use rules (e.g., time-dependent lane direction changes) and vehicle-to-infrastructure availability.
• Rules of engagement and expectations for interaction with the environment and other aspects of the operational state space, including traffic laws, social norms, and customary signaling and negotiation procedures with other agents (both autonomous and human, including explicit signaling as well as implicit signaling via vehicle motion control).
• Considerations for deployment to multiple regions/countries (e.g., blue stop signs, “right turn keep moving” stop sign modifiers, horizontal vs. vertical traffic signal orientation, side-of-road changes).
• Communication modes, bandwidth, latency, stability, availability, reliability, including both machine-to-machine communications and human interaction.
• Availability and freshness of infrastructure characterization data such as level of mapping detail and identification of temporary deviations from baseline data (e.g., construction zones, traffic jams, temporary traffic rules such as for hurricane evacuation).
• Expected distributions of operational state space elements, including which elements are considered rare but in-scope (e.g. toll booths, police traffic stops), and which are considered outside the region of the state space in which the system is intended to operate.

Special attention should be paid to ODD aspects that are relevant to inherent equipment limitations, such as the minimum illumination required by cameras.

Are there any other aspects of ODD we missed?

(This is an excerpt of Koopman, P. & Fratrik, F., "How many operational design domains, objects, and events?" SafeAI 2019, AAAI, Jan 27, 2019.)

Ethical Problems That Matter for Self Driving Cars

It's time to get past the irrelevant Trolley Problem and talk about ethical issues that actually matter in the real world of self driving cars.  Here's a starter list involving public road testing, human driver responsibilities, safety confidence, and grappling with how safe is safe enough.

• Public Road Testing. Public road testing clearly puts non-participants such at pedestrians at risk. Is it OK to test on unconsenting human subjects? If the government hasn't given explicit permission to road test in a particular location, arguably that is what is (or has been) happening. An argument that simply having a "safety driver" mitigates risk is clearly insufficient based on the tragic fatality in Tempe AZ last year. 
• In general, see: https://grants.nih.gov/policy/humansubjects.htm
• A credible, independently reviewed argument that a road testing campaign provides adequate safety might be enough. But how transparent should that process be? Should we just take a company's word for it that they are doing responsible, safe road testing? For an example of a safety argument approach, see: https://safeautonomy.blogspot.com/2019/04/safety-argument-consideration-for.html

• Expecting Human Drivers to be Super-Human. High-end driver assistance systems might be asking the impossible of human drivers. Simply warning the driver that (s)he is responsible for vehicle safety doesn't change the well known fact that humans struggle to supervise high-end autonomy effectively, and that humans are prone to abusing highly automated systems. This gives way to questions such as:
• At what point is it unethical to hold drivers accountable for tasks that require what amount to super-human abilities and performance?
• Are there viable ethical approaches to solving this problem? For example, if a human unconsciously learns how to game a driver monitoring system (e.g., via falling asleep with eyes open -- yes, that is a thing) should that still be the human driver's fault if a crash occurs?
• Is it OK to deploy technology that will result in drivers being punished for not being super-human if result is that the total death rate declines?

• Confidence in Safety Before Deployment.  There is work that advocates even slightly better than a human is acceptable (https://www.rand.org/blog/articles/2017/11/why-waiting-for-perfect-autonomous-vehicles-may-cost-lives.html). But there isn't a lot of discussion about the next level of what that really means. Important ethical sub-topics include:
• Who decides when a vehicle is safe enough to deploy? Should that decision be made by a company on its own, or subject to external checks and balances? Is it OK for a company to deploy a vehicle they think is safe based just on subjective criteria alone: "we're smart, we worked hard, and we're convinced this will save lives"
• What confidence is required for the actual prediction of casualties from the technology? If you are only statistically 20% confident that your self-driving car will be no more dangerous than a human driver, is that enough?
• Should limited government resources that could be used for addressing known road safety issues (drunk driving, driving too fast for conditions, lack of seat belt use, distracted driving) be diverted to support self-driving vehicle initiatives using an argument of potential public safety improvement?

• How Safe is Safe Enough? Even if we understand the relationship between an aggregate safety goal and self-driving car technology, where do we set the safety knob?  How will the following issues affect this?
• Will risk homeostatis apply? There is an argument that there will be pressure to turn up the speed/traffic volume dials on self-driving cars to increase permissiveness and traffic flow until the same risk as manual driving is reached. (Think more capable cars resulting in crazier roads with the same net injury and fatality rates.)
• Is it OK to deploy initially with a higher expected death rate than human drivers under an assumption that systems will improve over time, long term reducing the total number of deaths?  (And is it OK for this improvement to be assumed rather than proven to be likely?)
• What redistribution of demographics for victims is OK? If fewer passengers die but more pedestrians die, is that OK if net death rate is the same? Is is OK if deaths disproportionately occur to specific sub-populations? Did any evaluation of safety before deployment account for these possibilities?

I don't purport to have the definitive answers to any of these problems (except a proposal for road testing safety, cited above). And it might be that some of these problems are more or less answered. The point is that there is so much important, relevant ethical work to be done that people shouldn't be wasting their time on trying to apply the Trolley Problem to AVs. I encourage follow-ups with pointers to relevant work.

If you're still wondering about Trolley-esque situations, see this podcast and the corresponding paper. The short version from the abstract of that paper: Trolley problems are "too contrived to be of practical use, are an inappropriate method for making decisions on issues of safety, and should not be used to inform engineering or policy." In general, it should be incredibly rare for a safely designed self-driving car to get into a no-win situation, and if it does happen they aren't going to have information about the victims and/or aren't going to have control authority to actually behave as suggested in the experiments any time soon, if ever.

Here are some links to more about applying ethics to technical systems in general (@IEEESSIT) and autonomy in particular (https://ethicsinaction.ieee.org/), as well as the IEEE P7000 standard series (https://www.standardsuniversity.org/e-magazine/march-2017/ethically-aligned-standards-a-model-for-the-future/).

Car Drivers Do More than Drive

How will self-driving cars handle all the non-driving tasks that drivers also perform?  How will they make unaccompanied kids stop sticking their head out the window?

Hey Kids -- Don't stick your heads out the window!

The conversation about self-driving cars is almost all about whether a computer can safely perform the "dynamic driving task." As well it should be -- at first.  If that part isn't safe, then there isn't much to talk about.

But, looking forward, human drivers do more than drive. They also provide adult supervision (and, on a good day, mature judgement) about the operation of the vehicle in other respects. If you've never heard the phrase "stop doing that right now or I swear I'm going to stop the car!" then probably you've never ridden in a car with multiple children.  And yet, we're already talking about sending kids to school in an automated school bus. Presumably the point is to avoid the cost of the human supervision.

But is putting a bunch of kids in a school bus without an adult a good idea?  Will the red-faced person on the TV monitor yelling at the kids really be effective?  Or just provide entertainment for already screaming kids?

But there's more than that to consider. Here's my start at a list of things human drivers (including vehicle owners, taxi drivers, and so on) do that isn't really driving.

Some tasks will arguably be done by a fleet maintenance function:

• Preflight inspection of vehicle. (Flat tires, structural damage.)
• Preflight correction of issues. (Cleaning off snow and ice. Cleaning windshield.)
• Ensure routine maintenance has been performed. (Vehicle inspections, good tires, fueling/charging, fluid top-off if needed.)
• Maintain vehicle interior cleanliness.  And we're not just talking about empty water bottles here. (Might require taking vehicle out of service for cleaning up motion sickness results. But somehow the maintenance crew needs to know there has been a problem.)

But some things have to happen on the road when no human driver is present. Examples include:

• Ensure vehicle occupants stay properly seated and secured.
• Keep vehicle occupants from doing unsafe things. (Hand out window, head out sunroof, fighting, who knows what. Generally providing adult supervision. Especially if strangers or kids are sharing a vehicle.)
• Responding to cargo that comes loose.
• Emergency egress coordination (e.g., getting sleeping children, injured, and mobility impaired passengers out of vehicle when a dangerous situation occurs such as a vehicle fire)

Anyone who seriously wants to build vehicles that don't have a designated "person in charge" (which is the driver in conventional vehicles) will need to think through all these issues. And likely more. Any argument that a self driving vehicle is safe for unattended service will need to deal with all these issues, and more. (UL 4600 is intended to cover all this ground.)

Can you think of any other non-driving tasks that need to be handled?

Other Autonomous Vehicle Safety Argument Observations

Other AV Safety Issues:
We've seen some teams get it right. And some get it wrong. Don't make these mistakes if you're trying to ensure your autonomous vehicle is safe.

Defective disengagement mechanisms. Generally this involves the ability of an arbitrary fail-active autonomy failure to prevent successful disengagement by a human supervisor. As a concrete example, a system might read the state of the disengagement activation mechanism (the “big red button”) as an I/O device fed directly into the primary autonomy computer rather than using an independent safing mechanism. This is a special case of a single point of failure in the form of the autonomy computer.

Assuming perception failures are independent. Some arguments assume independent failures of multiple perception modes. While there is clearly utility in creating a safety case for the non-perception parts of an autonomous vehicle, one must argue rather than assume the safety of perception to create a credible safety case at the vehicle level.

Requiring perfect human supervision of autonomy. Humans are well known to struggle when assigned such monitoring tasks. Koopman et al. (2019) cover this topic in more detail as it relates to autonomous vehicle road testing safety.

Dismissing a potential fault as “unrealistic” without supporting data. For example, argumentation might state that a lightning strike on a moving vehicle is unrealistic or could not happen in the “real world,” despite data to the contrary (e.g. Holle 2008). To be sure, this does not mean that something like a lightning strike must be completely mitigated via keeping the vehicle fully operational. Rather, such faults must be considered in risk analysis. Dismissing hazards without risk analysis based on a subjective assertion that they are “unrealistic” results in a safety case with insufficient evidence.

Using multi-channel comparison approaches for autonomy. In general autonomy algorithms are nondeterministic, sensitive to initial conditions, and have many acceptable (or at least safe) behaviors for any given situation. Architectural approaches based on voting diverse autonomy algorithms tend to run into a problem of deciding whether the outputs are close enough to be valid. Averaging and other similar approaches are not necessarily appropriate. As a simple example, the average of veering to the right and veering to the left to avoid an obstacle could result in hitting the obstacle dead-on.

Confusion about fault vs. failure. While there is a widely recognized terminology document for dependable system design (Avizienis 2004), we have found that there is widespread confusion about the terms fault and failure in practical use. This is especially true when discussing malfunctions that are not due to a component fault, but rather a requirements gap or an excursion from the intended operational environment. It is beyond the scope of this paper to attempt to resolve this, but we note it as an area worthy of future work and particular attention in interdisciplinary discussions of autonomy safety.

(This is an excerpt of our SSS 2019 paper:  Koopman, P., Kane, A. & Black, J., "Credible Autonomy Safety Argumentation," Safety-Critical Systems Symposium, Bristol UK, Feb. 2019.  Read the full text here)

• Avizienis, A., Laprie, J.-C., Randell B., Landwehr, C. (2004) “Basic concepts and taxonomy of dependable and secure computing,” IEEE Trans. Dependability, 1(1):11-33, Oct. 2004.
• Holle, R. (2008) Lightning-caused deaths and injuries in the vicinity of vehicles, American Meteorological Society Conference on Meteorological Applications of Light-ning Data, 2008.
• Koopman, P. and Latronico, B., (2019) Safety Argument Considerations for Public Road Testing of Autonomous Vehicles, SAE WCX, 2019.