Number of miles as a self-driving car progress and safety metric (Metrics Episode 1)

Even if you have the best possible safety drivers, every test mile adds some risk. Make sure every mile of road testing is actually doing something important.

You hear people saying that testing for lots and lots of miles must mean some self-driving car company is better than the rest, or at least in the lead in the so-called race to autonomy. But not so fast, there’s more to it than that.

Some companies have millions of miles of road testing experience, and to be sure that’s an impressive accomplishment. If they have that many miles, certainly they have incentive to boast about it in saying, “Look how many miles we have.” And the press often says, “All right, these guys have lots and lots of miles, somehow that must be in they’re ahead.” But miles really doesn’t tell you who’s safer or even necessarily ahead. Miles is a mostly reflection about the resources they have available to deploy a test fleet.

If you have lots of money, you can buy a lot of cars, hire lots of people, and put them out there to rack up the miles. Sure, having lots of resources makes it easier to make progress, but it doesn’t tell you who’s better. In fact, some companies are taking pride in reducing their testing road miles and instead putting those resources into simulation and other sorts of engineering activities other than road testing. It’s hard to believe that if you don’t have any miles or you only have a few miles that you’re really ready to deploy, I get that. And it’s hard to believe that a company without lots of data collection miles is actually seeing all the things that are going to need to deal with when they pick an operational design domain. But nobody will ever get to the billions of representative miles necessary to say anything compelling about expected safety until after they actually deploy their fleet.

Even if you have a lot of miles, not all miles are created equal. Okay, so we have the billion miles -- are they in simulation around the same block? Are they all in sunny weather -- or do they include rain and hail and ice and all those other types of weather conditions you care about? Are they in a place with wide roads and no pedestrians or are they in a chaotic urban center? Oh, was that chaotic urban center at 5:00 AM? Was it during rush hour? Did you include things like construction zones or Halloween costumes or all sorts of things that don’t happen that often, but that you have to handle the right way? If somebody wants to talk about miles, they should also talk about how those miles show they’ve covered the entirety of their operational design domain.

There’s a potential problem with using miles as a measure of progress because they can motivate the wrong behavior. If you’re judged solely on how many miles you’ve racked up, then you’re going to optimize the easy miles perhaps, but worse, every mile on public roads is a chance to make a mistake. Even if you have the best possible safety drivers, every test mile adds some risk. Hopefully that risk is no worse than human driver vehicle risks, but that’s a different discussion. So every mile costs not only money, but also puts you at some risk of some adverse news or some unfortunate event. So you should think carefully about racking up a lot of miles for the sake of miles; you should make every mile earn its keep. Make sure every mile of road testing is actually doing something important.

Now you have to have miles. For sure you won’t know if your system is done until you’ve done some road testing. But the early miles don’t have to be about testing at all, they can be about data collection. You don’t need to run an autonomous system to collect sensor data. You can just collect sensor data with a human driver and have risk no different than someone just driving around normally. That means the road test miles should not be used for primary data collection, but rather as a way to confirm your design is solid. So don’t think of miles on the road is as debugging problems with your system while you’re on public roads. Rather think about road testing miles as a way of making sure you didn’t overlook anything in a system you think is just about ready to go. That means instead of going for road test miles, companies should instead be trying to get data collection miles for most of the miles and making sure they cover the ODD.

Then they can feed that information to a simulation, make sure the system handles all the miles properly, and then after they’re pretty sure they got it right, they should be doing road testing miles simply to confirm that all the engineering effort they put in resulted in a system that behaves the way they expected it to. In other words, road testing miles should be the boring part of tying a ribbon around and putting a bow on a great design. Road testing miles should not be the pointy end of the spear for development.

Going back up to why people talk about miles as a progress metric, sure, having no miles on public roads probably means you’re not ready to deploy because you haven’t checked to make sure it works. But having a ton of miles doesn’t mean you’re ahead, it just means you’re well-funded and you’re out there operating. If you really want to know how someone’s doing, it isn’t just the miles but rather it’s which miles, and it’s how those miles go together with all the other engineering activities to make sure that they’re taking a solid engineering approach to designing a safe self driving car.

For the podcast version of this posting, see: https://archive.org/details/metrics-02-miles-metrics

Thanks to podcast producer Jackie Erickson.