Showing posts from November, 2020

Disengagements as a progress metric is a bad idea (Metrics Episode 2)

  We should be worried about road testing safety metrics, not disengagements. A disengagement happens when the autonomy in a self driving car detects an internal problem, or a human test driver takes over control of a self driving car test platform because of safety concerns. Self driving car developers have to report these disengagements, for example, to California. The apparent rationale for requiring these reports is that all things being equal, disengagements per mile might decrease over time as technology matures. Along those lines, eventually when disengagements reached zero, you might think it’s time to deploy the vehicle without a human test driver. The problem is that this model is much too simplistic and more importantly, not all things are equal. Let’s start with some basics. Not all miles are equal. If you wanted to game disengagements, you could do so by driving around an empty block in beautiful weather at 4:00 AM with no traffic, no pedestrians, nothing on the road, arou

Number of miles as a self-driving car progress and safety metric (Metrics Episode 1)

Even if you have the best possible safety drivers, every test mile adds some risk.  Make sure every mile of road testing is actually doing something important. You hear people saying that testing for lots and lots of miles must mean some self-driving car company is better than the rest, or at least in the lead in the so-called race to autonomy. But not so fast, there’s more to it than that. Some companies have millions of miles of road testing experience, and to be sure that’s an impressive accomplishment. If they have that many miles, certainly they have incentive to boast about it in saying, “Look how many miles we have.” And the press often says, “All right, these guys have lots and lots of miles, somehow that must be in they’re ahead.” But miles really doesn’t tell you who’s safer or even necessarily ahead. Miles is a mostly reflection about the resources they have available to deploy a test fleet. If you have lots of money, you can buy a lot of cars, hire lots of people, and put t