System helps humans predict when self-driving cars will make mistakes

Researchers from MIT and autonomous vehicle technology company Motional developed a new method that provides explanations of a self-driving car's underlying model decisions, according to MIT News. The method, called the Concept-Wrapper Network (CW-Net), translates the internal reasoning of a deep learning model into understandable concepts, such as "approaching stopped vehicle" or "close to cyclist," without altering the vehicle's driving performance.

Machine-learning-based planners act as the "brain" of a self-driving car, processing camera and lidar data and outputting a trajectory. The planners are usually black-box models, making their decisions difficult to understand. CW-Net is a concept classifier that is plugged into the middle of an existing planner architecture. It forces the final piece of the planning model to use classified concepts when deciding the vehicle's next action, and outputs explanations alongside the vehicle trajectory in real time. The researchers trained CW-Net on a dataset of 130 million examples of self-driving car scenes with multiple labeled concepts per scene.

In road tests on a private track with a Motional robotaxi and a safety driver, CW-Net explanations helped the safety driver better predict vehicle behavior in surprising situations. In one instance, the vehicle consistently stopped when approaching a cyclist. The safety driver assumed it stopped because it detected the cyclist, but CW-Net revealed the model was not properly configured to detect the cyclist and chose a trajectory that would have caused a collision; it stopped because emergency braking engaged when it got too close. Armed with this information, the safety driver could reduce speed or engage manual driving sooner in similar situations, which could also help engineers fix the model. Larger online simulation studies using real driving situations captured in Las Vegas yielded similar results, with CW-Net explanations significantly improving participants' ability to predict vehicle behavior.

The research appears today in Nature. Julie Shah, an MIT professor of aeronautics and astronautics, director of the Interactive Robotics Group in CSAIL, and co-senior author, said the work shows how explanations support the human's mental model and understanding of a system's behavior and how it could be used in engineering and development to improve technology. She is joined on the paper by lead author Eoin Kenny, a former MIT postdoc now a senior AI researcher at J.P. Morgan Chase; co-senior author Momchil Tomov, a staff research scientist at Motional; and Motional team members Akshay Dharmavaram, Sang Uk Lee, Tung Phan-Minh, Shreyas Rajesh, Yunqing Hu, and Laura Major, president and CEO of Motional. The researchers could in the future extend CW-Net to cover more concepts and explore different training and design techniques.