Self-Driving Cars Could Someday Take Requests

Researchers at the Delft University of Technology (TU Delft) in the Netherlands have developed a system that uses a large language model to translate natural-language user requests, such as "I am running late, go fast," into adjustments to a self-driving control system. The researchers posted their preprint on arXiv and are presenting the work at the IEEE Intelligent Transportation Systems Conference in September.

The system does not give users direct control over the vehicle's driving decisions; it tunes the parameters of a safety-aware motion-planning algorithm to keep behavior within safe bounds. It keeps the human in the loop by describing planned changes in nontechnical language and asking the passenger to confirm before making them. When tested in simulation, the system adjusted speed and smoothness in line with natural-language instructions.

The system relies on a model predictive-path integral controller previously developed by the researchers, which identifies multiple possible paths and judges them on criteria including speed, steering angle, and collision probability. The team combined this with OpenAI's GPT-4o-mini model, which uses the user's prompt to rate the relative importance of those criteria, adjusting them up or down around a safe baseline set by the researchers. The scenario description was handwritten for the study but could ultimately come from a car's perception system, says lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft.

The researchers tested the system in the self-driving simulator nuPlan in scenarios involving merging onto a busy highway. Across eight prompts, the system changed controller parameters in ways matching user intent, with requests for comfort increasing smoothness and requests indicating urgency leading to higher speeds.

Nicolas Baumann, a Ph.D. student at ETH Zurich, previously published research in which an LLM tweaked the parameters of a model racing-car controller, allowing driving-style changes and concrete instructions. He says separating the LLM from the main controller means even a hallucination cannot do anything dangerous, though setting constraints requires considerable engineering work. Matthias Althoff, a professor at the Technical University of Munich, says his group built a system that gets an LLM to suggest driving decisions but uses a mathematical process to check them against traffic rules and predictions of other road users, providing verification the Delft paper does not. "As with any LLM, it is not guaranteed that the result is correct," Althoff says. "For that reason, we safeguard the decisions of the LLM in our works."