PlanAgent: A Multimodal Large Language Agent for Closed-Loop Vehicle Motion Planning

Vehicle motion planning. is an essential component of autonomous driving technology. Current rule-based vehicle motion planning methods perform satisfactorily in common scenarios but struggle to generalize to long-tailed situations. Meanwhile, learning-based methods have yet to achieve superior performance over rule-based approaches in large-scale closed-loop scenarios. To address these issues, we propose PlanAgent, the first mid-to-mid planning system based on a multimodal large language model (MLLM). MLLM is used as a cognitive agent to introduce humanlike knowledge, interpretability, and common-sense reasoning into closed-loop planning. Specifically, PlanAgent leverages the power of MLLM through three core modules. First, an environment transformation module constructs a bird’s eye view (BEV) map and a lane-graph-based textual description from the environment as inputs. Second, a reasoning engine module introduces a hierarchical chain-of-thought from scene understanding to lateral and longitudinal motion instructions, culminating in planner code generation. Last, a reflection module is integrated to simulate and evaluate the generated planner for reducing MLLM’s uncertainty. PlanAgent is endowed with the common-sense reasoning and generalization capability of MLLM, which empowers it to effectively tackle both common and complex long-tailed scenarios. Our proposed PlanAgent is evaluated on the large-scale and challenging nuPlan benchmarks.