Set the starting point
Initialize the ego vehicle's placement and state.
NEURIPS 2026 / ACCEPTED
SEE THE CAPABILITY. UNDERSTAND THE DRIVE.
DriveHierarchy.
A Benchmark for Diagnosing VLM Driving Capabilities
from Open-Loop Understanding to Closed-Loop Execution
1 Tongji University2 UNC Chapel Hill
DISCOVER THE BENCHMARK ↓OPEN-LOOP UNDERSTANDING → CLOSED-LOOP EXECUTION
01 / OVERVIEWBEYOND A SINGLE SCORE
What does it take
to understand the drive?
Driving intelligence is more than one ability. Its evaluation should be, too.
DriveHierarchy connects perceptual grounding, contextual memory, and mental reasoning to closed-loop driving. It reveals where VLMs succeed, where they struggle, and which capabilities offer targets for improvement.
4Capability ranks
76,798Full-corpus QA pairs
84,279Full-corpus frames
100Closed-loop scenarios
15VLMs evaluated
Paper evaluation: 14,000 open-loop records across 14 tasks, with 1,000 records per task. Full-corpus statistics are shown above.
View full figure ↗Read the research abstract ＋
Evaluating VLM-based autonomous driving remains difficult because driving competence is composite: a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future evolution, and act appropriately under closed-loop interaction. DriveHierarchy organizes these capabilities into four ranks. It integrates multiple open-source autonomous-driving datasets into a unified open-loop benchmark and provides a closed-loop simulation platform with interactive scenario construction on a real-world road network. Experiments on 15 VLMs reveal structured but non-redundant capability variation, connect open-loop understanding to closed-loop driving, and support targeted diagnosis and benchmark-guided optimization.
Condensed from the paper abstract.
02 / THE CAPABILITY HIERARCHYUNDERSTANDING → ACTION
Four ranks.
One connected picture.
Explore the abilities behind driving behavior.
Select a rank to see what it measures.
OPEN-LOOP / VISUAL OBSERVATIONR1 · 8 OPEN-LOOP TASKS
Ground the scene.
Recognize traffic participants and hazards, estimate distances, localize referred objects, and describe the current traffic situation.
- Object existence, counting, and attributes
- Nearest-object, referred-object, and bucketed distances
- Visual grounding and situation description
03 / INTERACTIVE SCENARIO EDITORNO PROGRAMMING REQUIRED
Your scenario.
Your design.
Turn a driving question into a custom scenario.
Visually configure actors, trajectories, and conditions without writing scenario scripts.
View full platform ↗Shape the interaction
Place vehicles, pedestrians, and other dynamic actors around the ego vehicle.
Design the movement
Edit actor trajectories to create merging, yielding, and crossing interactions.
Change the conditions
Configure weather and refine the scene around your research question.
DESIGN→CARLA + SUMO→EVALUATE→REPLAY & ANALYZE
CARLA supplies rendering, vehicle physics, and sensor simulation.
SUMO coordinates background traffic. Logs and replay help you investigate failures.
INSIDE THE EDITOR
Design directly on the road network.
The actual visual interface for configuring your driving scenarios.
Explore the interface ↗04 / SCENARIO EXPLORER100 SCENARIOS · 10 FAMILIES
A world of
driving interactions.
Built on a real-world road layout.
Curated to challenge interactive execution.
Enlarge scene ↗Pedestrian encounters
Detect crossing intent, decelerate in time, yield when required, and resume only after the conflict region is clear.
Explore the test-site scenario map ＋
05 / EXPERIMENTAL RESULTS15 OPEN-SOURCE VLMS
Different strengths.
Measurable differences.
Generalist and driving-specialized models,
evaluated across understanding and execution.
| Rank | Model | Parameters | Open-loop score ↑ | Closed-loop score ↑ |
|---|
15 models · Tables 1 & 2 in the paper. The two scores use different evaluation protocols and should not be compared numerically across columns.
CAPABILITY PROFILESA score tells you how much.
A score tells you how much.
A profile shows you where.
Compare task-level strengths within a model family or across similarly sized models.
Figure labels R2-1-A/B/C correspond to repository tasks R2_1, R2_2_A, and R2_2_B. These profiles cover R1–R3.
View full figure ↗06 / FROM DIAGNOSIS TO IMPROVEMENTBENCHMARK-GUIDED FINE-TUNING
Find the weakness.
Improve the drive.
Open-loop capability supervision can transfer to closed-loop behavior.
In a Qwen3-VL-8B-Instruct case study, jointly fine-tuning on the identified weak capability groups raises the average R4 score from 0.902 to 8.021. R4 was never a supervision target.
+7.119R4 score improvement
in the joint intervention
in the joint intervention
One base model, under the paper's protocol. Table 4 reports the closed-loop transfer study.
OPEN-LOOP ↔ CLOSED-LOOP
Related, yet not interchangeable.
Spearman correlations with R4 across the evaluated models.
0.664R1 ↔ R4
0.596R2 ↔ R4
0.418R3 ↔ R4
07 / BUILD ON DRIVEHIERARCHYRESOURCES & CITATION
Your next driving
research starts here.
Get started on GitHub ↗01 / PAPERRead the researchManuscript PDF · 41 MB ↗02 / DATAExplore the benchmarkData & simulator assets ↗03 / EVALUATIONEvaluate your modelSetup, inference & scoring ↗
BIBTEX
@inproceedings{xu2026drivehierarchy,
title = {{DriveHierarchy}: A Benchmark for Diagnosing {VLM} Driving
Capabilities from Open-Loop Understanding to Closed-Loop Execution},
author = {Xu, Chengkai and Liu, Jiaqi and Guo, Yicheng and Hang, Peng and Sun, Jian},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://github.com/PerfectXu88/DriveHierarchy}
}Citation based on the accepted manuscript; final proceedings metadata can be added when available.