ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation

Teaser figure for ECoSim showing data-efficient multi-modal control of pretrained traffic models.

Fig. 1: Data-Efficient Multi-Modal Control of Pretrained Traffic Models. We introduce a data-efficient adaptation framework that steers frozen traffic models via sketch, latent, or text signals. Using only 1% of the paired control data, the framework adds new control modalities and enables targeted generation of diverse user-specified traffic scenarios while preserving the pretrained model's performance.

Abstract

Controllable traffic simulation is critical for testing autonomous driving systems, yet existing approaches often require retraining large generative models with extensive annotated data. We introduce a lightweight control adaptation framework that enables multi-modal controllability (sketch, latent behavior codes, and text) for pretrained state-of-the-art diffusion and autoregressive traffic models.

By modulating intermediate features through identity-initialized FiLM layers, our method efficiently adds new control modalities while preserving the base model's generative prior. Evaluated on Waymo Open Sim Agents Challenge, our approach demonstrates strong controllability with less than 1% of the paired control data.

Through context-aware condition transfer, our framework enables counterfactual scenario generation and long-tail synthesis while maintaining stable closed-loop driving realism and safety. Our framework unlocks new possibilities for controllable traffic simulation, enabling targeted scenario generation through lightweight adaptation of pretrained generative models.

Video

Method Overview

Model-agnostic control adaptation architecture diagram.

Fig. 2: Model-Agnostic Control Adaptation Architecture. A lightweight adapter injects multi-modal control signals into a frozen pretrained traffic model by predicting FiLM parameters (γ, β) that modulate intermediate features h. The design is compatible with both autoregressive and diffusion backbones, and identity initialization preserves the base model's generative prior.

Fig. 3: BehaviorVAE overview. Agent trajectories and scene context are encoded into a Gaussian latent posterior; reparameterized per-agent latents are decoded for trajectory reconstruction and exported as paired latent codes in a single forward pass.

Context-match retrieval pipeline diagram.

Fig. 4: Context-Match Retrieval Pipeline. Query agents are encoded into a shared embedding space. Following heuristic filtering for dynamic feasibility, candidates are ranked via similarity scoring to retrieve the Top-K environmentally compatible scenarios from the dataset.

Long-tail Scenario generation

Select scenario

Base

Query

Control

BibTeX

@inproceedings{ecosim,
  title={ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation},
  author={Chen, Yu-Hsiang and Chang, Wei-Jer and Chen, Yi-Ting and Tomizuka, Masayoshi},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

For more details, please check our paper.