User Goals
- Convert driving simulations into photorealistic video.
- Scale synthetic data generation to support verification and validation of autonomous vehicles.
Current State and Key Challenges
cogniBIT is a Munich-based startup that develops cognitive traffic agents for the validation of automated driving and driver-assistance systems. Our software, cogniBOT, creates controllable and repeatable scenarios involving vehicles and other road users. The simulated agents behave based on insights from neuroscience, which enables them to model human perception, decision-making, and behavior more realistically than conventional traffic agents while also ensuring that cogniBOT remains computationally tractable for large-scale simulations. The scenarios are independent of any visualization platform.
In the project for which we use HammerHAI resources, we use the open-source autonomous driving simulation software CARLA to visualize our scenarios. We export synchronized RGB, depth, edge, and semantic-segmentation data. We then evaluate generative video models that transform CGI outputs into photorealistic dashcam-style footage while preserving the logic of the scenario being investigated. This allows us to obtain photorealistic footage of any scenario we wish, including the ones including safety-critical events. This addresses a key industry challenge, as safety-critical events are rare, costly, and difficult or impossible to reproduce in real-world testing. Photorealistic outputs could help autonomous vehicle (AV) deve lopers test their perception and decision-making software against scenarios that might not occur during normal operational testing.
We require HammerHAI’s computing infrastructure and AI expertise to overcome limitations of other available computing systems, including their limited computational power and insufficient GPU-memory capacity. Being able to use HammerHIA resources is essential for cogniBOT‘s ability to fulfil the objectives of this project, which is to obtain photorealistic videos from CGI inputs that maintain temporal consistency and visual fidelity.
Support from HammerHAI
Through HammerHAI, we seek access to high-performance NVIDIA GPU infrastructure and expert support in selecting, configuring, and evaluating generative video models. This work requires intensive computing resources that would be prohibitively expensive for us at standard commercial rates. We currently use NVIDIA Omniverse and Cosmos, making NVIDIA based systems particularly suitable.
Intended Value and Impact
Our goal is to design a reliable pipeline that transforms controllable simulated scenarios into realistic sensor videos while preserving the underlying ground
truth. This would let AV developers test perception an downstream driving functions on diverse, repeatable, safety-critical situations without depending solely on costly road recordings. Faster video generation and systematic quality metrics
could shorten autonomous vehicle development cycles and improve confidence
in simulation-based validation. This also addresses a very serious shortcoming of the EU-based AV companies.
Current Status within HammerHAI and Next Steps
Collaboration with HammerHAI is in an early testing phase. We have prepared example simulation outputs and carried out initial experiments using existing GPU resources at the High-Performance Computing Center Stuttgart (HLRS). These tests helped us understand what works and where the main difficulties remain. The next steps are to test additional models and input types, find the proper hyper-parameters for the model and define clear evaluation criteria. We would then study whether the approach can scale to larger numbers of scenarios.
Organizations Involved
Contact
Dr. Vahid S. Bokharaie, Chief Scientist at cogniBIT, vb@cognibit.ai
