The paper
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation (arXiv 2610.35864). Twenty-four authors, with Muxi Diao, Wenbin Xu and Lele Yang listed first and Weiran Xu, Kongming Liang and Zhanyu Ma listed last. The project page lists the institutions as BUPT, ZGCA, PKU, JLU and Kling, without saying which author belongs where. Version 1 was posted on 26 October 2026 and a revised version 2 followed on 27 October.
The finding in plain English
You can now ask an AI coding agent to make a finished video, with picture and sound, by writing code that draws every frame and generates every sound. No video-generation model is involved. The researchers set nine frontier agent systems loose on 239 video tasks and scored the results.
Three things stand out:
The best agents are good, but not reliable. GPT-6 Astra and Claude Opus 5.5 ranked first and second. Even so, the strongest system satisfied every core requirement on 76.3% of tasks. Roughly one video in four still missed something the brief asked for.
Most lost videos were lost to the clock. When an agent failed to hand over a video at all, about three quarters of the time the cause was the final render: it started too late, ran too slowly, or never started. In those cases, time ran out at the export step.
Turning up counts. Scoring a missing video as zero changes the leaderboard. On the project page, the authors say Opus delivered all 239 videos and passed every program check on 99.6% of tasks, Astra missed a single video, and Gemini 4 Argon delivered 184.
For anyone making explainers, data stories or lyric videos with AI, the lesson is practical. Code-built video handles exact charts, formulas and on-screen text well, and the main way it fails is poor time management. You can manage time yourself.
How they did it
The authors start from a known weakness of diffusion video models: they struggle to render exact formulas, correct charts, games that follow their own rules, or narration that lines up with what happens on screen. Code does those things precisely. So the benchmark asks agents to build the whole video as a program.
The setup, from the abstract and project page:
Tasks: 239 self-contained tasks in English and Chinese, spread across 12 content categories. Some tasks supply local files; others require the agent to search for and fetch real-world assets.
Rules: each run gets one hour inside an identical Docker sandbox with 8 CPUs and no GPU, and no image or video generation model. The output must be a playable MP4 with sound. Agents can use whatever software stack they choose.
Contestants: nine agent systems, each run in its own native harness (the page names Claude Code, Codex, Antigravity and Kimi Code) at maximum effort, for 2,151 scored runs in total.
Scoring in three layers: a program check (was a playable video delivered in time?), about 13,000 task-specific yes/no rubric checks, and five quality dimensions scored 1 to 5.
Judges: three panels of vision-language models watch each video at 4 frames per second with the soundtrack. Because Gemini models also compete, one panel pairs GPT-5.6-sol for the picture with Qwen3.8-omni for the sound as a cross-check. The authors report that human preferences and all three judge panels produce similar rankings of the nine systems.
The rule that matters most for the results: a run with no video fails every check and scores zero on every dimension. That is why delivery, and therefore time planning, moves the rankings.
Why it matters
Most creators meet AI video through text-to-video models. Those produce striking footage, but they are a poor fit for anything that has to be exactly right: a chart with real numbers, a maths step, a product price, captions in sync with a voiceover. Code-driven video is the other route. The agent writes an animation (in HTML and JavaScript, Python, or a dedicated animation library), generates or fetches audio, and renders the result to a file. Every number on screen is the number in the code.
This benchmark is a large, public attempt to measure how well agents do that whole job end to end, including the dull last step of exporting the file. Two results are useful beyond the leaderboard.
First, roughly three quarters of the best system's tasks came out fully correct. That is good enough to use for first drafts and repeatable formats, and not yet good enough to publish without watching the result.
Second, the biggest cause of missing videos is something you can fix with instructions and process, without waiting for a better model. It also fits a pattern from our recent coverage of agents that pass your test but break the rules: what you check, and when, shapes what an agent actually delivers.
Caveats
Brand new and not yet peer reviewed. The paper went up on 26 October and was already revised the next day.
Code and data are not out yet. The project page lists both the GitHub code and the Hugging Face dataset as "coming soon", so you cannot rerun the tasks yourself today.
AI judges do most of the scoring. The authors report agreement with human rankings, but quality scores still come from models. We have covered how AI judges can disagree on subjective calls, so treat small gaps between systems with care.
One fixed setup. One hour, 8 CPUs, no GPU. With a GPU or more time, render bottlenecks would look different, and so might the ranking.
Self-reported results. All figures here come from the authors' own paper and project page.
How to use it today
For creators making videos with AI agents
If you already use a coding agent such as Claude Code or Codex, you can try code-built video this week. It suits explainers, data stories, quote cards with motion, lyric videos and product walkthroughs: anything where the words and numbers on screen must be right.
A starting prompt:
Make a 45-second vertical video explaining [topic]. Build it as code (an HTML/JavaScript animation or a Python animation library), add a voiceover track and background music, and export an MP4 with sound. First, render a rough low-resolution version of the full video within the first 15 minutes and tell me where it is. Then improve it. Stop adding features with 15 minutes left and do the final full-quality render.
That last part is the point of this paper. Most missing videos were the final render left too late, so make the agent produce a complete rough cut early. If time runs out, you still have a video.
Then check the result yourself before publishing. A one-minute checklist:
Does the file play from start to finish, with sound?
Are every number, name and spelling on screen correct?
Does the narration line up with what is on screen at three moments you choose at random?
Is the length and aspect ratio what you asked for?
Is any fetched image or music something you are allowed to use?
If you make images with lots of text, our piece on MOSS-Image covers the same exact-text problem for still graphics.
For builders
Put the time budget in code. If you run video agents in a pipeline, enforce a checkpoint: a full low-resolution render must exist by a set time, or the job alerts you. Kill and fall back to the last good render at the deadline, so a run never ends with nothing.
Measure render speed early. Time a 5-second test render at the start of each job and extrapolate. If the full render will not fit, drop resolution or frame rate before the agent spends its time on polish.
Score delivery separately. Copy the benchmark's layers in your own tests: did a playable file arrive, did it pass your yes/no checks (length, captions present, correct figures), and only then how good it looks.
Turn your brief into checks. The roughly 13,000 rubric checks are a useful model. Write five to ten yes/no checks per video format you produce, and have a model or script run them on every output.
Watch for the release. Once the code and dataset are published, you can run the tasks closest to your own formats against the agents you are considering.
