World Model Benchmark世界模型基准测试
1Peking University北京大学2Tsinghua University清华大学3Beihang University北京航空航天大学4Shanghai Jiao Tong University上海交通大学5University of Science and Technology of China中国科学技术大学
World models are becoming an important tool for robot learning because they can generate experience, predict future states, and support planning. However, current evaluation of robot world models mainly focuses on a single external view. Robot manipulation is naturally observed through a head camera and cameras on both wrists: the head view shows the workspace and task outcome, while the wrist views reveal local contact, grasp state, and object motion that may be small or hidden globally. A model can therefore produce a plausible head video while failing near the gripper, or generate three strong videos that do not describe the same event.
世界模型正在成为机器人学习的重要基础工具，它能够帮助机器人积累经验、预测未来状态，并支持后续决策与规划。然而，目前针对机器人世界模型的评估仍主要集中于单一外部视角。
在真实机器人操作过程中，机器人通常需要依靠头部相机和双腕相机共同观察环境：头部视角负责呈现整体工作空间和任务结果，腕部视角则能够捕捉局部接触、抓取状态，以及那些在全局视角中容易被忽略或遮挡的物体变化。因此，一个模型可能生成一段整体看似合理的头部视角视频，却在夹爪操作等关键区域出现错误；也可能生成三段表现良好的视频，却无法准确描述同一个连续发生的事件。
We introduce TriWorldBench, a unified benchmark for synchronized head, left-wrist, and right-wrist videos generated by embodied world models. Rather than judging the three videos independently, TriWorldBench treats them as one prediction and evaluates both the quality of each video and whether they describe the same robot action and object state. It reports 19 evaluation signals across six dimensions and summarizes overall performance with TWB-Score, while retaining view-specific results for diagnosis. The benchmark contains 500 synchronized episodes covering 50 robot-manipulation tasks. TriWorldBench provides a framework for tracking progress toward world models that generate one consistent robot world across multiple cameras.
我们提出的 TriWorldBench，是一个面向具身世界模型多视角同步视频生成的统一评测基准。TriWorldBench 不会独立评判三段视频，而是将其视为一次完整预测，同时评估每段视频质量，以及它们是否描述了相同的机器人动作与物体状态。该基准包含 500 个同步片段，覆盖 50 项机器人操作任务，设计了六个维度的 19 项评测指标，并通过 TWB-Score 汇总整体表现。同时，TriWorldBench 保留视角级评测结果，用于进一步分析模型能力，旨在为多相机条件下生成一致机器人世界的世界模型研究提供评估框架。
TriWorldBench is a unified benchmark designed to systematically evaluate embodied world models from three synchronized robot-arm views: head, left wrist, and right wrist. It reports 19 evaluation signals across six dimensions: tri-view consistency, task alignment, physical and 3D coherence, motion quality, temporal consistency, and visual quality. Tri-view consistency is the central dimension. It combines reference-anchored signals, which compare every generated camera with its corresponding view from a consistent ground-truth triplet, with direct semantic judgments of head-wrist and joint tri-view agreement. The remaining dimensions test whether the prediction follows the task, preserves plausible interaction and geometry, exhibits the expected motion in each camera, remains stable over time, and produces usable images. Trajectory-derived STATE annotations connect these evaluations by identifying the action phase, active arm, and expected moving views. TriWorldBench retains per-view, head-wrist pair, and joint tri-view results, additionally reports visual-quality scores grounded by task and consistency performance, and names its final overall aggregate TWB-Score. Together, these dimensions track progress toward embodied world models that generate one consistent robot world across multiple cameras.
TriWorldBench 是一个统一评测基准，用于系统评估具身世界模型在头部、左腕和右腕三个同步机械臂视角下的表现。该基准围绕六个维度设计 19 项评测指标，包括三视角一致性、任务对齐、物理与三维一致性、运动质量、时间一致性和视觉质量。
其中，三视角一致性是核心评估维度。它结合两类信号：
其余维度分别评估模型生成结果是否符合任务要求、是否保持合理的交互过程和几何关系、是否在不同相机视角下呈现符合预期的运动，以及是否能够随时间保持稳定并生成高质量图像。
基于机械臂轨迹构建的 STATE 标注进一步串联这些评估过程，通过识别动作阶段、活动机械臂以及预期运动或静止状态，对模型表现进行更加细致的分析。TriWorldBench 保留逐视角、头部-腕部视角对以及联合三视角的评测结果，并结合任务完成度和一致性表现对视觉质量进行约束，最终汇总形成总体评分 TWB-Score。
这些评测维度共同用于衡量具身世界模型在多相机环境下生成通用基模的能力。
TriWorldBench extends embodied world-model evaluation from single-camera video quality to synchronized robot-world prediction. It evaluates whether head and wrist videos describe one manipulation, assigns each metric to the views where its evidence is reliable, and connects visual quality to task and cross-view correctness.
TriWorldBench 将具身世界模型评估从单一相机视角扩展到多视角同步机器人世界预测。它通过判断头部与腕部视角是否描述同一次机器人操作，并根据不同视角承担的信息价值分配评测指标，将视觉质量与任务完成度、跨视角一致性结合起来。
Evaluates synchronized head, left-wrist, and right-wrist videos as one robot world rather than three unrelated clips. Models must preserve the same robot action, object state, and manipulation phase across all viewpoints.
将同步的头部、左腕和右腕视角视为同一个机器人世界，而非彼此独立的视频片段。模型需要在不同视角中保持一致的机械臂动作、物体状态以及操作阶段。
Routes each metric according to the role of the camera instead of applying the same test blindly to every stream. The head view captures global task progress, the wrist views reveal local contact and geometry, and joint protocols evaluate agreement across views.
根据不同相机视角的功能分配评测指标，而不是对所有视频采用统一测试标准。头部视角用于捕捉整体任务进展，腕部视角用于观察局部接触和几何关系，并通过联合评估判断视角之间的一致性。
Uses reference robot trajectories to identify the action phase, active arm, and expected moving or static views. It penalizes both frozen active views and unwanted motion in inactive cameras.
基于参考机器人轨迹识别动作阶段、活动机械臂以及预期运动或静止视角。同时，对冻结的活跃视角和非活跃视角中的多余运动进行惩罚。
Prevents strong image or aesthetic quality from hiding an incorrect robot rollout. TriWorldBench discounts visual-quality scores when the head-view trajectory is wrong or the synchronized views disagree on robot and object state.
避免图像质量或美学表现掩盖错误的机械臂预测。当头部视角轨迹错误，或同步视角在机器人和物体状态上存在不一致时，TriWorldBench 会降低视觉质量评分。
This section presents representative rollouts generated by participating systems. Each clip is a synchronized multi-view prediction conditioned on a natural-language instruction. Clean and random represent two different video surroundings and backgrounds. The video is composed of three views; from left to right they are left, head, and right. Evaluating the consistency of these three perspectives is the unique feature and important content of this benchmark.本节展示参赛系统生成的代表性预测片段，每个片段均由自然语言指令作为条件，驱动同步多视角预测生成。干净背景与随机背景分别代表两类不同的视频环境设置。每段视频由三个视角组成，从左至右依次为左腕、头部和右腕视角。评估三个视角之间的一致性，是本基准的核心特征与重要组成部分。
TriWorldBench reports a primary composite score and 19 evaluation signals across six dimensions. The hierarchy below retains metric-, dimension-, and view-level evidence for diagnosis.
TriWorldBench 报告一个主要综合得分以及六个维度的 19 个评估信号。下方层级保留指标、维度和视角级证据，用于诊断。
Primary composite score used for leaderboard ranking. It summarizes performance across the six TriWorldBench dimensions while retaining dimension, metric, and view-level results for diagnosis.
用于榜单排名的主要综合得分。它汇总 TriWorldBench 六个维度的表现，同时保留维度、指标和视角级结果以支持诊断。
Measures pixel-level similarity between each generated view and its corresponding ground-truth camera. Because the synchronized ground-truth triplet is consistent, higher matched-view similarity provides indirect evidence of tri-view consistency.
衡量每个生成视角与对应真实相机视角之间的像素级相似性。由于真实三视角序列是同步且一致的，更高的匹配视角相似度可间接证明三视角一致性。
Measures structural similarity between each generated view and its corresponding ground-truth stream, including luminance, contrast, and local image structure. It is less sensitive to exact pixel values than PSNR.
衡量每个生成视角与对应真实视频流在亮度、对比度和局部图像结构上的结构相似性。与 PSNR 相比，它对精确像素值不那么敏感。
Compatibility-first head-to-wrist evaluation of robot state, action phase, and target-object state. Missing or occluded wrist evidence is tolerated, while concrete visual contradictions reduce the score.
以兼容性优先的方式评估头部与腕部之间的机器人状态、动作阶段和目标物体状态。允许腕部证据缺失或被遮挡，但明确的视觉矛盾会降低得分。
Verification-first head-to-wrist evaluation that separately checks phase, contact, temporal progress, spatial context, and object properties. Evidence that cannot be verified from both views is treated as a failed criterion.
以可验证性优先的方式评估头部与腕部，并分别检查阶段、接触、时间进展、空间上下文和物体属性。无法从两个视角验证的证据被视为未通过的条件。
Exemplar-calibrated consistency score guided by positive and negative image examples covering blur, incompatible robot states, incorrect held objects, and use of the wrong arm.
由正负图像示例校准的一致性得分，示例覆盖模糊、不兼容的机器人状态、错误持有物体以及使用错误机械臂等情况。
Measures accuracy on frozen questions constructed from the reference triplet, instruction, first frame, and STATE annotations. Questions cover object identity, spatial relations, active arms, contact states, and phase-specific manipulation events.
衡量基于参考三元组、指令、首帧和 STATE 标注构建的固定问题的回答准确率。问题覆盖物体身份、空间关系、活跃机械臂、接触状态和阶段特定的操作事件。
Measures whether the generated head-view rollout uses the required arm and target object, performs the requested action, and reaches the intended task state. A VLM judgment is normalized to a higher-is-better score.
衡量生成的头部视角预测是否使用指定机械臂和目标物体，完成所要求的动作并达到预期任务状态。视觉语言模型判断被归一化为越高越好的得分。
Measures task-level agreement between generated and reference head videos by comparing VLM-produced descriptions in a shared CLIP text space. It evaluates overall task meaning rather than directly matching the instruction to the candidate video.
通过在共享 CLIP 文本空间中比较视觉语言模型生成的描述，衡量生成与参考头部视频的任务级一致性。它评估整体任务语义，而非直接将指令与候选视频匹配。
Compares generated and ground-truth H/L/R task dynamics using features from a frozen V-JEPA encoder. The fixed camera order preserves view identity, and higher similarity indicates closer task progression across the triplet.
使用冻结的 V-JEPA 编码器特征比较生成结果与真实 H/L/R 三视角任务动态。固定相机顺序保留视角身份，更高相似度代表三元组中的任务进展更接近真实结果。
Evaluates the physical plausibility of robot-object interaction in the head, left-wrist, and right-wrist views. It considers contact behavior, force transmission, grasp stability, object response, and collision or penetration artifacts.
评估头部、左腕和右腕视角中机器人与物体交互的物理合理性。它考虑接触行为、力传递、抓取稳定性、物体响应以及碰撞或穿透伪影。
Assesses 3D plausibility through depth-dependent scale, occlusion, shape stability, lighting consistency, and camera geometry. It evaluates workspace-scale structure in the head view and close-range geometry in both wrist views.
通过深度相关的尺度、遮挡、形状稳定性、光照一致性和相机几何关系评估三维合理性。它在头部视角评估工作空间尺度结构，在两个腕部视角评估近距离几何关系。
Measures whether observed motion in each view follows the active or static pattern specified by STATE. It penalizes missing motion in an active camera and unwanted motion in an inactive camera.
衡量各视角中的观测运动是否遵循 STATE 指定的活跃或静止模式。它惩罚活跃相机缺少运动，也惩罚静止相机出现多余运动。
Summarizes the strength and continuity of optical motion independently for the head, left-wrist, and right-wrist streams. Per-view reporting exposes failures that occur in only one camera.
分别汇总头部、左腕和右腕视频流中的光学运动强度与连续性。逐视角报告能够揭示仅发生在单个相机中的运动失败。
Compares the predicted robot-arm or end-effector path with the ground-truth trajectory using normalized dynamic time warping over 16 head-view frames. Wrist views are excluded because moving cameras and occlusion make image-plane robot tracking unreliable.
使用 16 帧头部视角上的归一化动态时间规整，将预测机械臂或末端执行器路径与真实轨迹进行比较。腕部视角因相机运动和遮挡导致图像平面中的机器人跟踪不可靠，故不纳入该指标。
Measures whether robot and object identity and appearance remain stable across frames in each view. STATE-based adjustment prevents a frozen active view from receiving an artificially high score.
衡量每个视角中机器人与物体的身份和外观是否在跨帧过程中保持稳定。基于 STATE 的调整可防止静止的活跃视角获得虚高分数。
Measures scene stability by comparing background features across the current, first, and preceding frames of each stream. Expected view motion from STATE avoids treating legitimate wrist-camera movement as background failure.
通过比较每个视频流的当前帧、首帧和前序帧背景特征来衡量场景稳定性。STATE 指定的预期视角运动可避免将合理的腕部相机运动误判为背景失败。
Measures motion-aligned appearance and texture stability between adjacent frames. It penalizes flicker, abrupt brightness or color changes, and local texture drift while accounting for expected camera motion.
衡量相邻帧之间经运动对齐后的外观与纹理稳定性。在考虑预期相机运动的同时，它惩罚闪烁、突发亮度或颜色变化以及局部纹理漂移。
Measures frame clarity and technical quality in every camera stream, including blur, noise, overexposure, and compression artifacts. Separate H/L/R scores prevent a clear head view from hiding a degraded wrist view.
衡量每个相机视频流的帧清晰度和技术质量，包括模糊、噪声、过曝和压缩伪影。独立的 H/L/R 得分可防止清晰的头部视角掩盖退化的腕部视角。
Estimates frame-level visual appeal, including composition, lighting, color, and perceptual richness. It is reported separately from task correctness because an attractive video may still show an invalid robot action.
估计帧级视觉吸引力，包括构图、光照、色彩和感知丰富度。它与任务正确性分开报告，因为美观的视频仍可能展示无效的机器人动作。
Browse every published model in one leaderboard. Use a rank range to jump to its block, or locate a model name without changing the displayed ranking.全部已发布模型会保留在同一张榜单中，可通过排名区间跳转，或按模型名称定位，不会改变榜单的显示结果。
| Model模型 | #名次 | TWB-ScoreTWB-Score | Tri-View Consistency三视角一致性 | Task Alignment任务对齐 | Physical and3D Coherence物理与三维一致性 | Motion Quality运动质量 | Temporal Consistency时间一致性 | Visual Quality视觉质量 | Normalized PSNR归一化 PSNR | SSIMSSIM | VLM ConsistencyIVLM 一致性I | VLM ConsistencyIIVLM 一致性II | VLM ConsistencyIIIVLM 一致性III | VQA ConsistencyVQA 一致性 | Instruction Following指令遵循 | Semantic Alignment语义对齐 | JEPA SimilarityJEPA 相似度 | Interaction Quality交互质量 | Perspective透视合理性 | State Alignment状态对齐 | Flow Score光流得分 | Trajectory Accuracy轨迹准确性 | Subject Consistency主体一致性 | Background Consistency背景一致性 | Photometric Smoothness光度平滑性 | Image Quality图像质量 | Aesthetic Quality美学质量 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Top 5前 5 名 | |||||||||||||||||||||||||||
| ennerverseTWB-official | 1 | #153.70 | #171.94 | #252.21 | #249.17 | #138.27 | #259.55 | #220.12 | #164.61 | #176.97 | #181.42 | #175.25 | #189.70 | #343.67 | #438.80 | #287.73 | #230.11 | #226.17 | #272.17 | #163.03 | #135.39 | #216.38 | #574.09 | #281.35 | #123.20 | #222.00 | #118.23 |
| dreamdojoTWB-official | 2 | #251.72 | #269.63 | #156.24 | #343.84 | #327.96 | #160.84 | #121.02 | #359.78 | #471.58 | #277.14 | #273.26 | #288.19 | #147.82 | #249.72 | #188.28 | #130.73 | #128.28 | #359.39 | #260.76 | #36.52 | #116.61 | #277.37 | #189.29 | #215.86 | #124.45 | #217.58 |
| motusTWB-official | 3 | #342.35 | #366.70 | #349.69 | #534.60 | #424.63 | #526.56 | #516.26 | #260.13 | #276.31 | #463.68 | #367.56 | #387.22 | #245.33 | #152.28 | #386.54 | #410.24 | #325.00 | #544.20 | #459.31 | #40.80 | #313.77 | #277.37 | #51.00 | #51.31 | #516.08 | #316.44 |
| genie-envisionerTWB-official | 4 | #440.73 | #462.39 | #533.17 | #154.00 | #520.17 | #432.18 | #317.46 | #458.74 | #374.44 | #369.40 | #557.13 | #577.93 | #536.72 | #520.20 | #579.30 | #50.02 | #520.10 | #187.91 | #559.28 | #50.40 | #50.84 | #191.40 | #41.07 | #44.08 | #319.36 | #415.57 |
| Ctrl-WorldTWB-official | 5 | #538.98 | #557.42 | #443.72 | #434.67 | #229.28 | #346.17 | #416.64 | #546.39 | #564.62 | #546.44 | #463.69 | #482.46 | #440.90 | #338.90 | #481.94 | #310.32 | #424.19 | #445.15 | #360.47 | #221.29 | #46.09 | #477.05 | #349.02 | #312.44 | #418.37 | #514.92 |
The overall radar shows the six category scores on a 0-100 scale. Search by part of a model name to focus the radar and ranking on matching models, then click a model to read its values at the radar vertices. Use the overall, category, or individual-metric buttons to change the ranking view below.总体雷达图基于六个类别得分，以不同的分数尺度展示模型表现。用户可输入部分模型名称，筛选对应雷达图及下方排名结果；点击模型后，可在雷达图各顶点查看对应分数。通过总分、类别或具体指标按钮，可切换下方排名展示视图。
Download the 500-episode test_dataset from the official TriWorldBench Dataset. The Submission Guide defines the required HDF5 trajectories, three initial camera views, task instructions, and episode naming rules.
从官方 TriWorldBench Dataset 下载包含 500 个 episode 的 test_dataset。提交指南说明了所需 HDF5 轨迹、三个相机初始视角、任务指令及 episode 命名规则。
Submissions should follow the official Submission Guide, including its directory structure, video specifications, camera views, file counts, and episode-based filenames. submission_check script is provided to identify common issues before submission, such as missing views, invalid filenames, unsupported video properties, frame-count differences, and unexpected files. During evaluation, every video is normalized to the target frame count: shorter videos are padded, while longer videos are uniformly sampled. Frame-count differences may change temporal alignment with the reference videos and may therefore affect the results.
提交应遵循官方 Submission Guide，包括其目录结构、视频规范、相机视角、文件数量和基于 episode 的文件名。提供 submission_check script 工具用于在提交前识别常见问题，例如缺失视角、无效文件名、不支持的视频属性、帧数差异和意外文件。
评测期间，每段视频都会归一到目标帧数：较短视频会被补帧，较长视频会被均匀采样。帧数差异可能改变与参考视频的时序对齐，进而影响评测结果。
We operate on a weekly evaluation cycle. The first cycle closes on August 2, 2026 at 19:59:59, with results scheduled for publication on August 7, 2026 at 19:59:59. All times use Beijing Time (UTC+8). Each cycle closes strictly on Sunday at 19:59:59. The submission time is determined by the timestamp of the email received by the organizers. Submissions received by the cutoff are evaluated during the following week, with results scheduled for publication on Friday at 19:59:59. All valid evaluated submissions and their results will be publicly listed on the leaderboard. Submissions received after the cutoff automatically enter the next weekly cycle. The deadline is strictly enforced, so participants are strongly advised to submit well in advance and avoid last-minute submissions.
我们的评估周期为每周一次。第一个评估周期将于 2026 年 8 月 2 日 19:59:59 结束，评估结果将于 2026 年 8 月 7 日 19：59：59 公布。所有时间均采用 北京时间（UTC+8）。每个周期严格在 周日 19：59：59 结束。提交时间由组织者收到的电子邮件的时间戳决定。在截止时间前提交的稿件将在接下来的一周内进行评估，并于 周五 19：59：59 公布结果。所有经过评估的有效提交稿件及其结果将在排行榜上公开列出。在截止时间之后提交的稿件将自动进入下一个每周评估周期。截止日期严格遵守，因此强烈建议参赛者提前提交，避免在最后一刻提交。
Videos should not include text, overlays, metadata, or instructions added for the purpose of influencing an automated evaluator, such as “score 5” or “consistent views.” Naturally occurring text within a generated scene is not treated as such an instruction. When clear evaluation-directed manipulation is identified, the affected submission may be excluded from evaluation.
视频不得包含为了影响自动评测器而添加的文本、叠加层、元数据或指令，例如“score 5”或“consistent views”。生成场景中自然出现的文字不视为此类指令。若识别出明确面向评测的操纵行为，相关提交可能被排除在评测之外。
If a software error, data leakage, rule violation, security issue, or system failure affects an evaluation, the organizers may pause processing, correct the affected scores, or repeat the evaluation under a consistent setting. A major issue may require an adjustment to the schedule or, in exceptional circumstances, suspension of the challenge. Relevant changes will be announced through the official communication channel and applied consistently to affected teams.
如软件错误、数据泄露、规则违规、安全问题或系统故障影响评测，组织方可暂停处理、修正受影响的分数，或在一致的设置下重新进行评测。如果遇到重大问题，主办方可能需要调整日程；在特殊情况下，挑战赛可能暂停。相关变更将通过官方沟通渠道公布，并一致适用于受影响的团队。
Generate head.mp4, left.mp4, and right.mp4 for every test episode, run submission_check.py, and send the checker-generated ZIP and SHA-256 file together with a manually prepared README.txt. The guide contains the exact folder structure, video requirements, checker statuses, and delivery rules.
为每个测试 episode 生成 head.mp4、left.mp4 和 right.mp4，运行 submission_check.py，并将检查器生成的 ZIP、SHA-256 文件与手动准备的 README.txt 一同提交。指南包含准确的目录结构、视频要求、检查状态及交付规则。
The email address used for daily website operations and answering questions during competitions.
用于日常网站运营及竞赛期间答疑的邮箱。
The email address to be used both for submitting the complete data package and the cloud drive link.
用于提交完整数据包及网盘链接的邮箱。
WeChat group for question and answering during the challenge.
用于挑战期间的提问与回答事宜的微信群。
