Submission Guide提交指南

1. Download the 500-episode test_dataset.
2. Run your model on each episode and generate three videos: head.mp4, left.mp4, and right.mp4.
3. Organize the generated videos into the required episode1 to episode500 folder structure.
4. Fill in README.txt with your model and team information.
5. Run submission_check.py on the result folder to validate videos, automatically package the ZIP, and generate its SHA-256 checksum.
6. Send the ZIP, checksum file, and README.txt to the leaderboard submission email.

Leaderboard submission email:

triworld_submit@foxmail.com

The title of the submission email should be formatted as: {model_name}_{method_name}
Here, the method_name corresponds to the Affiliation / Organization / Team Name mentioned in the README.txt.

Download test_dataset from TriWorldBench/Dataset.

After extraction, the dataset should follow this structure:

test_dataset/
├── data/
│   ├── episode1.hdf5
│   ├── episode2.hdf5
│   ├── ...
│   └── episode500.hdf5
├── first_frame/
│   ├── episode1_head.jpg
│   ├── episode1_left.jpg
│   ├── episode1_right.jpg
│   ├── ...
│   └── episode500_right.jpg
└── instructions/
    ├── episode1.json
    ├── episode2.json
    ├── ...
    └── episode500.json

The test set contains 500 episodes, indexed from episode1 to episode500.

• data/episodeN.hdf5 contains the trajectory/action sequence for episodeN.
• first_frame/episodeN_head.jpg, episodeN_left.jpg, and episodeN_right.jpg provide the initial 320x240 observations for the three camera views.
• instructions/episodeN.json contains the task instruction for episodeN.

Submit one result folder containing generated videos for all 500 episodes. The episode ids must match the test dataset exactly.

<your_method>/
├── episode1/
│   ├── head.mp4
│   ├── left.mp4
│   └── right.mp4
├── episode2/
│   ├── head.mp4
│   ├── left.mp4
│   └── right.mp4
├── ...
└── episode500/
    ├── head.mp4
    ├── left.mp4
    └── right.mp4

Each episode must contain exactly three MP4 videos:

• head.mp4
• left.mp4
• right.mp4

The 500 submitted episode folders must correspond one-to-one with test_dataset/data/episode1.hdf5 through test_dataset/data/episode500.hdf5.

Video requirements:

• Resolution: 320x240, matching the images provided in test_dataset/first_frame/.
• Frame count: match the episode sequence length indicated by the corresponding test_dataset/data/episodeN.hdf5 file. In the released test files, this can be read from the first dimension of the trajectory/action datasets, for example joint_action/vector when present.
• FPS: choose according to your own generation or visualization preference. The current evaluation extracts frames from the submitted videos, so FPS does not affect the metric results; if unsure, use 30 FPS.

The checker reports frame-count mismatches as warnings when the difference is within its tolerance, and as errors for large mismatches. Matching the official sequence length is still strongly recommended for reliable evaluation.

Use submission_check.py from TriWorldBench/Dataset to validate and package the result folder from Section 2. The checker takes an uncompressed result folder as input, not a ZIP file.

Please use the ZIP and .sha256.txt generated by this script as the final submission files. The checker validates the official episode set, checks video views, reads video metadata with ffprobe, reports metadata issues, and writes the packaged output.

The checker requires Python 3.10+ and ffprobe available on PATH. It reports videos whose resolution does not match 320x240 as errors.

python submission_check.py /path/to/<your_method> --output /path/to/submission_output

Checker status:

• PASS: all required checks pass; the checker packages the ZIP and .sha256.txt automatically.
• WARNING: non-blocking issues are found; the checker asks whether to continue packaging. You may confirm to proceed, or stop, fix the reported issues, and run the checker again.
• ERROR: blocking issues are found; the checker does not package the submission.

One complete submission consists of three parts sent together in one email to the official challenge address:

1. The final .zip — the ZIP generated by the checker, either attached to the email or provided through an external download link in the email body. Its top level contains episode1/ through episode500/.

Expected ZIP layout after extraction:

   episode1/
   ├── head.mp4
   ├── left.mp4
   └── right.mp4
   episode2/
   ├── head.mp4
   ├── left.mp4
   └── right.mp4
   ...
   episode500/
   ├── head.mp4
   ├── left.mp4
   └── right.mp4

1. The checker-generated .sha256.txt file — for an input folder named <your_method>, the checker writes <your_method>.sha256.txt (see Section 4).
2. A README.txt file — attached to the email (see Section 5).

The two text files (.sha256.txt and README.txt) must remain outside the ZIP; they are attached to the email as separate files alongside the ZIP or its download link.

The submission checker automatically generates <your_method>.sha256.txt together with <your_method>.zip. The checksum file contains one line with the SHA-256 hash of the generated ZIP:

<sha256_hash>

Do not modify or replace the ZIP after the checker generates the checksum. Any change to the ZIP changes its checksum and invalidates the .sha256.txt file.

The README.txt attached to the email provides the information shown on the challenge website. Please use the following structure:

Model name:                         # Name displayed on the website
Contact email:
Affiliation / Organization / Team Name:
Project / Code repository:          # Optional
Brief introduction:                 # Optional, no more than 100 words

• The submitted Model name and Affiliation / Organization / Team Name will be displayed on the challenge website together with the public result,at most 30 characters.
• Participants can freely choose the Affiliation / Organization / Team Name. They can decide for themselves whether to disclose the actual name of the institution or team. However, the organizers are not responsible for this name and will not conduct any verification of its authenticity or authority.
• The Contact email is used for submission communication and does not need to be displayed publicly unless stated on the official challenge website.
• README.txt is not generated by submission_check.py; please create and attach it manually.
• Please ensure the display information is accurate.

• The Email and Model name jointly identify a model's competition record, and a single Model name can only be associated with one email address.If a registered model is used by another email address, a notification will be sent by email instructing the participant to change the Model name and resubmit.
• The submitted model in each cycle is based on the latest compressed file or link. Please ensure that the last submitted model and information in each submission cycle are valid.
• If the same Email + Model name already exists in the database, the score will only be updated if the uploaded TWBScore is higher; if the existing record has a higher score or is the same, no update will be made.

Send one complete email to the official challenge address containing the ZIP (attached or via an external link in the email body), the .sha256.txt, and the README.txt.

• A submission is considered complete only when all three required parts are included.
• External download links are accepted only from Baidu Netdisk or Google Drive.
• The link must point to the specific ZIP file, not a folder or a generic page.
• Include any required access code in the email body.
• If the ZIP is provided through an external download link, keep the linked file unchanged after submission and make sure the link remains accessible during the evaluation period.
• Please refer to the official challenge website for submission schedule and evaluation-cycle details.