Add OmniDreams evaluation and regression for long running outputs

[jarcherNV]

• Add resumable HF scene discovery, batching, staging, generation, and validation tooling
• Add DrivingGen FVD-lite integration with split-specific reference staging
• Capture machine-readable evaluator logs/results for regression tracking
• Add tests for eval manifests, staging, validation, and FVD adapter behavior
• Add WorldLens checkout, staging, evaluation, and result collection helpers
• Add CLI commands for setup-worldlens, prepare-worldlens, and worldlens-evaluate
• Add tests for WorldLens config, staging, manifest, and result handling

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jarcherNV]

This is mostly just a prototype for quality regression tests using the method from the DrivingGen paper. It's not clear to me yet if this is actually useful or not, but it does seem to work at least.

[greptile-apps]

Greptile Summary

Adds a complete omnidreams-eval CLI and supporting library (omnidreams/eval/) that automates resumable batch evaluation of OmniDreams scenes: HuggingFace discovery → batch planning → staging → FlashDreams generation → validation → DrivingGen FVD-lite → WorldLens consistency metrics → Markdown/JSON summary report.

• All previously flagged issues (bare FileNotFoundError in validate_generated_run, KeyError in parse_byte_size, CalledProcessError escaping _cmd_generate, TypeError in summarize-run for None frame counts, and missing generation.json on resume) have been addressed in this revision.
• The DrivingGen integration patches the cloned checkout in-place to make checkpoint paths configurable via environment variables, and the FVD calculation runs in-process via _calculate_styleganv_fvd with sys.path and CWD temporarily mutated and restored.
• The WorldLens adapter writes a Hydra config, stages frame-matched reference clips alongside generated ones, and collects per-artifact metric JSONs after evaluation.

Confidence Score: 5/5

Safe to merge; all previously blocking defects are resolved in this revision.

Every issue raised in prior review rounds has been corrected and is covered by the new test suite. The one remaining note is a sys.modules caching concern in the in-process FVD computation path, which only matters when multiple drivinggen_root checkouts are used in the same Python process.

integrations/omnidreams/omnidreams/eval/drivinggen.py — specifically _calculate_styleganv_fvd and its importlib.import_module call.

Important Files Changed

Filename	Overview
integrations/omnidreams/omnidreams/eval/cli.py	784-line CLI with 15 subcommands; generation error-handling updated to raise RuntimeError (not CalledProcessError) and is caught correctly; all previously flagged issues addressed.
integrations/omnidreams/omnidreams/eval/drivinggen.py	DrivingGen checkout, patch, staging, and in-process FVD-lite path; the importlib.import_module call in _calculate_styleganv_fvd caches the StyleGAN-V module in sys.modules, which can cause the wrong code to execute on repeated calls with different drivinggen_root values in the same process.
integrations/omnidreams/omnidreams/eval/generation.py	Generation orchestration; resume path now correctly writes missing generation.json when the video already exists; all previously flagged issues resolved.
integrations/omnidreams/omnidreams/eval/report.py	Run summary builder; None-guarded set comprehensions prevent TypeError in sorted() for missing frame counts; previously reported issue is resolved.
integrations/omnidreams/omnidreams/eval/validation.py	Validation now checks generated_root.exists() before iterdir(), resolving the previously reported bare FileNotFoundError.
integrations/omnidreams/omnidreams/eval/worldlens.py	WorldLens checkout, config, staging, and result collection; incremental manifest merging is well-handled; frame-count alignment between generated and reference videos is correct for WorldLens.
integrations/omnidreams/tests/test_eval_tooling.py	1129-line test covering manifests, batches, staging, validation, DrivingGen adapter, and WorldLens; includes regression for the off-by-one frame-skip guard and the missing-metadata resume path.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[discover] -->|manifest.jsonl| B[plan-batches]
    B -->|batches.json| C[stage-batch]
    C -->|staged manifest| D[generate]
    D -->|generated.mp4 per UUID| E[validate-generation]
    E -->|validation.json| F[summarize-run]

    D --> G[prepare-drivinggen]
    G --> H[drivinggen-fvd-lite]
    H -->|fvd-lite.json| F

    D --> I[prepare-worldlens]
    I --> J[worldlens-evaluate]
    J -->|worldlens.json| F

    subgraph Setup
        S1[setup-evaluator] -->|DrivingGen checkout| G
        S2[setup-worldlens] -->|WorldLens checkout + config| I
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[discover] -->|manifest.jsonl| B[plan-batches]
    B -->|batches.json| C[stage-batch]
    C -->|staged manifest| D[generate]
    D -->|generated.mp4 per UUID| E[validate-generation]
    E -->|validation.json| F[summarize-run]

    D --> G[prepare-drivinggen]
    G --> H[drivinggen-fvd-lite]
    H -->|fvd-lite.json| F

    D --> I[prepare-worldlens]
    I --> J[worldlens-evaluate]
    J -->|worldlens.json| F

    subgraph Setup
        S1[setup-evaluator] -->|DrivingGen checkout| G
        S2[setup-worldlens] -->|WorldLens checkout + config| I
    end

Loading

_{Reviews (7): Last reviewed commit: "Document OmniDreams evaluation workflow" | Re-trigger Greptile}

[jarcherNV]

Quick summary of the evaluation work:

This is an automated evaluation path around the existing Hugging Face scene inputs. The tooling can stage scenes in batches, run FlashDreams/OmniDreams inference, validate generated clips, and run two separate evaluators.

What it can do now:

• Download/stage existing scene inputs in manageable batches.
• Generate clips with FlashDreams from the existing first frames, prompts, and HD-map videos.
• Validate basic generation health: expected files, frame counts, total blocks, log-derived metadata, and failures.
• Run DrivingGen FVD-lite as a fixed-split regression metric. This is useful for comparing future OmniDreams checkpoints on the same scenes.
• Run WorldLens temporal/subject consistency metrics. These give standalone higher-is-better video consistency signals.
• Write a compact evaluation-summary.json and evaluation-summary.md via summarize-run.

What it cannot currently do:

• It cannot give a universal “model quality” score that is meaningful without comparison.
• It cannot directly measure ego path correctness, off-road behavior, collisions, traffic-rule compliance, or planner usefulness.
• It cannot infer reliable 3D scene state from the generated RGB alone.
• DrivingGen FVD-lite should not be used to compare different scene batches against each other.
• Current WorldLens mode measures video consistency, not full simulator fidelity.

A future driving-quality evaluator would need:

• Intended ego trajectory or control inputs.
• Map geometry in metric coordinates, not just rendered HD-map pixels.
• Camera calibration and pose timing.
• Object detections/tracks or ground-truth 3D boxes.
• A way to align generated video frames back to world/map coordinates.
• Metrics for lane adherence, off-road rate, collision risk, object persistence, traffic-agent behavior, and action-following.
• Ideally a planner/perception stack to test whether downstream driving behavior remains safe in the generated world.

[jarcherNV]

/ok to test 9baf68d

[jarcherNV]

/ok to test 531bbeb

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[jarcherNV]

/ok to test b846bf1

[jarcherNV]

/ok to test a2df8b1

[gtong-nv]

why is this patching needed? Can we consider forking the repo and modify the path there if needed?
Or host the checkpoint somewhere else to avoid another dependency on dropbox

[jarcherNV]

We could copy the checkpoint to somewhere local I guess. I'm not sure where would be the best place for it that everyone will have access to. I'll see if I can find something.

[gtong-nv]

Overall looks good. There some improvements we can address in follow-up PRs:

1. Group the metrics (e.g. Wordlens and DrivingGen) in a dedicated directory, or even make a parent class which exposes common run_eval APIs, to be distinguish from other utility files
2. Document the running instruction, runtime, metrics/report interpretation
3. Clean up the dependency setup process. Right now the github URL and commit hash hardcoded in the same file, as long with some patching loci. Considering a dedicated fork and group the URLs in a config file

[jarcherNV]

Overall looks good. There some improvements we can address in follow-up PRs:

1. Group the metrics (e.g. Wordlens and DrivingGen) in a dedicated directory, or even make a parent class which exposes common run_eval APIs, to be distinguish from other utility files
2. Document the running instruction, runtime, metrics/report interpretation
3. Clean up the dependency setup process. Right now the github URL and commit hash hardcoded in the same file, as long with some patching loci. Considering a dedicated fork and group the URLs in a config file

Thanks, these suggestions make sense. I added README documentation covering the evaluation workflow, runtime expectations, evaluator setup, and metric/report interpretation. I agree the evaluator organization and dependency pin/config cleanup would be useful, but it might make more sense to cover those in a follow-up PR once we have decided where to permanently host the checkpoint file.

[jarcherNV]

/ok to test 376678a