feat(lingbot-video): add native Dense/MoE inference and four-GPU runtime#15
Merged
Conversation
- add Dense and MoE model implementations with T2V/TI2V pipelines and refiner support - add service contracts, examples, documentation, and source attribution - add benchmark, parity, replay, and handoff validation tooling with unit coverage - preserve VAE, visual conditioning, RGB handoff, geometry, and prompt-envelope behavior Verification: - git diff --cached --check
- add four-GPU FSDP and Ulysses execution for Dense and refiner stages - keep complete scheduler loops inside distributed stage workers - split the generic entrypoints into Dense and MoE model-specific examples - add native-parallel, distributed, and sequence-parallel validation tools Verification: - git diff --cached --check
- add inference-only FSDP2 sharding and source-style Ulysses communication - support packed SDPA and distributed MoE routing with grouped-GEMM fallback behavior - cap per-rank CPU intra-op threads and avoid unnecessary worker cache flushes - document and test the validated four-GPU execution constraints Verification: - git diff --cached --check
Fix distributed base-worker restoration after refiner handoff, official zero-null refiner CFG defaults, T2I refiner rejection, prompt-duration precedence, and single-frame output suffix handling. Restore grouped-MoE outputs with FP32 route reduction, disable CFG for guidance scales at or below one, and reject packed sequences on attention backends that do not implement packed semantics. Keep validation tools aligned and add regression coverage for the corrected lifecycle and contracts. Verification: - .venv/bin/python -m pytest <LingBot-Video related suites> (119 passed) - H100 grouped_mm versus sorted eager exact parity (0 mismatches) - ruff check on all changed files - ruff format --check on all changed files - git diff --check - Dense 1.3B 4-GPU example, 121 frames at 832x480 - MoE 30B plus refiner 4-GPU example, 121 frames at 1920x1088
- add CFG2 x SP2 execution for Dense, MoE, and refiner stages with batch-CFG conflict validation - add native per-channel W8A8 expert execution while retaining grouped BF16 as the performance default - keep base and refiner DiT workers co-resident during refinement and release both before high-resolution VAE decode - expose CLI/config controls, document memory and backend tradeoffs, and cover lifecycle and numerical behavior Verification: - ruff format --check on all changed Python files - ruff check on all changed Python files - git diff --check - pytest LingBot-Video unit/model/tool suite: 89 passed - 4x H100 Dense CFG2 x SP2, 832x480, 25 frames - 4x H100 MoE FP8 CFG2 x SP2 with co-resident refiner, 1920x1088, 25 frames - 4x H100 SP4 versus CFG2 x SP2 Dense and MoE grouped_mm benchmarks
- keep examples/lingbot_video/README.md as the single user-facing guide\n- remove duplicate English and Chinese MkDocs pages and stale navigation entries\n- update model-zoo defaults, CFG/SP, FP8, and refiner lifecycle references\n- update supported pipeline matrices and service contract tests\n\nVerification: git diff --check
Replace datetime.UTC with timezone.utc so the reference-capture tool imports under Python 3.10, the project's minimum supported version.\n\nVerification:\n- ruff check tools/validation/capture_lingbot_video_reference.py\n- ruff format --check tools/validation/capture_lingbot_video_reference.py\n- .venv/bin/python -m pytest tests/unit/tools/test_capture_lingbot_video_reference.py tests/unit/pipelines/lingbot_video/test_refiner.py -v\n- git diff --cached --check
Register Dense and MoE LingBot transformer checkpoint hashes and load them through ModuleManager, removing the dedicated pipeline loader. Configure example runtime devices directly through ModelRuntimeConfig and support Diffusers checkpoint directories in ModuleManager. Document that shared framework APIs cannot be broadened during pipeline integration without explicit approval. Verification: ruff format and ruff check on changed files; pytest tests/unit/pipelines/lingbot_video/test_runtime.py tests/unit/models/test_lingbot_video_moe.py -q; four-GPU Dense and MoE plus refiner smoke runs.
Align the LingBot example contract test with the service loader's canonical PIPELINE_CONTRACT export and remove the obsolete CONTRACT alias expectation. Verification: ruff format tests/unit/pipelines/lingbot_video/test_service_contract.py; ruff check tests/unit/pipelines/lingbot_video/test_service_contract.py; pytest tests/unit/pipelines/lingbot_video/test_service_contract.py -q.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add native TeleFuser support for LingBot-Video Dense 1.3B and MoE 30B checkpoints.
The PR provides single-process and four-GPU inference for T2I, T2V, and TI2V,
including the MoE low-noise refiner. It preserves upstream conditioning and
sampling semantics while exposing TeleFuser-native service, parallelism, and
validation workflows.
Motivation
LingBot-Video previously required the upstream runtime and did not have a
TeleFuser pipeline, service contract, or validated distributed execution path.
This change makes LingBot-Video available through the standard TeleFuser
pipeline/service interfaces, supports four-GPU inference without wrapping the
service in
torchrun, and keeps upstream-compatible behavior for structuredcaptions, VAE latents, visual conditioning, and base-to-refiner handoff.
Type of Change
Changes Made
structured-caption encoding, Wan VAE stages, scheduler integration, and T2I/T2V/TI2V pipelines.
normalization, TI2V visual and clean-latent conditions, geometry handling,
prompt envelopes, and upstream-compatible default negative conditions.
TF_MODEL_ZOO_PATHmodel locations, and user-facing example documentation.MoE routing, CFG2×SP2 support, batch-CFG conflict validation, and per-rank CPU
thread limits.
expert_backend=fp8; grouped BF16 GEMMremains the throughput default.
them before high-resolution VAE decode; retain the lower-memory serial fallback.
and base-to-refiner handoff validation tools with unit coverage.
FP32 route reduction, CFG behavior at guidance scale <= 1, refiner constraints,
prompt-duration precedence, and output suffix handling.
and project guidance documentation.
Testing
Test commands: