Skip to content

feat(lingbot-video): add native Dense/MoE inference and four-GPU runtime#15

Merged
lzx1413 merged 9 commits into
mainfrom
lingbot-video
Jul 23, 2026
Merged

feat(lingbot-video): add native Dense/MoE inference and four-GPU runtime#15
lzx1413 merged 9 commits into
mainfrom
lingbot-video

Conversation

@lzx1413

@lzx1413 lzx1413 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Description

Add native TeleFuser support for LingBot-Video Dense 1.3B and MoE 30B checkpoints.

The PR provides single-process and four-GPU inference for T2I, T2V, and TI2V,
including the MoE low-noise refiner. It preserves upstream conditioning and
sampling semantics while exposing TeleFuser-native service, parallelism, and
validation workflows.

Motivation

LingBot-Video previously required the upstream runtime and did not have a
TeleFuser pipeline, service contract, or validated distributed execution path.

This change makes LingBot-Video available through the standard TeleFuser
pipeline/service interfaces, supports four-GPU inference without wrapping the
service in torchrun, and keeps upstream-compatible behavior for structured
captions, VAE latents, visual conditioning, and base-to-refiner handoff.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Other (please describe):

Changes Made

  • Add LingBot-Video Dense and MoE DiT implementations, checkpoint loaders, Qwen3-VL
    structured-caption encoding, Wan VAE stages, scheduler integration, and T2I/T2V/TI2V pipelines.
  • Add native MoE refiner execution with in-memory RGB handoff; preserve VAE
    normalization, TI2V visual and clean-latent conditions, geometry handling,
    prompt envelopes, and upstream-compatible default negative conditions.
  • Add Dense and MoE service examples, contracts, CLI entry points, default
    TF_MODEL_ZOO_PATH model locations, and user-facing example documentation.
  • Add four-GPU FSDP/Ulysses execution, worker-owned scheduler loops, grouped-GEMM
    MoE routing, CFG2×SP2 support, batch-CFG conflict validation, and per-rank CPU
    thread limits.
  • Add opt-in FP8 W8A8 expert execution via expert_backend=fp8; grouped BF16 GEMM
    remains the throughput default.
  • Keep base and refiner workers co-resident for four-GPU refinement, then release
    them before high-resolution VAE decode; retain the lower-memory serial fallback.
  • Add benchmark, capture, replay, parity, distributed/SP, checkpoint inspection,
    and base-to-refiner handoff validation tools with unit coverage.
  • Add regression fixes for distributed refiner handoff restoration, grouped-MoE
    FP32 route reduction, CFG behavior at guidance scale <= 1, refiner constraints,
    prompt-duration precedence, and output suffix handling.
  • Add source attribution and update README, service, configuration, parallelism,
    and project guidance documentation.

Testing

  • Targeted LingBot-Video unit/model/pipeline/tool tests pass
  • Manual GPU testing performed
  • Benchmarks and validation tooling added

Test commands:

ruff check <changed-python-files>
ruff format --check <changed-python-files>
git diff --check

python -m pytest \
  tests/unit/models/test_lingbot_video_dit.py \
  tests/unit/models/test_lingbot_video_moe.py \
  tests/unit/pipelines/lingbot_video \

Validated GPU scenarios:

- 4x H100: Dense 1.3B SP4, 832x480, 121 frames
- 4x H100: MoE 30B + refiner, 1920x1088, 121 frames
- 4x H100: Dense CFG2×SP2, 832x480, 25 frames
- 4x H100: MoE FP8 CFG2×SP2 with co-resident refiner, 1920x1088, 25 frames
- H100: grouped_mm versus sorted eager MoE exact-parity check, 0 mismatches
- 4x H100: SP4 versus CFG2×SP2 Dense and MoE grouped_mm benchmarks

## Checklist

- [x] Code follows the project's coding standards (ruff)
- [x] Pre-commit hooks pass
- [x] Targeted LingBot-Video tests pass
- [x] New tests added for new functionality
- [x] Documentation updated (README, CLAUDE.md, docstrings)
- [x] Commit messages are clear and descriptive
- [x] PR title follows the convention: [TYPE] Brief description

## Related Issues

Fixes #

## Additional Notes

- expert_backend=fp8 is memory-oriented and requires PyTorch support for
  torch._scaled_mm; grouped GEMM remains the default four-GPU MoE backend.

- MoE expert parallelism and external FlashAttention are intentionally not enabled.
- The source-equivalent sorted eager MoE path remains the single-GPU correctness fallback.
- LingBot-Video usage documentation lives in examples/lingbot_video/README.md,
  consistent with other model-specific examples.

## GPU Architecture Support

- [ ] SM80 (Ampere, Ada Lovelace)
- [x] SM90 (Hopper H100)
  four-GPU refinement and releases both stages before high-resolution VAE decode.

- FP8 expert execution reduces expert-weight memory, but is currently slower than
  grouped BF16 GEMM and is therefore opt-in.

lzx1413 added 6 commits July 22, 2026 16:50
- add Dense and MoE model implementations with T2V/TI2V pipelines and refiner support
- add service contracts, examples, documentation, and source attribution
- add benchmark, parity, replay, and handoff validation tooling with unit coverage
- preserve VAE, visual conditioning, RGB handoff, geometry, and prompt-envelope behavior

Verification:
- git diff --cached --check
- add four-GPU FSDP and Ulysses execution for Dense and refiner stages
- keep complete scheduler loops inside distributed stage workers
- split the generic entrypoints into Dense and MoE model-specific examples
- add native-parallel, distributed, and sequence-parallel validation tools

Verification:
- git diff --cached --check
- add inference-only FSDP2 sharding and source-style Ulysses communication
- support packed SDPA and distributed MoE routing with grouped-GEMM fallback behavior
- cap per-rank CPU intra-op threads and avoid unnecessary worker cache flushes
- document and test the validated four-GPU execution constraints

Verification:
- git diff --cached --check
Fix distributed base-worker restoration after refiner handoff, official zero-null refiner CFG defaults, T2I refiner rejection, prompt-duration precedence, and single-frame output suffix handling.

Restore grouped-MoE outputs with FP32 route reduction, disable CFG for guidance scales at or below one, and reject packed sequences on attention backends that do not implement packed semantics. Keep validation tools aligned and add regression coverage for the corrected lifecycle and contracts.

Verification:
- .venv/bin/python -m pytest <LingBot-Video related suites> (119 passed)
- H100 grouped_mm versus sorted eager exact parity (0 mismatches)
- ruff check on all changed files
- ruff format --check on all changed files
- git diff --check
- Dense 1.3B 4-GPU example, 121 frames at 832x480
- MoE 30B plus refiner 4-GPU example, 121 frames at 1920x1088
- add CFG2 x SP2 execution for Dense, MoE, and refiner stages with batch-CFG conflict validation
- add native per-channel W8A8 expert execution while retaining grouped BF16 as the performance default
- keep base and refiner DiT workers co-resident during refinement and release both before high-resolution VAE decode
- expose CLI/config controls, document memory and backend tradeoffs, and cover lifecycle and numerical behavior

Verification:
- ruff format --check on all changed Python files
- ruff check on all changed Python files
- git diff --check
- pytest LingBot-Video unit/model/tool suite: 89 passed
- 4x H100 Dense CFG2 x SP2, 832x480, 25 frames
- 4x H100 MoE FP8 CFG2 x SP2 with co-resident refiner, 1920x1088, 25 frames
- 4x H100 SP4 versus CFG2 x SP2 Dense and MoE grouped_mm benchmarks
- keep examples/lingbot_video/README.md as the single user-facing guide\n- remove duplicate English and Chinese MkDocs pages and stale navigation entries\n- update model-zoo defaults, CFG/SP, FP8, and refiner lifecycle references\n- update supported pipeline matrices and service contract tests\n\nVerification: git diff --check
@lzx1413 lzx1413 mentioned this pull request Jul 22, 2026
22 tasks
Replace datetime.UTC with timezone.utc so the reference-capture tool imports under Python 3.10, the project's minimum supported version.\n\nVerification:\n- ruff check tools/validation/capture_lingbot_video_reference.py\n- ruff format --check tools/validation/capture_lingbot_video_reference.py\n- .venv/bin/python -m pytest tests/unit/tools/test_capture_lingbot_video_reference.py tests/unit/pipelines/lingbot_video/test_refiner.py -v\n- git diff --cached --check
lzx1413 added 2 commits July 23, 2026 02:25
Register Dense and MoE LingBot transformer checkpoint hashes and load them through ModuleManager, removing the dedicated pipeline loader. Configure example runtime devices directly through ModelRuntimeConfig and support Diffusers checkpoint directories in ModuleManager. Document that shared framework APIs cannot be broadened during pipeline integration without explicit approval.

Verification: ruff format and ruff check on changed files; pytest tests/unit/pipelines/lingbot_video/test_runtime.py tests/unit/models/test_lingbot_video_moe.py -q; four-GPU Dense and MoE plus refiner smoke runs.
Align the LingBot example contract test with the service loader's canonical PIPELINE_CONTRACT export and remove the obsolete CONTRACT alias expectation.

Verification: ruff format tests/unit/pipelines/lingbot_video/test_service_contract.py; ruff check tests/unit/pipelines/lingbot_video/test_service_contract.py; pytest tests/unit/pipelines/lingbot_video/test_service_contract.py -q.
@lzx1413
lzx1413 merged commit acf172d into main Jul 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant