Skip to content

Add coalesced full gradient access - #8374

Open
Marchematics wants to merge 3 commits into
deepspeedai:masterfrom
Marchematics:master
Open

Add coalesced full gradient access#8374
Marchematics wants to merge 3 commits into
deepspeedai:masterfrom
Marchematics:master

Conversation

@Marchematics

Copy link
Copy Markdown

Add safe_get_full_model_grads to collect model gradients with coalesced ZeRO-1/2 communication.

This replaces one collective per parameter with one coalesced collective per data-parallel group.

Fixes #7644

Tests: targeted gradient API test, pre-commit.

Signed-off-by: Marchematics <Marchematics@163.com>
@Marchematics
Marchematics requested a review from tjruwase as a code owner August 31, 2026 14:11
Copilot AI lite review requested due to automatic review settings August 31, 2026 14:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ad7b07915

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deepspeed/utils/tensor_fragment.py Outdated
Comment thread deepspeed/utils/__init__.py
Comment thread deepspeed/utils/__init__.py
@sfc-gh-truwase

Copy link
Copy Markdown
Collaborator

@Marchematics thanks for the PR. Please address the agent feedback.

Signed-off-by: Marchematics <Marchematics@163.com>
@Marchematics

Copy link
Copy Markdown
Author

Addressed the agent feedback in 5e314ed: fixed unmapped ZeRO parameters in the coalesced collective, added the regression test, and documented the public API. All commits are signed off.

Comment thread tests/unit/v1/zero/test_zero_tensor_fragment.py
Signed-off-by: Marchematics <Marchematics@163.com>
@pengdurice

Copy link
Copy Markdown
Contributor

Thanks for the PR, how does this work with when autoTP or autoEP is enabled.

@Marchematics

Copy link
Copy Markdown
Author

The current implementation handles ZeRO data-parallel groups only. AutoTP and AutoEP use additional tensor/expert-parallel sharding, so this helper does not yet reconstruct their full gradients. I can add dedicated AutoTP/AutoEP coverage and reduction handling if you want this API to support those paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[REQUEST] Fast access to whole model gradient during training

4 participants