Skip to content

docs(puzzletron): install lm-eval in container setup - #2020

Open
rishiskhare wants to merge 1 commit into
NVIDIA:mainfrom
rishiskhare:docs/puzzletron-install-lm-eval
Open

docs(puzzletron): install lm-eval in container setup#2020
rishiskhare wants to merge 1 commit into
NVIDIA:mainfrom
rishiskhare:docs/puzzletron-install-lm-eval

Conversation

@rishiskhare

@rishiskhare rishiskhare commented Jul 27, 2026

Copy link
Copy Markdown

What does this PR do?

Type of change: documentation

The container setup section uninstalls nvidia-lm-eval and the note above it states that it is "replaced with lm-eval from the repo", but none of the install commands actually install lm-eval. Following the README therefore leaves the environment without lm-eval, which the Evaluation step requires.

lm-eval ships in examples/llm_eval/requirements.txt, so this adds that one install line to the setup block.

Usage

N/A

Testing

Traced the dependency graph to confirm lm-eval is reachable from no install
path the README has currently:

  • puzzletron extra: fire, hydra-core, immutabledict, lru-dict, pandas, typeguard
  • dev-test extra: pytest/coverage tooling, timm, torchprofile, torchvision, torch-geometric
  • examples/puzzletron/requirements.txt: math-verify, ray, transformers<5.0

lm_eval[api,ifeval]>=0.4.10 appears only in examples/llm_eval/requirements.txt.

pre-commit run --files examples/puzzletron/README.md passes (markdownlint included).

Before your PR is "Ready for review"

  • Is this change backward compatible?: ✅
  • If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
  • Did you write any new necessary tests?: N/A
  • Did you update Changelog?: N/A
  • Did you get Claude approval on this PR?: N/A

Additional Information

Fixes #1786

Summary by CodeRabbit

  • Documentation
    • Updated Puzzletron container setup instructions to include installing the LLM evaluation dependencies.
    • Clarified the setup sequence before running smoke tests.

The setup block uninstalls nvidia-lm-eval but never installed a
replacement, so following the README left lm-eval missing for the
Evaluation step. lm-eval ships in examples/llm_eval/requirements.txt.

Fixes NVIDIA#1786

Signed-off-by: Rishi Khare <rishiskhare@gmail.com>
@rishiskhare
rishiskhare requested a review from a team as a code owner July 27, 2026 05:08
@copy-pr-bot

copy-pr-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The Puzzletron NeMo container setup instructions now install dependencies from examples/llm_eval/requirements.txt before smoke testing.

Changes

Puzzletron setup

Layer / File(s) Summary
Add llm-eval installation step
examples/puzzletron/README.md
Adds the llm-eval requirements installation command after the Puzzletron editable install.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: aanoosheh, achidiac-nv, ajrasane

🚥 Pre-merge checks | ✅ 6
✅ Passed checks (6 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The added requirements install line addresses issue #1786 by documenting and installing lm-eval for Puzzletron.
Out of Scope Changes check ✅ Passed The change is a single documentation update and does not introduce unrelated scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Security Anti-Patterns ✅ Passed Only examples/puzzletron/README.md changed; no Python, requirements, or security-sensitive patterns were introduced.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the docs change to install lm-eval in the Puzzletron container setup.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@kevalmorabia97

Copy link
Copy Markdown
Collaborator

/ok to test 920d63a

@kevalmorabia97
kevalmorabia97 enabled auto-merge (squash) July 28, 2026 05:34
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.74%. Comparing base (33d05b0) to head (920d63a).
⚠️ Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (33d05b0) and HEAD (920d63a). Click for more details.

HEAD has 15 uploads less than BASE
Flag BASE (33d05b0) HEAD (920d63a)
unit 3 1
examples 23 10
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2020      +/-   ##
==========================================
- Coverage   66.83%   61.74%   -5.09%     
==========================================
  Files         519      519              
  Lines       58916    58916              
==========================================
- Hits        39376    36378    -2998     
- Misses      19540    22538    +2998     
Flag Coverage Δ
examples 33.31% <ø> (-10.08%) ⬇️
unit 54.89% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rishiskhare

Copy link
Copy Markdown
Author

@kevalmorabia97 Is there any action remaining for me? I see some checks are failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Using lm-eval for puzzletron fails - lm-eval not installed

2 participants