docs(puzzletron): install lm-eval in container setup - #2020
Conversation
The setup block uninstalls nvidia-lm-eval but never installed a replacement, so following the README left lm-eval missing for the Evaluation step. lm-eval ships in examples/llm_eval/requirements.txt. Fixes NVIDIA#1786 Signed-off-by: Rishi Khare <rishiskhare@gmail.com>
📝 WalkthroughWalkthroughThe Puzzletron NeMo container setup instructions now install dependencies from ChangesPuzzletron setup
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/ok to test 920d63a |
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #2020 +/- ##
==========================================
- Coverage 66.83% 61.74% -5.09%
==========================================
Files 519 519
Lines 58916 58916
==========================================
- Hits 39376 36378 -2998
- Misses 19540 22538 +2998
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@kevalmorabia97 Is there any action remaining for me? I see some checks are failing. |
What does this PR do?
Type of change: documentation
The container setup section uninstalls
nvidia-lm-evaland the note above it states that it is "replaced withlm-evalfrom the repo", but none of the install commands actually installlm-eval. Following the README therefore leaves the environment withoutlm-eval, which the Evaluation step requires.lm-evalships inexamples/llm_eval/requirements.txt, so this adds that one install line to the setup block.Usage
N/A
Testing
Traced the dependency graph to confirm
lm-evalis reachable from no installpath the README has currently:
puzzletronextra:fire,hydra-core,immutabledict,lru-dict,pandas,typeguarddev-testextra: pytest/coverage tooling,timm,torchprofile,torchvision,torch-geometricexamples/puzzletron/requirements.txt:math-verify,ray,transformers<5.0lm_eval[api,ifeval]>=0.4.10appears only inexamples/llm_eval/requirements.txt.pre-commit run --files examples/puzzletron/README.mdpasses (markdownlint included).Before your PR is "Ready for review"
CONTRIBUTING.md: N/AAdditional Information
Fixes #1786
Summary by CodeRabbit