Issues

A None caption in a Hugging Face VLM dataset raises TypeError mid-run on the rank that reads the row, while the other ranks wait in the next collective

KempnerInstitute/KempnerForge#427 · opened

on Oct 31, 2026

Checkpoint retention reads any step_* name as a step number: step_best makes every save raise, and step_3_old is deleted as step 3

KempnerInstitute/KempnerForge#426 · opened

on Oct 31, 2026

AsyncCheckpointer.wait() keeps a failed flush's future, so every later save() and wait() re-raises the same error and nothing more is dispatched

KempnerInstitute/KempnerForge#425 · opened

on Oct 31, 2026

An error before the DCP write in CheckpointManager.save(), such as a failed mkdir, raises on one rank only and leaves the other ranks waiting

KempnerInstitute/KempnerForge#424 · opened

on Oct 31, 2026

A [video] run silently ignores the [data] Hugging Face fields, including hf_dataset_prompt_field

KempnerInstitute/KempnerForge#423 · opened

on Oct 30, 2026

test_7b_tp_fsdp_compile always fails: it trains from 7b.toml's placeholder dataset_path

KempnerInstitute/KempnerForge#420 · opened

on Oct 30, 2026

A micro-batch whose labels are all -100 crashes the step because the loss has no autograd graph

KempnerInstitute/KempnerForge#419 · opened

on Oct 30, 2026

check_env passes Databricks setups the MLflow backend cannot authenticate

KempnerInstitute/KempnerForge#418 · opened

on Oct 30, 2026

VLM parallelism checks are a denylist, so a new VLM arch passes EP, and TP with sequence parallelism off, by default

KempnerInstitute/KempnerForge#417 · opened

on Oct 30, 2026

A --vlm.arch override keeps the preset's VLM config class and builds a mixed model

KempnerInstitute/KempnerForge#416 · opened

on Oct 30, 2026

Routed-expert gradients are ep times too large under expert parallelism

KempnerInstitute/KempnerForge#415 · opened

on Oct 30, 2026

Make MLflow a core dependency

KempnerInstitute/KempnerForge#401 · opened

on Oct 27, 2026

Discussion: expose runtime-optional dependencies as extras instead of uv dependency groups

KempnerInstitute/KempnerForge#399 · opened

on Oct 27, 2026

An I/O error in rank 0's part of CheckpointManager.save() skips the save barrier and leaves the other ranks waiting

KempnerInstitute/KempnerForge#396 · opened

on Oct 26, 2026

Move the MLflow tracking code out of metrics/tracker.py into its own module

KempnerInstitute/KempnerForge#391 · opened

on Oct 26, 2026

e2e failure messages hide the child traceback of a multi-rank run

KempnerInstitute/KempnerForge#390 · opened

on Oct 26, 2026

README claims "Dense pipeline parallelism works" and a 125M to 70B preset range that the repo does not have

KempnerInstitute/KempnerForge#387 · opened

on Oct 24, 2026

FP8 docs link to "measured FP8 MFU" in benchmarks.md, which has no FP8 numbers

KempnerInstitute/KempnerForge#386 · opened

on Oct 24, 2026

mechanistic-interpretability.md says attention weights cannot be captured; capture_attention_weights does it

KempnerInstitute/KempnerForge#385 · opened

on Oct 24, 2026

dcp-model.md calls async the production default and credits async_with_pinned_mem with staging it does not do

KempnerInstitute/KempnerForge#384 · opened

on Oct 24, 2026

Preemption docs show ckpt_mgr.wait() before finish(), but the loop calls finish() first

KempnerInstitute/KempnerForge#382 · opened

on Oct 24, 2026

SLURM docs give two contradictory, incorrect explanations of --signal=B:

KempnerInstitute/KempnerForge#381 · opened

on Oct 24, 2026

nccl-liveness.md says check_nccl_health ignores timeout_sec; it has honored it since #64

KempnerInstitute/KempnerForge#378 · opened

on Oct 24, 2026

Two doc pages still say the training loop does not save dataloader state

KempnerInstitute/KempnerForge#377 · opened

on Oct 24, 2026

The MoMa + expert-parallel rejection in JobConfig.validate can never run

KempnerInstitute/KempnerForge#376 · opened

on Oct 24, 2026

Navigation Menu

Repository navigation

Issues

A None caption in a Hugging Face VLM dataset raises TypeError mid-run on the rank that reads the row, while the other ranks wait in the next collective

Checkpoint retention reads any step_* name as a step number: step_best makes every save raise, and step_3_old is deleted as step 3

AsyncCheckpointer.wait() keeps a failed flush's future, so every later save() and wait() re-raises the same error and nothing more is dispatched

An error before the DCP write in CheckpointManager.save(), such as a failed mkdir, raises on one rank only and leaves the other ranks waiting

A [video] run silently ignores the [data] Hugging Face fields, including hf_dataset_prompt_field

test_7b_tp_fsdp_compile always fails: it trains from 7b.toml's placeholder dataset_path

A micro-batch whose labels are all -100 crashes the step because the loss has no autograd graph

check_env passes Databricks setups the MLflow backend cannot authenticate

VLM parallelism checks are a denylist, so a new VLM arch passes EP, and TP with sequence parallelism off, by default

A --vlm.arch override keeps the preset's VLM config class and builds a mixed model

Routed-expert gradients are ep times too large under expert parallelism

Make MLflow a core dependency

Discussion: expose runtime-optional dependencies as extras instead of uv dependency groups

An I/O error in rank 0's part of CheckpointManager.save() skips the save barrier and leaves the other ranks waiting

Move the MLflow tracking code out of metrics/tracker.py into its own module

e2e failure messages hide the child traceback of a multi-rank run

README claims "Dense pipeline parallelism works" and a 125M to 70B preset range that the repo does not have

FP8 docs link to "measured FP8 MFU" in benchmarks.md, which has no FP8 numbers

mechanistic-interpretability.md says attention weights cannot be captured; capture_attention_weights does it

dcp-model.md calls async the production default and credits async_with_pinned_mem with staging it does not do

Preemption docs show ckpt_mgr.wait() before finish(), but the loop calls finish() first

SLURM docs give two contradictory, incorrect explanations of --signal=B:

nccl-liveness.md says check_nccl_health ignores timeout_sec; it has honored it since #64

Two doc pages still say the training loop does not save dataloader state

The MoMa + expert-parallel rejection in JobConfig.validate can never run