Add mktestdocs to catch docs/code drift by cianc · Pull Request #1096 · mlco2/codecarbon

cianc · 2026-03-03T21:47:42Z

Description

Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library.

Add mktestdocs and pytest to the doc dependency group in pyproject.toml so they are available alongside the other doc-build tools without pulling in the full dev group.
Add scripts/check-docs-drift.py: a pytest-based script that uses mktestdocs.grab_code_blocks() to collect every ```python fenced block under docs/, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.
Fix all ```python code blocks across docs/ so they are correctly picked up by mktestdocs:
- Remove the stray space in python fences (changed topython) so that mktestdocs can identify them (it matches on the exact string "python" immediately after the backticks).
- Add save_to_file=False, log_level="error" to EmissionsTracker and OfflineEmissionsTracker instantiations to avoid creating CSV files or noisy output during CI runs.
- Add # skip as the first line of blocks that cannot run in CI because they depend on external services or optional heavy dependencies (TensorFlow, Prometheus, Logfire, Google Cloud, Comet ML, live CodeCarbon API).
- Correct a pip install command that was incorrectly fenced as python in `comet.md`; changed to console.
Update .github/workflows/build-docs.yml to run docs-check-drift as a step before the docs build, triggered on changes to docs/**, mkdocs.yml, or scripts/check-docs-drift.py.
Document the drift check and the # skip convention in CONTRIBUTING.md under the "Build Documentation" section.

Related Issue

Please link to the issue this PR resolves: #1083

Motivation and Context

Helps prevent drift of code blocks in documentation

How Has This Been Tested?

Ran all tests including new mktestdocs tests.

Screenshots (if appropriate):

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING.md document.
I have added tests to cover my changes.
All new and existing tests passed.

Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library. Changes: - Add `mktestdocs` and `pytest` to the `doc` dependency group in `pyproject.toml` so they are available alongside the other doc-build tools without pulling in the full `dev` group. - Add `scripts/check-docs-drift.py`: a pytest-based script that uses `mktestdocs.grab_code_blocks()` to collect every ```python fenced block under `docs/`, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`. - Fix all ```python code blocks across `docs/` so they are correctly picked up by mktestdocs: - Remove the stray space in ``` python fences (changed to ```python) so that mktestdocs can identify them (it matches on the exact string "python" immediately after the backticks). - Add `save_to_file=False, log_level="error"` to `EmissionsTracker` and `OfflineEmissionsTracker` instantiations to avoid creating CSV files or noisy output during CI runs. - Add `# skip` as the first line of blocks that cannot run in CI because they depend on external services or optional heavy dependencies (TensorFlow, Prometheus, Logfire, Google Cloud, Comet ML, live CodeCarbon API). - Correct a `pip install` command that was incorrectly fenced as ```python in `comet.md`; changed to ```console. - Update `.github/workflows/build-docs.yml` to run `docs-check-drift` as a step before the docs build, triggered on changes to `docs/**`, `mkdocs.yml`, or `scripts/check-docs-drift.py`. - Document the drift check and the `# skip` convention in `CONTRIBUTING.md` under the "Build Documentation" section.

codecov · 2026-03-03T23:17:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.22%. Comparing base (9be333a) to head (9057de3).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1096   +/-   ##
=======================================
  Coverage   78.22%   78.22%           
=======================================
  Files          38       38           
  Lines        3632     3632           
=======================================
  Hits         2841     2841           
  Misses        791      791

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

davidberenstein1957 · 2026-03-04T14:49:30Z

Hi @cianc,

Thanks a lot for this PR. Do you have time to also move some of the documentation examples to docstring examples in the Python code? We would then be able to load them from here, while keeping a single source of truth directly from the Python code, while also exposing examples in the docstring :)
If not, we can focus on keeping the current approach.

Also, I feel that we might not need a separate script directory, but rather just keep Python file as part of the tests directory.

Anywhow. Thahnks a lot for tackling this :)

davidberenstein1957 · 2026-03-04T14:50:12Z

docs/getting-started/api.md

+```python
+# skip
 from codecarbon import track_emissions

 @track_emissions(save_to_api=True)
 def train_model():
-    # GPU intensive training code  goes here
+    # GPU intensive training code goes here
+    pass

-if __name__ =="__main__":
+if __name__ == "__main__":
    train_model()
 ```


So, as proposed, we coul add this code example to the track_emissions examples docstring and directly import it from there.

This would mean that we don't directly test our docs but rather do that for the exmaples in our docstrings.

davidberenstein1957

I left some small remarks and a comment in the main thread. Happy to hear your thoughts and thanks a lot for the help :)

davidberenstein1957 · 2026-03-04T14:50:40Z

docs/getting-started/api.md

+```python
+# skip
 from codecarbon import track_emissions

 @track_emissions(save_to_api=True)
 def train_model():
-    # GPU intensive training code  goes here
+    # GPU intensive training code goes here
+    pass

-if __name__ =="__main__":
+if __name__ == "__main__":
    train_model()
 ```


This would mean that we don't directly test our docs but rather do that for the exmaples in our docstrings.

davidberenstein1957 · 2026-03-04T14:51:27Z

docs/getting-started/api.md

    experiment_id="your experiment id",
    save_to_api=True,
+    save_to_file=False,
+    log_level="error",


Was this updated automatically? Or did you manually add these arguments? If so, it would be nice to explore the way proposed to do this automatically :)

To have it all time you could add a .codecarbon.config in CI, or environment variables:

CODECARBON_LOG_LEVEL="error" CODECARBON_SAVE_TO_FILE=False

So every will use these settings.

BTW @cianc this is awesome for the quality of the doc ! I did not know it was possible.

davidberenstein1957 · 2026-03-04T14:52:15Z

docs/getting-started/examples.md


-``` python
+```python
+# skip


we can actually run this for a small subset of the data, right?

davidberenstein1957 · 2026-03-04T14:52:35Z

docs/getting-started/examples.md


-``` python
+```python
+# skip


same here, we could run this with a small subset of the data, right?

davidberenstein1957 · 2026-03-04T14:53:25Z

docs/getting-started/usage.md

+```python
 from codecarbon import EmissionsTracker
-tracker = EmissionsTracker()
+tracker = EmissionsTracker(save_to_file=False, log_level="error")


why do we add these arguments here specifically and not in all other places?

davidberenstein1957 · 2026-03-09T12:36:20Z

@cianc just following up here to see if the extended contribution fits in your time schedule, if not feel free to let us know, and we'll handle the rest :)

cianc · 2026-03-09T22:08:18Z

Yes sorry, work has been quite busy. These suggestions seems reasonable and I hope to get to them soon.

…

On Mon, Mar 9, 2026 at 12:36 PM David Berenstein ***@***.***> wrote: *davidberenstein1957* left a comment (mlco2/codecarbon#1096) <#1096 (comment)> @cianc <https://github.com/cianc> just following up here to see if the extended contribution fits in your time schedule, if not feel free to let us know, and we'll handle the rest :) — Reply to this email directly, view it on GitHub <#1096 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADD7PTBJQDHYNZWPCHSYZD4P23FVAVCNFSM6AAAAACWF6E7M2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMRTGQ2TAOJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cianc marked this pull request as ready for review March 3, 2026 21:51

cianc requested a review from a team as a code owner March 3, 2026 21:51

davidberenstein1957 reviewed Mar 4, 2026

View reviewed changes

davidberenstein1957 self-assigned this Mar 4, 2026

Uh oh!

Conversation

cianc commented Mar 3, 2026

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

codecov bot commented Mar 3, 2026

Codecov Report

Uh oh!

davidberenstein1957 commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidberenstein1957 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidberenstein1957 commented Mar 9, 2026

Uh oh!

cianc commented Mar 9, 2026 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants