Skip to content

feat(release): ship a DWARF debug companion for prod kernels#24

Open
kalyazin wants to merge 2 commits into
mainfrom
kalyazin/kernel-debuginfo
Open

feat(release): ship a DWARF debug companion for prod kernels#24
kalyazin wants to merge 2 commits into
mainfrom
kalyazin/kernel-debuginfo

Conversation

@kalyazin

Copy link
Copy Markdown
Contributor

Why

Debug the prod guest kernel with source-level symbols (snapshot resume investigations)
without running a different kernel. So: keep building the exact prod kernel, also
publish its debug info, and prove adding DWARF doesn't change the running image.

What

  • DWARF config + split build — enable CONFIG_DEBUG_INFO_DWARF5 on
    configs/x86_64/6.1.158.config; build.sh splits the DWARF x86_64 build into a lean
    vmlinux.bin (objcopy --strip-debug, loadable segments + symtab) and a separate
    vmlinux.debug (--only-keep-debug). arm64 / non-DWARF builds unchanged. binutils
    added as an explicit dep. release.yml/upload-release-to-gcs.sh publish the
    .debug companion to a parallel GCS path; vmlinux.bin path/name unchanged
    (forward-only, never overwrites the boot kernel).
  • Code-neutrality gateverify-dwarf-code-neutral.sh builds each DWARF config
    twice (with/without DWARF, build metadata pinned) and asserts the boot image's
    loadable segments are byte-identical (check-loadable-sections.sh). With no arg it
    discovers every x86_64 config enabling DWARF — no hardcoded version. Wired as a
    verify-dwarf job that publish (→ deploy) depends on, so a non-neutral kernel
    can't be released; also runs on PRs via verify-dwarf-neutral.yml.

So a customer sandbox's guest kernel can be debugged with full source-level
symbols, prod kernels now build with DWARF and publish a vmlinux.debug companion
alongside the boot image (forward-only). Enabling DWARF is code-neutral —
confirmed via olddefconfig: only debug-info options change, no codegen impact —
so the boot image's loadable segments are unchanged.

- configs/x86_64/6.1.158.config: enable CONFIG_DEBUG_INFO_DWARF5 (KASLR and the
  rest unchanged).
- build.sh: for x86_64 builds carrying DWARF, split into a lean boot image
  (objcopy --strip-debug -> vmlinux.bin, loadable + symtab) and a vmlinux.debug
  companion (objcopy --only-keep-debug); non-DWARF/arm64 builds unchanged. Declare
  binutils explicitly in install_dependencies (objcopy/readelf are used by the
  split; previously only pulled in transitively via gcc).
- release.yml / upload-release-to-gcs.sh: attach and upload vmlinux.debug
  companions to the parallel GCS path; the boot vmlinux.bin path is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
@cla-bot cla-bot Bot added the cla-signed label Jun 12, 2026
@cursor

cursor Bot commented Jun 12, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes the shipped prod kernel build (debug info in config and split artifacts) and the release pipeline, but release is gated on code-neutrality verification before publish.

Overview
Production x86_64 6.1.158 now builds with CONFIG_DEBUG_INFO_DWARF5. build.sh splits DWARF builds into lean vmlinux.bin (objcopy --strip-debug) and vmlinux.debug, adds binutils, and keeps legacy copy paths unchanged for non-DWARF and arm64.

Release and GCS upload now include *.debug assets beside existing *.bin paths. verify-dwarf-code-neutral.sh and check-loadable-sections.sh double-build each DWARF config (DWARF on/off, pinned metadata, IKCONFIG off for compare) and require byte-identical executable sections. That check runs on PRs via verify-dwarf-neutral.yml and blocks publish on manual release via verify-dwarf in release.yml, auto-discovering any x86_64 config with DWARF5.

Reviewed by Cursor Bugbot for commit 6a2adb9. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread scripts/verify-dwarf-code-neutral.sh Outdated
@kalyazin kalyazin force-pushed the kalyazin/kernel-debuginfo branch 2 times, most recently from 549a69e to 5d719e9 Compare June 12, 2026 17:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5d719e9. Configure here.

Comment thread scripts/check-loadable-sections.sh Outdated
Guard that enabling DWARF does not change the kernel's machine code:
scripts/verify-dwarf-code-neutral.sh builds a version both WITH (the
committed config) and WITHOUT DWARF and asserts their executable
sections (.text and friends) are byte-identical, via
check-loadable-sections.sh.

It compares the executable code, not the whole loadable image, because
the non-code loadable data legitimately differs between two independent
builds: the GNU build-id (a hash that also covers the differing
.debug_*) and the .version build counter (the "#N" in linux_banner).
Both builds also disable CONFIG_IKCONFIG, so the embedded
/proc/config.gz blob (whose gzip size depends on the config text, which
differs by the debug-info lines) does not shift .init.data and the
addresses that .init.text references. With no version argument the
script discovers every x86_64 config that enables
CONFIG_DEBUG_INFO_DWARF5.

Run two ways:
- verify-dwarf-neutral.yml: PRs touching the kernel build inputs, plus
  manual dispatch (optional version override).
- release.yml: a verify-dwarf job that publish (and thus deploy) depends
  on, so a non-code-neutral kernel is never released.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
@kalyazin kalyazin force-pushed the kalyazin/kernel-debuginfo branch from 5d719e9 to 6a2adb9 Compare June 15, 2026 09:01
@kalyazin kalyazin marked this pull request as ready for review June 15, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant