Export BEAM + cgroup/OS host metrics for reconciliation#4678
Draft
alco wants to merge 13 commits into
Draft
Conversation
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
alco
force-pushed
the
alco--host-cgroup-metrics
branch
from
July 4, 2026 22:39
7248e1b to
9357ea7
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4678 +/- ##
==========================================
+ Coverage 60.03% 60.24% +0.21%
==========================================
Files 395 414 +19
Lines 43763 44502 +739
Branches 12587 12589 +2
==========================================
+ Hits 26271 26809 +538
- Misses 17411 17612 +201
Partials 81 81
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alco
force-pushed
the
alco--host-cgroup-metrics
branch
from
July 5, 2026 21:57
2a943f8 to
1e1c98e
Compare
Scaffold ElectricTelemetry.SystemMetrics, a module of poller-invoked measurement functions complementing ApplicationTelemetry's VM stats. This first piece delivers: - Cached platform/cgroup detection (system_info/0) stored in :persistent_term. Detects OS via :os.type() and cgroup version (v1/v2/ none) by stat-ing /sys/fs/cgroup. Later tasks (cgroup/host readers) build on this. - recon_alloc aggregate metrics (every 5s tick): vm.alloc.allocated, vm.alloc.used, vm.alloc.unused (allocated - used), and vm.alloc.carrier_usage (used/allocated ratio). - Per-allocator fragmentation (vm.alloc.fragmentation.unused, tagged by allocator), gated via an atomic :counters ref created once and cached in :persistent_term (one put per gate key), bumped via :counters.add/3 on the hot path. Roughly once a minute since :recon_alloc.fragmentation/1 is O(carriers). Registers the measurements in ApplicationTelemetry and adds the corresponding Telemetry.Metrics definitions. Adds recon ~> 2.5 dep.
Adds ElectricTelemetry.SystemMetrics.Cgroup, a defensive reader for cgroup
v1 and v2 accounting files, plus a thin cgroup_measurement/2 delegator in
SystemMetrics registered as a builtin periodic measurement.
Emits cgroup.* memory (current/anon/file/working_set/max), cpu
(usage_usec/nr_throttled/throttled_usec) and io (rbytes/wbytes) metrics,
plus PSI full avg10 for memory and cpu on v2. v2 reads /sys/fs/cgroup/*
directly; v1 reads per-controller subdirs. The cgroup version comes from the
cached system_info/0 detection; :none (incl. non-Linux) no-ops cleanly.
memory.max is skipped when unlimited ("max" on v2, the huge sentinel on v1);
working_set is skipped when inactive_file is missing; cpu times are converted
ns->us on v1 for a consistent microsecond unit. All reads are exhaustively
defensive: missing/malformed files skip the metric rather than crash. IO is
v2-only (v1 blkio is awkward and absent on our hosts). The cgroup root is
configurable via opts for fixture-based tests.
Add ElectricTelemetry.SystemMetrics.Proc, a defensive Linux /proc reader that emits host.mem.* (from /proc/meminfo) and host.proc.beam.* (from /proc/<beam_pid>/status and /proc/<beam_pid>/io). This bridges the cgroup plane and the BEAM allocator plane: per-process anon/file/shmem RSS shows cgroup anon tracking the BEAM footprint, and host meminfo gives the host-RAM ceiling/cache picture (one task per host on EC2, so host ~= container). Parsing of the flat /proc text files lives in a shared ElectricTelemetry.SystemMetrics.ProcfsParse helper. All reads are exhaustively defensive: missing files, EACCES on /proc/<pid>/io, missing keys, and malformed values skip the affected metric rather than crash. meminfo and status Rss*/VmRSS values (kB) are converted to bytes; io read_bytes/write_bytes are already bytes and left unscaled. The reader no-ops cleanly on non-Linux. It is wired in as a thin proc_measurement/2 delegator on SystemMetrics, registered in ApplicationTelemetry.builtin_periodic_measurements/1 (read every ~5s tick) with last_value metric defs in metrics/1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend ElectricTelemetry.DiskUsage to optionally compute per-directory
subtotals during the existing periodic disk walk (a single traversal,
no second pass and no new timer).
Disk.recursive_usage_grouped/3 returns {total, %{dir_name => bytes}}
where directories are bucketed by name at a configurable depth. The
returned total is byte-identical to recursive_usage/2, including the
legacy reset-to-0 behaviour on an unreadable directory; the bucket map
is a best-effort tally that only grows on regular files.
DiskUsage gains :group_depth (default nil = no bucketing, preserving
current behaviour) and :top_n (default 10) options, caches the top-N
largest buckets in ETS, and exposes them via current_dirs/1.
StackTelemetry wires group_depth: 4 (the electric shape storage layout
<storage_dir>/<stack_id>/<p1>/<p2>/<shape_handle>) and defines the new
electric.storage.dir.bytes metric tagged [:stack_id, :shape].
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
report_disk_usage/2 now also reads DiskUsage.current_dirs/1 and emits electric.storage.dir.bytes (tagged by shape handle) for the top-N largest shapes, reusing the existing periodic disk-usage walk and its emission cadence. Cardinality is bounded to the top-N buckets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shape-handle directory depth relative to the disk-usage walk root differs by deployment: standalone walks the bare storage dir with shapes nested under shapes/<stack_id>/<p1>/<p2>/<shape_handle> (depth 5), while cloud walks the per-tenant shapes dir directly (depth 4). The previous hard-coded depth of 4 in the generic telemetry package was therefore wrong for standalone, where it bucketed the per-shape electric.storage.dir.bytes metric by the 2-char hash-shard prefix (aggregating many shapes) instead of by shape handle. StackTelemetry now reads :shape_dir_group_depth from its opts, supplied by the electric-side caller, which computes it from the path that PureFileStorage.shape_data_dir/3 actually produces for a sample handle, measured relative to the walk root. This stays correct if the storage sharding scheme changes, and passes nil through to disable grouping for storage backends without an on-disk shape layout. Cross-stack attribution is a non-issue: each stack's walk root is already its own subtree, so a stack only ever buckets its own shapes. Also document the transient full-map + O(n log n) sort in top_n/2 as a known tradeoff that is fine at ~10k shapes per stack, and use a realistic two-level-sharded fixture layout in the tests with a regression guard asserting a too-shallow depth buckets by the hash-shard prefix instead of the shape handle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the otel-export.lux test to assert that the new SystemMetrics families reach the collector end-to-end: vm.alloc.* (allocator carriers), host.mem.* / host.proc.beam.* (/proc), and cgroup.memory.current. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Electric Cloud runs on cgroup v2 and the v1 path was a parallel read/emit implementation (per-controller subdirs, rss/cache key mapping, ns->us conversion, unlimited-limit sentinel) that doubled the reader's surface for hosts we no longer target. cgroup.* metrics are now v2-only; v1 hosts cleanly no-op like non-Linux platforms. This also lets cgroup version detection drop the `stat -fc` shell-out in favour of checking for /sys/fs/cgroup/cgroup.controllers, the canonical v2 marker, and memory.max no longer needs a sentinel filter (v2 spells unlimited as the literal "max", which already parses to nil and is skipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the patterns the cgroup and /proc readers had each grown a private copy of: - SystemMetrics.emit/2 drops nil measurements and fires a single :telemetry.execute per event. The readers previously emitted one single-key event per scalar (~19 executes per poll tick); each plane now emits one multi-key event (6 executes), matching the package convention (cf. get_system_memory_usage) with no metric-definition changes. - ProcfsParse.read_kv_file/1 replaces the three byte-identical read_meminfo/read_status/read_proc_io wrappers and Cgroup's own read_stat_file: one whitespace-tolerant "key[:] value [unit]" parser covers meminfo, status, io, memory.stat and cpu.stat. parse_float moves here next to parse_int, and read_int_file is shared too. Also: - Register Cgroup.measurement / Proc.measurement MFAs directly with the poller instead of going through thin delegators on SystemMetrics (the poller accepts any MFA; the extra layer bought nothing). - recon_alloc_measurement reads :allocated and :used once and derives unused/carrier_usage locally instead of triggering four full allocator sweeps per tick; the fragmentation reduce no longer computes unused_bytes twice per instance. - Fold the duplicated persistent_term get-or-compute code into a memoized/2 helper, drop the unused default_root/0 public functions, sum io.stat rbytes/wbytes in a single pass, and trim comment essays down to the load-bearing points. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Disk.recursive_usage/2 and recursive_usage_grouped/3 were two parallel copies of the same traversal whose totals had to be kept byte-identical by hand (with a comment and a test enforcing it). The grouped walk with a nil group_depth never establishes a bucket key, so it already IS the ungrouped walk: recursive_usage/2 is now a thin wrapper over it and the duplicate walker is deleted, making the byte-identical contract true by construction. This in turn collapses DiskUsage.read_disk_usage's group_depth branching into a single call. Also share the defensive ETS lookup between current/1 and current_dirs/1, and rename the telemetry opt shape_dir_group_depth -> disk_usage_group_depth: the generic package buckets directories at a depth and knows nothing about shapes, so the shape vocabulary now stays in sync-service where the depth is derived. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ApplicationTelemetry already reads host memory via :memsup.get_system_memory_data() (meminfo-backed on Linux) and emits it as the [:system, :memory] event, so the meminfo-derived host.mem.total/ free/cached family added on this branch was a second reader for the same three numbers. Drop it; the Proc reader now covers only the genuinely new data — the BEAM's own /proc/<pid>/status RSS breakdown and /proc/<pid>/io counters (host.proc.beam.*). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-allocator fragmentation breakdown is O(carriers) and was registered with the shared ~5s poller, self-gating to every 12th tick through a hand-rolled :counters ref cached in :persistent_term — plus :force/:gate_key test-only options and a public interval accessor to make the gate testable. SystemMonitor is already a long-lived GenServer doing periodic work (the hourly full GC), so give it a one-minute timer that calls the now ungated allocator_fragmentation_measurement/0 directly. This deletes the whole gate: the counter, the persistent_term entry, both test-only options, and the accessor. The sampling call is wrapped in a rescue so a metrics hiccup can't take down the system monitor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two once-ever answers were being recomputed on every ~5s poll tick: - read_kv_file/2 now takes a set of wanted keys and skips value parsing for every other line. /proc/<pid>/status is ~60 lines of which we consume 4 and cgroup memory.stat is ~40 of which we consume 3, so the bulk of the per-tick Integer.parse/Map.put work disappears. - The readers' file paths are constant for the life of the VM. Cgroup precomputes the default-root paths at compile time (tests passing :cgroup_root still build them dynamically); Proc resolves the BEAM pid and joins its two paths once, cached via SystemMetrics.memoized/2 (made public for this). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alco
force-pushed
the
alco--host-cgroup-metrics
branch
from
July 7, 2026 10:56
1e1c98e to
dbd1b85
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In-app telemetry poller that exports a complete, reconcilable memory/CPU/disk picture for Electric — the BEAM plane (what the VM thinks it uses) alongside the cgroup/OS plane (what the kernel charges), in one event stream with identical resource attributes.
What's included
New
ElectricTelemetry.SystemMetricspoller (+Cgroup,Proc,ProcfsParsesubmodules) reading/procand/sys/fs/cgroupdirectly and emitting OTel gauges:vm.alloc.{allocated,used,unused,carrier_usage}(5s), plus gated per-allocatorvm.alloc.fragmentation.unused{allocator}(~60s, O(carriers), gated via:counters).allocated − usedis the carrier-fragmentation / OOM-risk signal.cgroup.memory.*,cgroup.cpu.*,cgroup.io.*, and PSI*.pressure.full.avg10(v2). Version detected once at boot from cachedsystem_info/0; stable metric names regardless of host cgroup version.host.mem.*andhost.proc.beam.*(incl.io.{read,write}_bytes) to bridge BEAM ↔ cgroup accounting.electric.storage.dir.bytes{stack_id,shape}(top-N), by extending the existingDiskUsage60s walk (single traversal, no second walk, no hot-path counters). Group depth is derived fromPureFileStorage.shape_data_dir/3, so it's correct for both standalone (depth 5) and cloud (depth 4) layouts. Namedelectric.storage.*to sit next toelectric.storage.used.bytesinStackTelemetry.Not included (deferred follow-ups)
app.fs.capacity/used/avail(statvfs filesystem-capacity gauges)Why it matters
Lets us answer, from one Honeycomb query, whether memory growth is real (BEAM heap/binaries), reclaimable (page cache), or fragmentation (carriers held from the OS) — and the equivalent for CPU saturation and disk fill. Without the cgroup/OS plane next to the BEAM plane, there's no way to tell a genuine leak from reclaimable page cache, or to attribute CPU to real scheduler work vs. kernel accounting.
🤖 Generated with Claude Code