Skip to content

filesystem: Deduplicate on the labels each metric carries - #3819

Open
neoLsH wants to merge 1 commit into
prometheus:masterfrom
neoLsH:filesystem-dedup-label-sets
Open

neoLsH wants to merge 1 commit into
prometheus:masterfrom
neoLsH:filesystem-dedup-label-sets

Conversation

@neoLsH

@neoLsH neoLsH commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

What this fixes

The filesystem collector deduplicates mount table entries on the whole filesystemLabels struct, but no metric carries all of those fields. Two entries that differ only in a field no metric exposes are treated as distinct, and then emit byte-identical series — which client_golang rejects, failing the whole scrape rather than just dropping the repeated filesystem.

label set fields
every metric except node_filesystem_mount_info device, mountpoint, fstype, device_error
node_filesystem_mount_info device, major, minor, mountpoint
dedup key today both of the above plus mountOptions, superOptions, major, minor

Each set now gets a key matching the labels it actually emits, so an entry is skipped per metric set instead of as a whole. Both device numbers in the case below stay visible through mount_info, which is where they belong, instead of one of them taking the rest of the filesystem's metrics down with it.

Why this is still broken after #3376

#3376 blanked mountOptions/superOptions before the check, removing two of the extra fields. major/minor are still in the key and still absent from every metric but mount_info, so the bug is still reachable. A multihomed NFS export reaches it: one mountinfo line per server address, same device/mountpoint/fstype, and a superblock — hence a minor — of its own per address.

The reports say so directly. #2514 and the filesystem reports in #2805 all fail with exactly 7 duplicated metrics — device_error, readonly, size_bytes, free_bytes, avail_bytes, files, files_free — and never node_filesystem_mount_info.

That rules the mount options out as the cause. Had the two entries differed only in options, major/minor would have matched too and mount_info would have collided as well, giving 8 errors. Seven errors with mount_info spared means the entries differ in the device numbers.

Reverting just the dedup keys on this branch reproduces that signature verbatim:

--- FAIL: TestFilesystemDeduplicatesRepeatMountWithDifferentDeviceNumbers
    gathering filesystem metrics: 7 error(s) occurred:
    * collected metric "node_filesystem_device_error" {...} was collected before with the same name and label values
    * collected metric "node_filesystem_readonly" {...} was collected before ...
    * collected metric "node_filesystem_size_bytes" {...} was collected before ...
    * collected metric "node_filesystem_free_bytes" {...} was collected before ...
    * collected metric "node_filesystem_avail_bytes" {...} was collected before ...
    * collected metric "node_filesystem_files" {...} was collected before ...
    * collected metric "node_filesystem_files_free" {...} was collected before ...

This is also the decoupling @SuperQ was pointing at in #3376 (comment)"this is tricky to fix cleanly as we're overloading the 'labels' with non-label filesystem information." major/minor are exactly that: non-label information for everything but mount_info.

Tests

filesystem_common.go is built on every supported platform, so filesystem_common_test.go runs on all of them including CI's Linux job. Emission moved into collectStats because Update calls the platform-specific GetStats() and offered no way to feed in a mount table.

Three cases, each asserting the full metric set so a silently dropped or extra series fails:

  • different device numbers, same fstype — the bug above; mount_info keeps 2 series, the other 7 collapse to 1
  • different fstype, same device numbers — the mirror case; the 7 keep 2 series, mount_info collapses to 1. This is why mount_info needs a key of its own rather than a subset of the first
  • identical repeat — pins the behaviour the original dedup intended, and passes both before and after

Gathered through prometheus.NewPedanticRegistry(), which rejects a repeated label set the same way a scrape does.

e2e

Unaffected. filesystem is in the e2e disabled_collectors list (it reads the real mount table, not fixtures), so no golden file contains a single node_filesystem_ line and nothing needs regenerating.

Fixes #2514

Also addresses the filesystem-collector reports in #2805. Deliberately not using a closing keyword there: that issue additionally covers node_fstab_mount_status emitted by the textfile collector's fstab-check.sh, which this change does not touch.

The mount table deduplication keys on the whole filesystemLabels struct,
but no metric carries all of those fields. Two entries that differ only
in a field no metric exposes are treated as distinct and then emit
byte-identical series, which client_golang rejects. That fails the whole
scrape rather than just dropping the repeated filesystem.

filesystemLabels is wider than either emitted label set: every metric
except node_filesystem_mount_info carries device, mountpoint, fstype and
device_error, while mount_info carries device, major, minor and
mountpoint. Key each set on the labels it actually emits so an entry is
skipped per metric set instead of as a whole.

Blanking the mount options in prometheus#3376 removed two of the extra fields, but
major and minor are still part of the key and still absent from every
metric but mount_info, so the bug remains reachable. A multihomed NFS
export reaches it: one mountinfo line per server address, same device,
mount point and fstype, and a superblock of its own per address.

That matches the reports, which all fail with exactly seven duplicated
metrics and never mention mount_info. Had the entries differed only in
their options the device numbers would match too and mount_info would
collide as well, giving eight.

Emission moves into collectStats because Update calls the
platform-specific GetStats and offered no way to feed in a mount table.
The filesystem collector is disabled in the end-to-end tests, so no
golden output changes.

Fixes prometheus#2514

Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
@neoLsH
neoLsH force-pushed the filesystem-dedup-label-sets branch from f0df4d2 to d4b4358 Compare September 13, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

node_filesystem collector unable deduplicate data from multihomed mounts like nfs

1 participant