Conversation
The mount table deduplication keys on the whole filesystemLabels struct, but no metric carries all of those fields. Two entries that differ only in a field no metric exposes are treated as distinct and then emit byte-identical series, which client_golang rejects. That fails the whole scrape rather than just dropping the repeated filesystem. filesystemLabels is wider than either emitted label set: every metric except node_filesystem_mount_info carries device, mountpoint, fstype and device_error, while mount_info carries device, major, minor and mountpoint. Key each set on the labels it actually emits so an entry is skipped per metric set instead of as a whole. Blanking the mount options in prometheus#3376 removed two of the extra fields, but major and minor are still part of the key and still absent from every metric but mount_info, so the bug remains reachable. A multihomed NFS export reaches it: one mountinfo line per server address, same device, mount point and fstype, and a superblock of its own per address. That matches the reports, which all fail with exactly seven duplicated metrics and never mention mount_info. Had the entries differed only in their options the device numbers would match too and mount_info would collide as well, giving eight. Emission moves into collectStats because Update calls the platform-specific GetStats and offered no way to feed in a mount table. The filesystem collector is disabled in the end-to-end tests, so no golden output changes. Fixes prometheus#2514 Signed-off-by: 霏承 <huangleshu.hls@alibaba-inc.com>
neoLsH
force-pushed
the
filesystem-dedup-label-sets
branch
from
September 13, 2026 13:29
f0df4d2 to
d4b4358
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
The filesystem collector deduplicates mount table entries on the whole
filesystemLabelsstruct, but no metric carries all of those fields. Two entries that differ only in a field no metric exposes are treated as distinct, and then emit byte-identical series — which client_golang rejects, failing the whole scrape rather than just dropping the repeated filesystem.node_filesystem_mount_infodevice,mountpoint,fstype,device_errornode_filesystem_mount_infodevice,major,minor,mountpointmountOptions,superOptions,major,minorEach set now gets a key matching the labels it actually emits, so an entry is skipped per metric set instead of as a whole. Both device numbers in the case below stay visible through
mount_info, which is where they belong, instead of one of them taking the rest of the filesystem's metrics down with it.Why this is still broken after #3376
#3376 blanked
mountOptions/superOptionsbefore the check, removing two of the extra fields.major/minorare still in the key and still absent from every metric butmount_info, so the bug is still reachable. A multihomed NFS export reaches it: one mountinfo line per server address, same device/mountpoint/fstype, and a superblock — hence a minor — of its own per address.The reports say so directly. #2514 and the filesystem reports in #2805 all fail with exactly 7 duplicated metrics —
device_error,readonly,size_bytes,free_bytes,avail_bytes,files,files_free— and nevernode_filesystem_mount_info.That rules the mount options out as the cause. Had the two entries differed only in options,
major/minorwould have matched too andmount_infowould have collided as well, giving 8 errors. Seven errors withmount_infospared means the entries differ in the device numbers.Reverting just the dedup keys on this branch reproduces that signature verbatim:
This is also the decoupling @SuperQ was pointing at in #3376 (comment) — "this is tricky to fix cleanly as we're overloading the 'labels' with non-label filesystem information."
major/minorare exactly that: non-label information for everything butmount_info.Tests
filesystem_common.gois built on every supported platform, sofilesystem_common_test.goruns on all of them including CI's Linux job. Emission moved intocollectStatsbecauseUpdatecalls the platform-specificGetStats()and offered no way to feed in a mount table.Three cases, each asserting the full metric set so a silently dropped or extra series fails:
mount_infokeeps 2 series, the other 7 collapse to 1mount_infocollapses to 1. This is whymount_infoneeds a key of its own rather than a subset of the firstGathered through
prometheus.NewPedanticRegistry(), which rejects a repeated label set the same way a scrape does.e2e
Unaffected.
filesystemis in the e2edisabled_collectorslist (it reads the real mount table, not fixtures), so no golden file contains a singlenode_filesystem_line and nothing needs regenerating.Fixes #2514
Also addresses the filesystem-collector reports in #2805. Deliberately not using a closing keyword there: that issue additionally covers
node_fstab_mount_statusemitted by the textfile collector'sfstab-check.sh, which this change does not touch.