Skip to content

inhibitor merge cache and index so there's only one lock - #5559

Draft
Spaceman1701 wants to merge 1 commit into
prometheus:mainfrom
Spaceman1701:more-inhibitor-refactor
Draft

Spaceman1701 wants to merge 1 commit into
prometheus:mainfrom
Spaceman1701:more-inhibitor-refactor

Conversation

@Spaceman1701

@Spaceman1701 Spaceman1701 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

This change replaces the store.Alerts with a new cache type in the inhibitor. The cache is essentially a merged store.Alerts and inhibitor index that work under a single lock. This allows operations to modify the cache and index atomically, solving a bunch of possible race conditions in the Inhibitor.

I believe that this resolves the last problems caught by the new tests added in #5449.

[BUGFIX]: Fix a race condition where the inhibitor could get into a bad state and fail to inhibit some alerts in some circumstances

@SoloJacobs

SoloJacobs commented Sep 14, 2026

Copy link
Copy Markdown
Contributor
❯ ./run.sh
=== inhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.79s)
ok  	github.com/prometheus/alertmanager/inhibit	4.799s
=== patchedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.93s)
ok  	github.com/prometheus/alertmanager/patchedinhibit	4.940s
=== v29inhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (10.24s)
ok  	github.com/prometheus/alertmanager/v29inhibit	10.254s
=== 5449updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (5.55s)
ok  	github.com/prometheus/alertmanager/5449updatedinhibit	5.556s
=== 5542updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (5.24s)
ok  	github.com/prometheus/alertmanager/5542updatedinhibit	5.250s
=== 5559updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.52s)
ok  	github.com/prometheus/alertmanager/5559updatedinhibit	4.527s

(inhibit test was rebased on main, so that is why this test no longer finds any missing mutes).

Signed-off-by: Ethan Hunter <ehunter@hudson-trading.com>
@Spaceman1701
Spaceman1701 force-pushed the more-inhibitor-refactor branch from 6ad58fa to bdaeab5 Compare September 15, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants