[Feature]: Allow Icinga Alert States to match Prometheus Alert Manager

[drapiti]

Describe the feature request

Currently alerts are picked up only as critical by the check. Allow Icinga to report warning if the Alert Manager reports warning for the same alert rather than critical.

[martialblog]

Hi, could you elaborate maybe a bit more what the use case is? The check plugin current is only to check directly at the Prometheus API, not the Alertmanager. Since the Alertmanager is a different tool with it's own API we decided not to include it (for now).

[drapiti]

Hi, could you elaborate maybe a bit more what the use case is? The check plugin current is only to check directly at the Prometheus API, not the Alertmanager. Since the Alertmanager is a different tool with it's own API we decided not to include it (for now).

The alerts in prometheus are the same. What I mean though is that if an alert in prometheus is in a warning state, I would like the prometheus icinga check to report warning for the same alert not critical as it does now.

[martialblog]

There is no real concept of "warning" in Prometheus, since everything is based on labels users can define what every state they need in any label they want. We decided to use the only constant state (the alert firing/pending/ok) to map to the Icinga states, that way it works for all Prometheus-Icinga setups without too much configuration.

For the use case you describe there's the option to add an additional label to the alerts that can then be used to override the wanted state in Icinga for firing alerts:

  -S, --label-key-state string      Use the given AlertRule label to override the exit state for firing alerts.
                                    If this flag is set the plugin looks for warning/critical/ok in the provided label key