Alerting is an art. One must be sure to alert just enough to be aware of all problems arising in the monitored system while at the same time not drown out the signal with excess noise. In this blogpost we’ll explain some of the best practices to use when alerting with Prometheus.
read morealerting
All posts tagged alerting
Alerting on crash loops with Prometheus
Brian Brazil March 19, 2018If your applications are restarting regularly, whether due to segfaults or OOMs, it’d be nice to know.
read moreAlerting on gauges in Prometheus 2.0
Conor Broderick February 12, 2018One of the major changes introduced in Prometheus 2.0 was that of staleness handling. Previously for instant vectors, Prometheus would return a point up to 5 minutes in the past which caused a number of different issues.
read moreWhat’s the difference between group_interval, group_wait, and repeat_interval?
Conor Broderick December 18, 2017In this blogpost we try and clear up some confusion by outlining the key differences between commonly confused alerting configuration options: group_interval, group_wait, and repeat_interval.
Using time series as alert thresholds
Brian Brazil December 4, 2017Usually alert thresholds are hardcoded in the alert. In more sophisticated setups, it would be useful for it to be parameterised based on another time series.
read moreAt what point should you consider an alert resolved?
read moreAvoid irate() in alerts
Brian Brazil August 28, 2017While the irate() function is useful for granular graphs, it is not suitable for alerting.
Combining alert conditions
Brian Brazil March 27, 2017Prometheus alerts use the same powerful PromQL expressions as queries and graphs. This can be used to produce sophisticated alerts.
read moreBooleans, logic and math
Brian Brazil March 6, 2017Prometheus doesn’t have an explicit boolean type or functionality. However there is a convention and enough power in PromQL to work with booleans.
read moreUsing labels to direct email notifications
Brian Brazil February 13, 2017A handy feature of the Alertmanager is that almost all notification fields are templatable. This can be used to route emails based on labels.
read more