Reliable Insights

A blog on monitoring, scale and operational Sanity

August 6, 2018

Aggregating across batch job runs with push_time_seconds

For counting how many times a thing has happened you can use a counter and rate(), but that doesn't work across batch jobs.

 

Read more

July 23, 2018

Absent Alerting for Scraped Metrics

In the previous post we looked at dealing with when all the targets for a job had disappeared. What if you wanted to alert on specific metrics from one target disappearing?

Read more

July 16, 2018

Absent Alerting for Jobs

Alerting on numbers being too big or small is easy with Prometheus. But what if the numbers go missing?

Read more

May 28, 2018

Extracting raw samples from Prometheus

Sometimes you want the raw samples inside Prometheus for analysis or debugging. How do you get that?

Read more

April 30, 2018

Identifying expensive alerting rules

Since Prometheus 2.1 there is a feature to view alerting rule evaluation times in the rules UI. In this blogpost we'll see an example of how this can be used to identify an expensive rule expression.

Read more

April 23, 2018

Why can count(x > 5) not return 0?

When using the count aggregation operator you may have noticed that it sometimes returns nothing rather than 0. Why is this?

Read more

March 19, 2018

Alerting on crash loops with Prometheus

If your applications are restarting regularly, whether due to segfaults or OOMs, it'd be nice to know.

Read more

February 12, 2018

Alerting on gauges in Prometheus 2.0

One of the major changes introduced in Prometheus 2.0 was that of staleness handling. Previously for instant vectors, Prometheus would return a point up to 5 minutes in the past which caused a number of different issues.

Read more

February 5, 2018

What percentage of time is my service down for?

Have you ever wondered what percentage of time a given service or application spends up or down?

Read more

January 1, 2018

Rule groups for hierarchical aggregation

Prometheus 2.0 brought with it rule groups, making hierarchical aggregation easier than ever.

Read more

twitter
youtube
linkedin

Blog   |   Training   |   ´╗┐Book   |   Careers   |   Privacy   |   Demo