Reliable Insights

A blog on monitoring, scale and operational Sanity

June 17, 2019

Idempotent Cron Jobs are Operable Cron Jobs

Having to reconstruct how far a failed cron job had gotten and what exact parameters it was run with can be error prone and time consuming. There is a better way.

Read more

December 17, 2018

Limiting PromQL resource usage

Prometheus has gained a number of features to limit the impact of expensive PromQL queries.

Read more

August 27, 2018

Dealing with “too many open files”

While not a problem specific to Prometheus, being affected by the open files ulimit is something you're likely to run into at some point.

Read more

March 5, 2018

Using sample_limit to avoid overload

Worried that your application metrics might suddenly explode in cardinality? sample_limit can save you.

Read more

July 17, 2017

High Availability Prometheus Alerting and Notification

Prometheus is architected for reliability of alerting, how do you set it up?

Read more

July 14, 2016

Monitoring without Consensus

When designing a monitoring system and the datastore that goes with it, it can be tempting to go straight for a clustered highly consistent approach. But is that the best approach?

Read more

December 11, 2015

It’s overloaded? Try harder!

Failed requests are a fact of life, network weirdness and machine failures are inevitable. It can be tempting to simply retry the request when this happens, but this may cause more harm than good.

Read more

November 18, 2015

Do you have basic infrastructure?

When starting out it's easy to think that you need Docker, Kubernetes, Microservices, Continuous Deployment and all the other trending topics on Hacker News/Reddit/Lobsters. What do you really need?

Read more

November 2, 2015

Avoid outages: Beware the Knee

Your service's traffic is steadily growing, latency has increased a bit but it's within reason. One day you launch a new customer and the latency jumps through the roof causing an outage. What happened? You hit the knee.

Read more

October 29, 2015

Cooking a meal isn’t the same as running a restaurant

I enjoy cooking and regularly make scrumptious meals for myself. Does this mean that I'm capable of running a busy kitchen? Of course not! So why assume that all software engineers can automatically run production services?

Read more

twitter
youtube
linkedin

Blog   |   Training   |   Book   |   Careers   |   Privacy   |   Demo