Reliable Insights

A blog on monitoring, scale and operational sanity

Taking snapshots of Prometheus data

Prometheus 2.0 added a feature to take snapshots, let’s see how to use it.

 

While Prometheus is not intended to be a long-term durable data store (that’s what remote storage is for), some users may still wish to take backups of their data. With Prometheus 1.x this was a slow and disruptive process, requiring Prometheus to be completely restarted. The good news is that, due to its new storage engine, Prometheus 2.0 has a much better way of doing this.

To use it, you must enable the Admin API endpoints when running Prometheus:

$ ./prometheus --storage.tsdb.path=data/ --web.enable-admin-api

Then you can use a simple HTTP POST request to ask for a snapshot:

$ echo -n | POST http://localhost:9090/api/v2/admin/tsdb/snapshot 
{"name":"2017-11-10T14:45:57Z-38b7d4afb0900484"}

Here a few seconds later it has returned the name of the new snapshot in a JSON object. If you look under the snapshots directory of your data directory you’ll see this snapshot:

$ cd data/snapshots
$ ls
2017-11-10T14:45:57Z-38b7d4afb0900484

You can then copy this directory off to wherever you like.

The snapshots are comprised of hard links of existing blocks, and a dump of the current open blocks. As hard links are in use this means that the snapshots of older blocks take no additional disk space as there’s only one copy kept on disk, however you may break Prometheus if you change them, their permissions or their user/group. When you’re done you can rm -rf the snapshot directory, as while the snapshot takes little additional disk space initially, once the original block gets deleted/compacted the snapshot would then be what is keeping that disk space used.

 

Want to architect your monitoring for reliability? Contact us.

Brian BrazilTaking snapshots of Prometheus data
Share this post

Related Posts