Reliable Insights

When relabelling is suggested for selecting targets with service discovery there's sometimes a misunderstanding that relabelling can only change labels. That's not the case.

The default action for relabelling is indeed replace, and it is the most commonly used action. There are other actions on offer though.

 

The drop and keep actions will halt processing if they do or don't match, respectively. For relabel_configs this will cause the target not to be scraped, for metric_relabel_configs this will cause a time series not to be ingested, for alert_relabel_configs an alert won't be sent to the Alertmanager and for write_relabel_configs the timeseries won't be sent to the remote write endpoint.

You can think of drop and keep as filters.

 

Here's an example for use with EC2 service discovery that will discard all machines whose name doesn't begin with "Example":

    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        regex: Example.*
        action: keep

We've also previously looked at using this with metric_relabel_configs to drop expensive timeseries.

 

Keep in mind that for all relabel actions other than drop and keep, processing will keep on going whether the regex matches or not.

 

Looking for expert advice on architecting Prometheus? Contact us.