Reliable Insights

We've previously looked at scraping services from consul and ssh checks. How can we combine those?

The use of relabelling for the blackbox exporter isn't tied to static_configs, you can use it with any service discovery mechanism. I'll presume you already have a working Consul setup.

To demonstrate first we run the blackbox exporter:

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar -xzf blackbox_exporter-*.linux-amd64.tar.gz
cd blackbox_exporter-*amd64
./blackbox_exporter

Then run Prometheus:

wget https://github.com/prometheus/prometheus/releases/download/v2.15.2/prometheus-2.15.2.linux-amd64.tar.gz
tar -xzf prometheus-*.tar.gz
cd prometheus-*amd64
cat <<'EOF' > prometheus.yml
global:
  scrape_interval: 10s
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    # Use consul as SD.
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*?)(:.*)?
        replacement: ${1}:22
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115
EOF
./prometheus

 

You'll note that this is very similar to the previous ssh example, which is to be expected. We are cheating a little here and as we're taking advantage of the fact that if relabelling for a given scrape config produces identical targets, those targets are merged. So even if a host has multiple Consul services (which it usually will), we'll end up with only one target per host in Prometheus which is fine for ssh running on a fixed well known port.

 

The Consul service discovery provides some metadata, so if you were probing something other than ssh on port 22 you could for example limit it to hosts that run the consul service:

      - source_labels: [__meta_consul_service]
        regex: consul
        action: keep

Or add target labels based on tags, probe specific ports for a given service and so on.

 

This shows some of the power of relabelling. There's no special code or configuration needed to tie Consul to the blackbox exporter, instead you can compose existing features.

 

Wondering about service discovery? Contact us.