We've previously looked at scraping services from consul and ssh checks. How can we combine those?

The use of relabelling for the blackbox exporter isn't tied to static_configs, you can use it with any service discovery mechanism. I'll presume you already have a working Consul setup.

To demonstrate first we run the blackbox exporter:

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar -xzf blackbox_exporter-*.linux-amd64.tar.gz
cd blackbox_exporter-*amd64

Then run Prometheus:

wget https://github.com/prometheus/prometheus/releases/download/v2.15.2/prometheus-2.15.2.linux-amd64.tar.gz
tar -xzf prometheus-*.tar.gz
cd prometheus-*amd64
cat <<'EOF' > prometheus.yml
  scrape_interval: 10s
  - job_name: 'blackbox'
    metrics_path: /probe
      module: [ssh_banner]
    # Use consul as SD.
      - server: 'localhost:8500'
      - source_labels: [__address__]
        regex: (.*?)(:.*)?
        replacement: ${1}:22
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__


You'll note that this is very similar to the previous ssh example, which is to be expected. We are cheating a little here and as we're taking advantage of the fact that if relabelling for a given scrape config produces identical targets, those targets are merged. So even if a host has multiple Consul services (which it usually will), we'll end up with only one target per host in Prometheus which is fine for ssh running on a fixed well known port.


The Consul service discovery provides some metadata, so if you were probing something other than ssh on port 22 you could for example limit it to hosts that run the consul service:

      - source_labels: [__meta_consul_service]
        regex: consul
        action: keep

Or add target labels based on tags, probe specific ports for a given service and so on.


This shows some of the power of relabelling. There's no special code or configuration needed to tie Consul to the blackbox exporter, instead you can compose existing features.


Wondering about service discovery? Contact us.