Having to manually update a list of machines in a configuration file gets annoying after a while. One of the features of Prometheus is service discovery, allowing you to automatically discover and monitor your EC2 instances!
To start with, I spun up two EC2 instances running the node exporter. I ensured that ports 9100 and 9090 were open in the Security Group.
Next I setup an IAM user with the
AmazonEC2ReadOnlyAccess policy, noting down the access and secret key. This policy gives the user permission to list EC2 instances.
Finally I installed Prometheus on one of the machines, with the following configuration in
global: scrape_interval: 1s evaluation_interval: 1s scrape_configs: - job_name: 'node' ec2_sd_configs: - region: eu-west-1 access_key: PUT_THE_ACCESS_KEY_HERE secret_key: PUT_THE_SECRET_KEY_HERE port: 9100
/consoles/index.html on port 9090 of the Prometheus server I see my two nodes have been automatically discovered:
This is what we wanted, but it'd be nice if we could have the instance ID displayed rather than the private IP address. Maybe we also want to only scrape some nodes based on a tag.
Relabelling gives you the power to customise what labels Prometheus applies to your target. For an example, I put the following into
/etc/prometheus/prometheus.yml and reloaded Prometheus:
global: scrape_interval: 1s evaluation_interval: 1s scrape_configs: - job_name: 'node' ec2_sd_configs: - region: eu-west-1 access_key: PUT_THE_ACCESS_KEY_HERE secret_key: PUT_THE_SECRET_KEY_HERE port: 9100 relabel_configs: # Only monitor instances with a Name starting with "SD Demo" - source_labels: [__meta_ec2_tag_Name] regex: SD Demo.* action: keep # Use the instance ID as the instance label - source_labels: [__meta_ec2_instance_id] target_label: instance
Visiting the console again, I see that it has worked perfectly! The same technique can be used to monitor instances in an auto-scaling group.
As always with Prometheus you can drill down to get more details information about each instance:
Check out the documentation if you'd like to learn more about relabelling and service discovery.