In this blogpost we'll be looking at how to setup Prometheus and use it to scrape the metrics of another service deployed using OpenShift and Docker on a local machine.
For this simple example, we'll be using Google's cAdvisor to export the metrics that Prometheus will be scraping from.
In order to do this we'll be using the OpenShift command line tool, oc, which is available for all major operating systems on OpenShift's website.
For the sake of brevity, I'll assume you already have Docker installed and the daemon running on your machine. Note that you'll have to launch the Docker daemon with the following argument
Once everything is installed, fire up your favourite terminal application and use the following command to start up a local OpenShift all-in-one cluster.
oc cluster up
With the cluster up you'll then need to login to the OpenShift platform that is now running on your machine.
oc login -u developer -p developer
You should now see a message stating that you have logged in successfully.
Let's create a new empty project with the following command:
oc new-project prometheus-example
Next we'll need to deploy the existing Docker images for both cAdvisor and Prometheus to the OpenShift platform.
To do this enter the following two commands to pull the images for cAdvisor and Prometheus from the public Docker Hub registry.
oc new-app google/cadvisor oc new-app prom/prometheus
This will store the two images in the OpenShift registry on our machine and create Kubernetes objects for the images, deploying each of them inside of their own pods.
If we head over to our OpenShift console, login (u:developer, p:developer), and click on our
prometheus-example project - we should see our two applications successfully deployed. Note that it may take a moment or two for them to spin up completely!
We now need to setup routes to these pods so that they are accessible on the network. Do this by running the two commands below.
oc expose service cadvisor oc expose service prometheus
Two routes have now been created to each of the deployed applications. The URLs for each can be seen from the Web UI console page above or from the following command:
oc get routes
This will produce a list of your routes and the URL you can use in your browser to reach them.
Before we start scraping metrics from the now deployed cAdvisor instance, we need to configure Prometheus.
In order to keep the image and configuration decoupled, we'll be using the
ConfigMap object to inject our Prometheus deployment with the appropriate configuration data.
Let's create the configuration file and use it to create the desired ConfigMap.
cat <<'EOF' > prometheus.yml global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'cadvisor' static_configs: - targets: ['cadvisor:8080'] EOF oc create configmap prom-config-example --from-file=prometheus.yml
Next we'll need to edit the deployment configuration for Prometheus to include this ConfigMap.
oc edit dc/prometheus
There's two parts we need to add in here.
The first is a new volume name with our ConfigMap and the second is the volumeMount which will give the path of where the prometheus.yml file will be.
First let's add the new volume:
Now we'll add the new volumeMount:
Here's a gist of my config file in case you would like to take a closer look.
The following ConfigMap you just created and added to your deployment will now result in the prometheus.yml file being generated at /etc/prometheus/ with the contents of the file config file we generated on our machine earlier.
This configuration will configure Prometheus to scrape both itself and the metrics generated by cAdvsisor.
Head over to your running Prometheus instance on OpenShift and execute the
up PromQL query in the Expression browser to see if everything is setup correctly.
You should see two results returned to the console representing the Prometheus and cAdvisor instances, each with a value of 1 indicating that they're running and successfully being scraped by Prometheus itself.
For something a little more interesting, let's take a look at the
container_cpu_system_seconds_total metric which when entered into the Expression browser will produce a time series of the cumulative system cpu time consumed in seconds over time.
In the graph below, I've used the PromQL
rate() method to calculate the per-second average rate of increase of the time series in the range vector.
Note that as this is just a quick demo, we've manually set our targets for Prometheus to scrape. In reality you would use Kubernetes service discovery to locate the cAdvisor service.
Want to learn more about monitoring with Prometheus on OpenShift? Contact us.