Getting metrics from Apache logs using the grok exporter

Not all applications produce useful metrics, but some of them do produce logs.

There are a class of applications which have little to no useful metrics, however the information you'd like to have can be found in logs. One option to deal with this is the grok exporter. Let's say you were working with Apache, here are some example logs from a typical access.log:

x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1"
x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0"
x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"

To start off download, configure, and run the grok exporter:

wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip
unzip grok_exporter-*.zip
cd grok_exporter*amd64
cat << 'EOF' > config.yml
global:
    config_version: 2
input:
    type: file
    path: access.log
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: apache_http_response_codes_total
      help: HTTP requests to Apache
      match: '%{COMBINEDAPACHELOG}'
      labels:
          method: '{{.verb}}'
          path: '{{.request}}'
          code: '{{.response}}'
server:
    port: 9144
EOF
./grok_exporter -config config.yml

If you visit :9144/metrics you'll see the metric:

# HELP apache_http_response_codes_total HTTP requests to Apache
# TYPE apache_http_response_codes_total counter
apache_http_response_codes_total{code="200",method="GET",path="/"} 5
apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1
apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1
apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1

For the sake of demonstration readall has been enabled, which will have the exporter read the log files from the start rather than the usual desired behaviour of only processing newly added log lines.

To back up a bit, Grok is a way that you can parse log lings in Logstash (Logstash is the L in the ELK stack). Accordingly patterns have been written for many common applications, including Apache. The Grok exporter allows you to build on the existing corpus of patterns, plus add your own, to extract metrics from logs. COMMMONAPACHELOG from above for example is defined as

COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

which itself relies on many other patterns. So you should be glad you don't have to write that yourself from scratch. The label fields provide you with Go templating (as seen in Prometheus alerting and notification templating) to access these values.

Beyond counting log lines, the Grok exporter can allow you to observe event sizes such as how many bytes are in responses:

    - type: summary
      name: apache_http_response_bytes
      help: Size of HTTP responses
      match: '%{COMMONAPACHELOG}'
      value: '{{.bytes}}'

Or gauges such as when the last request was:

    - type: gauge 
      name: apache_http_last_request_seconds
      help: Timestamp of the last HTTP request
      match: '%{COMMONAPACHELOG}'
      value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'

This is using the grok exporter's timestamp function, which is basically Go's time.Parse. There's also a divide function, in case you need to convert milliseconds to seconds.

This is a small taste of what you can do with the Grok exporter. As you can see parsing logs does require some configuration, so you should only use an exporter like this if you've no way to get the application itself to produce good metrics.

Want to get metrics into Prometheus? Contact us.

Published by Brian Brazil in Posts

Tags: exporters, logs, prometheus

Reliable Insights