job label is one of the labels your targets will always have. So how can you use it?
job label is special as if after target relabelling if there is no
job label present on a target, then the value of the
job_name field will be used. In this way your targets will always have a
How best to use it then? What I would recommend is using the
job label to organise applications which do the same thing, which almost always means processes running the same binary with the exact same configuration. For example you might have web frontends with
job="frontend" and a Redis used as a cache with
job="redis". Using labels like this it's easy to aggregate across a job, for example CPU usage per job would be
sum by (job)(rate(process_cpu_seconds_total[5m])). As a counterexample a
job="kubernetes" for all of your processes running on Kubernetes isn't particularly useful, you should aim for more meaningful
If you were running different sets of Redis servers for different purposes, it'd make sense to have different
job labels for each set. Each set will likely have distinct performance characteristics after all, and aggregating them together would be unlikely to be of much use. If there are subdivisions within a set, such as sharding, it often makes sense to add a
shard label or similar to subdivide the job.
Something to avoid is encoding information beyond that what sort of binary/configuration the process is in a
job label. For example rather than having
job="redis_production" it'd be better to have a separate
env="production" to distinguish all of the production environment from development and staging environments. This makes it easier to do cross-environment analysis, without having to resort to regular expressions against label values.
Wondering how to organise your scrape configs? Contact us.