Prometheus relabel configs are notoriously badly documented, so here’s how to do something simple that I couldn’t find documented anywhere: How to add a label to all metrics coming from a specific scrape target.
Example
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
# Add your relabel config under the scrape configs
relabel_configs:
# source label must be one that exists, so use __address__
- source_labels: [__address__]
# target label is the one you want to create
target_label: my_new_label
replacement: "my-new-label-value"
And there you have it.
This will create a new label “my_new_label” with the fixed value “my-new-label-value“.
How does it work?
If you don’t supply them, the default settings for a relabel_config are:
action: replaceregex: (.*)separator: ;
By choosing a single always existing source label (__address__ always exists), you are guaranteed to get a source match for replacing the target_label with. The default regex wil always match, which causes the replacement to be carried out. However, we’re not specifying any match group’s in our replacement string, so the entire string is just copied into target_label. This is just a very specific case of how you can use a relabel_config to copy (parts of) a label into another (new) label.
Hi, this is nice, but what is the benefit or difference to just adding new label for a target?
I mean like this:
static_configs:
– targets: [‘localhost:9090’]
labels:
my_new_label: my_new_label-value
Good question!
Because you can only add labels using the labels: setting to static_configs:
You can’t do this when you are using any of the service discovery scrape configurations.
Properly understanding how to use relabel_configs is essential when working with service discovery.
Badly documented alright.
Thank you for saving me the time and pain of figuring it out!
can overwriting the `__address__` predefined label be achieved ? the intent is to use the legacy monitoring which already collects a small set of system level metrics from thousands of hosts (so adding the equivalent number of ports to listen is not an option, and neither is switching to node exporter on real subject nodes -Thanks
How about when you have multiple targets?
E.g.
targets: [‘localhost:9090′,10.50.100.23:9090’]
Do you have to add each target in its own scrape config? Or is there a more efficient way to do it?
Exactly what I needed, thanks!
Hi.
I have this problem and maybe you have a solution for it. I would like to collect only a selected metric and I would like to add my own label to this:
static_configs:
– targets:
– ‘localhost:9100’
labels:
environment: TEST
metric_relabel_configs:
– source_labels: [__name__]
regex: ‘up|node_uname_info’
action: keep
With up I got only instance name. I wanted to add a new configuration to add a label, but then prometheus collected metrics twice
metric_relabel_configs:
– source_labels: [__name__]
regex: ‘up|node_uname_info’
action: keep
– source_label: [__name__]
regex: regex: “([a-zA-Z0-9-]+).*”
target_label: hostname
Any ideas how to do that?
Thank you! I was tearing my hair out trying to figure this out. The `source_labels` phrase makes no sense in this context since I have no intention of using the `__address__` here, but your method does work.