Prometheus Process Exporter and the 15 character groupname limit (using .Comm)

Written by Claudio Kuenzler - 0 comments

Published on August 28th 2025 - Listed in Observability Linux

---

The Prometheus Process Exporter allows to retrieve metrics from all kinds of processes running on a system. With the correct YAML configuration, you can tell Process Exporter to retrieve metrics from all processes.

However you may run into a limitation if your machine runs multiple processes with similar names.

Process names are cut to 15 chars

The following Process Exporter configuration file allows to retrieve metrics from all running processes:

ck@linux ~ # cat process-exporter.yaml
process_names:
  - name: "{{.Comm}}"
    cmdline: 
    - '.+'

ck@linux ~ # ./process_exporter --config.path=process-exporter.yaml --web.listen-address=127.0.0.1:9256

The metrics can now be scraped using curl or a metrics scraper (such as Grafana Alloy or OpenTelemetry Collector):

ck@linux ~ # curl localhost:9256/metrics -s | grep 'memtype="resident"'
namedprocess_namegroup_memory_bytes{groupname="NetworkManager",memtype="resident"} 1.077248e+07
namedprocess_namegroup_memory_bytes{groupname="agetty",memtype="resident"} 94208
namedprocess_namegroup_memory_bytes{groupname="bash",memtype="resident"} 5.05856e+06
namedprocess_namegroup_memory_bytes{groupname="crond",memtype="resident"} 2.00704e+06
namedprocess_namegroup_memory_bytes{groupname="curl",memtype="resident"} 8.372224e+06
namedprocess_namegroup_memory_bytes{groupname="dbus-broker",memtype="resident"} 315392
namedprocess_namegroup_memory_bytes{groupname="dbus-broker-lau",memtype="resident"} 700416
namedprocess_namegroup_memory_bytes{groupname="grep",memtype="resident"} 2.404352e+06
namedprocess_namegroup_memory_bytes{groupname="process_exporte",memtype="resident"} 1.9275776e+07
namedprocess_namegroup_memory_bytes{groupname="rsyslogd",memtype="resident"} 1.6420864e+07
namedprocess_namegroup_memory_bytes{groupname="systemd",memtype="resident"} 8.962048e+06
namedprocess_namegroup_memory_bytes{groupname="systemd-journal",memtype="resident"} 1.83083008e+08
namedprocess_namegroup_memory_bytes{groupname="systemd-logind",memtype="resident"} 3.764224e+06

In this example I simply grepped the metrics for the resident (RSS) memory usage of each running process on this Linux machine. 

Each process can be identified by the label "groupname". However one thing catches the trained eye: The process name in the groupname label might be cut, if the command is longer than 15 chars. This can be verified with - *drumrolls* - the process_exporter itself:

namedprocess_namegroup_memory_bytes{groupname="process_exporte",memtype="resident"} 1.9275776e+07

The string "process_exporter" contains 16 characters, however only 15 are showing in the label. 

Where does this 15 character limit come from?

The 15 character limit is documented in the README of the process_exporter:

Every name in the procnames list becomes a process group. The default name of a process is the value found in the second field of /proc//stat ("comm"), which is truncated at 15 chars. Usually this is the same as the name of the executable.

Manually checking the path /proc/PID/stat proofs this:

ck@linux ~ # cat /proc/$(pidof process_exporter)/stat
56003 (process_exporte) S 1 56003 56003 0 -1 4194560 1532 0 136 0 2 5 0 0 20 0 15 0 618812689 1268781056 3971 18446744073709551615 4194304 9520279 140726844806864 0 0 0 1002060288 0 2143420159 0 0 0 17 17 0 0 0 0 0 15122432 15493840 43151360 140726844808738 140726844808866 140726844808866 140726844809169 0

The second field indeed shows a truncated process name: process_exporte instead of process_exporter.

Missing metrics

As long as the process names are very different, the 15 character limit isn't a big problem. But on certain machines you might be running the same process multiple times, located under different paths. 

A quick example to show this is by running a second instance of Process Exporter on the same machine:

anotheruser@linux ~ $ ./process_exporter --config.path=process-exporter.yaml --web.listen-address=127.0.0.1:9257
2025/08/28 09:58:35 Reading metrics from /proc based on "process-exporter.yaml"
level=info ts=2025-08-28T09:58:35.513Z caller=tls_config.go:313 msg="Listening on" address=127.0.0.1:9257
level=info ts=2025-08-28T09:58:35.513Z caller=tls_config.go:316 msg="TLS is disabled." http2=false address=127.0.0.1:9257
[...]

With ps we can now see two process_exporter processes running on this machine:

anotheruser@linux ~ $ ps aux| grep process_exporter
ck     56003  0.0  0.0 1239044 15088 ?       Ssl  09:36   0:00 /home/ck/process_exporter --config.path=/home/ck/process-exporter.yaml --web.listen-address=0.0.0.0:9256
another+   56826  0.0  0.0 1238468 16960 pts/5   Sl   09:58   0:00 ./process_exporter --config.path=process-exporter.yaml --web.listen-address=127.0.0.1:9257
another+   56869  0.0  0.0   6464  2320 pts/5    S+   10:01   0:00 grep --color=auto process_exporter

anotheruser@linux ~ $ pidof process_exporter
56826 56003

Now checking the RSS metrics for all processes:

anotheruser@linux ~ $ curl localhost:9257/metrics -s | grep 'memtype="resident"'
namedprocess_namegroup_memory_bytes{groupname="(sd-pam)",memtype="resident"} 4.186112e+06
namedprocess_namegroup_memory_bytes{groupname="NetworkManager",memtype="resident"} 1.077248e+07
namedprocess_namegroup_memory_bytes{groupname="agetty",memtype="resident"} 94208
namedprocess_namegroup_memory_bytes{groupname="bash",memtype="resident"} 9.629696e+06
namedprocess_namegroup_memory_bytes{groupname="crond",memtype="resident"} 2.00704e+06
namedprocess_namegroup_memory_bytes{groupname="curl",memtype="resident"} 8.572928e+06
namedprocess_namegroup_memory_bytes{groupname="dbus-broker",memtype="resident"} 315392
namedprocess_namegroup_memory_bytes{groupname="dbus-broker-lau",memtype="resident"} 700416
namedprocess_namegroup_memory_bytes{groupname="grep",memtype="resident"} 2.367488e+06
namedprocess_namegroup_memory_bytes{groupname="process_exporte",memtype="resident"} 3.2268288e+07
namedprocess_namegroup_memory_bytes{groupname="rsyslogd",memtype="resident"} 1.671168e+07
namedprocess_namegroup_memory_bytes{groupname="su",memtype="resident"} 6.979584e+06
namedprocess_namegroup_memory_bytes{groupname="systemd",memtype="resident"} 2.0672512e+07
namedprocess_namegroup_memory_bytes{groupname="systemd-journal",memtype="resident"} 1.83721984e+08
namedprocess_namegroup_memory_bytes{groupname="systemd-logind",memtype="resident"} 3.764224e+06

The is only one entry for "process_exporte" ! What happened to the second process?

Process Exporter only considered one of the processes in the metrics - because they have the same name.

Obviously the same problem applies to all kinds of scenarios. For instance if you have process names "mysuperfancyapp1", "mysuperfancyapp2" and "mysuperfancyapp3", they would all be cut to 15 chars. All of them would be named "mysuperfancyapp". Process Exporter would only show the metrics of one of these processes. This means: You lose the metrics of the other running processes.

Workaround: FullExe (path) instead of command name (.Comm)

Luckily there are multiple "template variables" available for the Process Exporter YAML configuration. The documentation shows:

Template variables available:

    {{.Comm}} contains the basename of the original executable, i.e. 2nd field in /proc/<pid>/stat
    {{.ExeBase}} contains the basename of the executable
    {{.ExeFull}} contains the fully qualified path of the executable
    {{.Username}} contains the username of the effective user
    {{.Matches}} map contains all the matches resulting from applying cmdline regexps
    {{.PID}} contains the PID of the process. Note that using PID means the group will only contain a single process.
    {{.StartTime}} contains the start time of the process. This can be useful in conjunction with PID because PIDs get reused over time.
    {{.Cgroups}} contains (if supported) the cgroups of the process (/proc/self/cgroup). This is particularly useful for identifying to which container a process belongs.

At the beginning of this article I showed the YAML configuration using the {{.Comm}} variable. This defines the value of the label "groupname". 

Let's switch to the {{.ExeFull}} variable, which contains the full path of the executable:

anotheruser@linux ~ $ vi process-exporter.yaml
anotheruser@linux ~ $ cat process-exporter.yaml
process_names:
  - name: "{{.ExeFull}}"
    cmdline: 
    - '.+'

anotheruser@linux ~ $ ./process_exporter --config.path=process-exporter.yaml --web.listen-address=127.0.0.1:9257 &
[1] 56965
2025/08/28 10:13:52 Reading metrics from /proc based on "process-exporter.yaml"
level=info ts=2025-08-28T10:13:52.078Z caller=tls_config.go:313 msg="Listening on" address=127.0.0.1:9257
level=info ts=2025-08-28T10:13:52.078Z caller=tls_config.go:316 msg="TLS is disabled." http2=false address=127.0.0.1:9257

Now let's check the metrics of this second running process_exporter:

anotheruser@linux ~ $ curl localhost:9257/metrics -s | grep 'memtype="resident"'
namedprocess_namegroup_memory_bytes{groupname="(sd-pam)",memtype="resident"} 4.186112e+06
namedprocess_namegroup_memory_bytes{groupname="-bash",memtype="resident"} 5.087232e+06
namedprocess_namegroup_memory_bytes{groupname="./process_exporter",memtype="resident"} 1.6592896e+07
namedprocess_namegroup_memory_bytes{groupname="/bin/bash",memtype="resident"} 5.087232e+06
namedprocess_namegroup_memory_bytes{groupname="/home/ck/process_exporter",memtype="resident"} 1.9726336e+07
namedprocess_namegroup_memory_bytes{groupname="/sbin/agetty",memtype="resident"} 94208
namedprocess_namegroup_memory_bytes{groupname="/sbin/init",memtype="resident"} 9.043968e+06
namedprocess_namegroup_memory_bytes{groupname="/usr/bin/dbus-broker-launch",memtype="resident"} 700416
namedprocess_namegroup_memory_bytes{groupname="/usr/lib/systemd/systemd",memtype="resident"} 1.1350016e+07
namedprocess_namegroup_memory_bytes{groupname="/usr/lib/systemd/systemd-journald",memtype="resident"} 1.83721984e+08
namedprocess_namegroup_memory_bytes{groupname="/usr/lib/systemd/systemd-logind",memtype="resident"} 3.764224e+06
namedprocess_namegroup_memory_bytes{groupname="/usr/sbin/NetworkManager",memtype="resident"} 1.077248e+07
namedprocess_namegroup_memory_bytes{groupname="/usr/sbin/crond",memtype="resident"} 2.00704e+06
namedprocess_namegroup_memory_bytes{groupname="/usr/sbin/rsyslogd",memtype="resident"} 1.671168e+07
namedprocess_namegroup_memory_bytes{groupname="curl",memtype="resident"} 8.62208e+06
namedprocess_namegroup_memory_bytes{groupname="dbus-broker",memtype="resident"} 315392
namedprocess_namegroup_memory_bytes{groupname="grep",memtype="resident"} 2.37568e+06
namedprocess_namegroup_memory_bytes{groupname="su",memtype="resident"} 6.979584e+06

Eureka! The process_exporter process now shows up twice using the executed path!

anotheruser@linux ~ $ curl localhost:9257/metrics -s | grep 'memtype="resident"' | grep process_exporter
namedprocess_namegroup_memory_bytes{groupname="./process_exporter",memtype="resident"} 1.8690048e+07
namedprocess_namegroup_memory_bytes{groupname="/home/ck/process_exporter",memtype="resident"} 1.912832e+07

With this workaround, the full executed path is showing up as label value and it allows to separate metrics between processes with similar names.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.