Prometheus node_exporter fails to start inside container

Written by - 0 comments

Published on - last updated on June 19th 2025 - Listed in Containers Linux Docker Observability

While working on a project which uses Gitlab pipelines, I added additional tests for automated QA. One of these tests is to start multiple Prometheus exporters and verify if the binaries would start. 

Interestingly the pipeline failed.

gitlab pipeline failed

node_exporter problem with udev path

In the output of the failed tasks the following error could be seen:

node_exporter error in gitlab pipeline task output

time=2025-06-18T14:53:37.400Z level=ERROR source=diskstats_linux.go:264 msg="Failed to open directory, disabling udev device properties" collector=diskstats path=/run/udev/data

Unfortunately this does not show which of the started Prometheus exporters caused the problem. But a quick research led to a GitHub issue mentioning the same problem.

The problem turns out to be a container-related problem and a non-existing (default) udev path. The default path is set to /run/udev/data, which can be found on any "real" machine (and VMs).

But inside a container, this path doesn't exist and needs to be adjusted to: /rootfs/run/udev/data.

To do this, node_exporter needs to be started with the relevant parameter --path.udev.data:

- echo "Starting node-exporter (binary)"
- ./exporters/node_exporter --path.udev.data=/rootfs/run/udev/data &

After this change in .gitlab-ci.yaml, node_exporter successfully started and the pipeline was fixed. 

Additional workaround needed for EL8

While the pipeline was working nicely for EL9 using a Rocky Linux 9 container, it still failed on EL8 with Rocky Linux 8. According to the task's output, there was no /rootfs path inside the Rocky8 container. 

As a workaround, the path (/rootfs/run/udev) can be manually created before launching node_exporter:

- mkdir -p /rootfs/run/udev
- echo "Starting node-exporter (binary)"
- ./exporters/node_exporter --path.udev.data=/rootfs/run/udev/data &

This fixed the pipeline jobs for EL8, too.

pipeline job succeeded in el8


More recent articles:

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Observability   Office   OpenSearch   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder