Prometheus node_exporter fails to start inside container

Written by Claudio Kuenzler - 0 comments

Published on June 18th 2025 - last updated on June 19th 2025 - Listed in Containers Linux Docker Observability

---

While working on a project which uses Gitlab pipelines, I added additional tests for automated QA. One of these tests is to start multiple Prometheus exporters and verify if the binaries would start. 

Interestingly the pipeline failed.

node_exporter problem with udev path

In the output of the failed tasks the following error could be seen:

time=2025-06-18T14:53:37.400Z level=ERROR source=diskstats_linux.go:264 msg="Failed to open directory, disabling udev device properties" collector=diskstats path=/run/udev/data

Unfortunately this does not show which of the started Prometheus exporters caused the problem. But a quick research led to a GitHub issue mentioning the same problem.

The problem turns out to be a container-related problem and a non-existing (default) udev path. The default path is set to /run/udev/data, which can be found on any "real" machine (and VMs).

But inside a container, this path doesn't exist and needs to be adjusted to: /rootfs/run/udev/data.

To do this, node_exporter needs to be started with the relevant parameter --path.udev.data:

- echo "Starting node-exporter (binary)"
- ./exporters/node_exporter --path.udev.data=/rootfs/run/udev/data &

After this change in .gitlab-ci.yaml, node_exporter successfully started and the pipeline was fixed. 

Additional workaround needed for EL8

While the pipeline was working nicely for EL9 using a Rocky Linux 9 container, it still failed on EL8 with Rocky Linux 8. According to the task's output, there was no /rootfs path inside the Rocky8 container. 

As a workaround, the path (/rootfs/run/udev) can be manually created before launching node_exporter:

- mkdir -p /rootfs/run/udev
- echo "Starting node-exporter (binary)"
- ./exporters/node_exporter --path.udev.data=/rootfs/run/udev/data &

This fixed the pipeline jobs for EL8, too.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.