OpenTelemetry Collector error: failed to create SDK bind: address already in use

Written by Claudio Kuenzler - 0 comments

Published on August 13th 2025 - Listed in Observability Linux Systemd

---

While I'm helping a large enterprise to roll out a new telemetry/metrics based monitoring using Grafana Alloy and OpenTelemetry Collector (OtelCol), eventually I had to run into a problem with the default listener port (8888).

OtelCol start error: bind: address already in use

When OtelCol (Contrib) was started on a particular machine, the process didn't come up and showed the following error message:

Aug 13 04:27:24 rocky9 otelcol-contrib[2152]: Error: failed to create SDK: binding address 127.0.0.1:8888 for Prometheus exporter: listen tcp 127.0.0.1:8888: bind: address already in use
Aug 13 04:27:24 rocky9 otelcol-contrib[2152]: 2025/08/13 04:27:24 collector server run finished with error: failed to create SDK: binding address 127.0.0.1:8888 for Prometheus exporter: listen tcp 127.0.0.1:8888: bind: address already in use

The "address already in use" error clearly describes what the problem is: The listener port 8888 is already in use/occupied by another process.

root@rocky9 ~ # ss -lnp | grep 8888
tcp   LISTEN 0      10     0.0.0.0:8888   0.0.0.0:*    users:(("nc",pid=2032,fd=4))
tcp   LISTEN 0      10     [::]:8888      [::]:*    users:(("nc",pid=2032,fd=3)) 

Note: In the original situation, another daemon was listening on port tcp/8888, in this situation I simply reproduced the error with a netcat listener (nc -l 8888).

The listener port tcp/8888 is used by OtelCol for its own internal statistics and metrics. From the OpenTelemetry documentation:

By default, the Collector generates basic metrics about itself and exposes them using the OpenTelemetry Go Prometheus exporter for scraping at http://127.0.0.1:8888/metrics. 

To my current knowledge the listener can't be disabled, the listener port can be changed though.

How to change listener port in Collector YAML

The OtelCol configuration is a YAML file - which can grow very large and complex.

The listener configuration usually isn't even a part of the YAML, as it's a default listener and automatically enabled. To modify it, the listener port must be defined deep within the service::telemetry context:

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 8888

Note: Prior to OpenTelemetry Collector v0.123.0, the listener port configuration happened in the service::telemetry::metrics::address context. This is no longer valid. 

This is how the default configuration would look like. Obviously the port must be changed to something else than 8888, to avoid the conflict with another listener port. I also recommend to change the host from '0.0.0.0' to '127.0.0.1'.

After adjusting the port in the YAML, the OtelCol process can now be started anymore.

Change listener port using Systemd service

In the server environment, I mentioned at the beginning, OpenTelemetry Collector is (Enterprise Linux) packaged and can be installed through a dedicated system package from an internal RPM repository. This package also contains the Systemd service definitions for the "otelcol-contrib" service - effectively launching and managing the OtelCol process.

Systemd services can use environment variables or an external EnvironmentFile. On the other hand, OtelCol is able to read these environment variables and use them in the YAML configuration file.

This combination makes it possible, to use the environment variable in the port definition in the OtelCol YAML configuration - and the port itself is defined in the EnvironmentFile read by the Systemd service.

root@rocky9 ~ # cat /etc/otelcol-contrib-config.yaml
[...]
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: ${env:OTELCOL_PORT}
    logs:
      level: "INFO" # Log level for the collector's own logging. Change to "DEBUG" for very verbose logs
[...]

The Systemd service unit refers to an EnvironmentFile:

root@rocky9 ~ # grep Environment /usr/lib/systemd/system/otelcol-contrib.service
EnvironmentFile=/etc/sysconfig/otelcol-contrib

And inside this environment file - you guessed it right - the OTELCOL_PORT can be defined:

root@rocky9 ~ # cat /etc/sysconfig/otelcol-contrib
# Listener port for otelcol-contrib internal metrics
OTELCOL_PORT=8888

I kept the default listener port (8888) in this case, but it's easy for every user to change this. At least it's easier than fiddling around with the OtelCol YAML and risking to introduce a YAML syntax error.

Changed port - OtelCol startup fixed

On the server in question, the port was adjusted to 8889:

root@rocky9 ~ # cat /etc/sysconfig/otelcol-contrib
# Listener port for otelcol-contrib internal metrics
OTELCOL_PORT=8889

And the service stared up correctly, now listening on tcp/8889 on localhost:

root@rocky9 ~ # systemctl start otelcol-contrib

root@rocky9 ~ # ss -lnp | grep 8889
tcp   LISTEN 0      4096    127.0.0.1:8889  0.0.0.0:*    users:(("otelcol-contrib",pid=2343,fd=6))

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.