Address already in use (os error 98)

[hightwoltt]

When trying to configure Vector, I encounter an error.
2025-01-15T06:32:04.648465Z INFO vector: Vector has started. debug="false" version="0.43.1" arch="x86_64" revision="e30bf1f 2024-12-10 16:14:47.175528383" 2025-01-15T06:32:04.651295Z ERROR vector::internal_events::socket: Error binding socket. error=error creating server listener: Address already in use (os error 98) error_code="socket_bind" error_type="io_failed" stage="receiving" mode=tcp internal_log_rate_limit=true 2025-01-15T06:32:04.651461Z ERROR vector::app: An error occurred that Vector couldn't handle: error creating server listener: Address already in use (os error 98). 2025-01-15T06:32:04.651511Z INFO vector: Vector has stopped.
I can only see this error if I go into the container and type the "Vector" command, when starting the container itself or when trying to debug using docker logs vector -f this error is not displayed.

Vector runs in a Docker container, which is running in a single instance. There are no more containers or Vector instances on the host.

The container was started using:
docker run -d \ --name vector \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /etc/vector/vector.yaml:/etc/vector/vector.yaml:ro \ timberio/vector:0.43.1-debian \ --config /etc/vector/vector.yaml

I tried reinstalling the container.
Can you tell me how to solve this problem? how do I see what makes Vector think it's already running on the host?

[jszwedko]

Hi @hightwoltt ,

Can you share your configuration?

[hightwoltt]

In my infrastructure has 2 Vector:
Both of them are deployed from Dokker, and they both have errors with the address already in use.
Vector agent(On the host where I monitor anything), and intermediate Vector. Vector agents send data to an intermediate vector, which in turn sends it to VMetrics.

Vector agent configuration:

data_dir: /var/lib/vector

api:
   enabled: true

sources:

  host_metrics:
    type: host_metrics
    scrape_interval_secs: 1
    collectors:
      - cpu
      - memory
      - disk
      - network
      - filesystem
      - process

transforms:
  add_host:
    type: remap
    inputs:
      - host_metrics
    source: |
      .host = "10.20.0.13"

sinks:

  intermediary_vector:
    type: vector
    inputs:
      - add_host
    address: "10.15.1.102:9020"

    batch:
      max_bytes: 1048576
      timeout_secs: 3

    healthcheck:
      enabled: true
    acknowledgements: true

Intermediate Vector:

# /etc/vector/vector.yaml

data_dir: /var/lib/vector

api:
   enabled: true

sources:

  intermediary_vector:
    type: vector
    address: "0.0.0.0:9020"

transforms:
  prepare_metrics:
    type: remap
    inputs:
      - intermediary_vector
    source: |
      .labels = {"source": "vector"}
      .value = 1.0
      .timestamp = now()

sinks:

  # VictoriaMetrics VM-Insert
  vm_insert:
    type: prometheus_remote_write
    inputs:
      - prepare_metrics
    endpoint: 'http://1.1.1.103:8480/insert/0/prometheus/api/v1/write'

    auth:
      strategy: "basic"
      user: "vector_user"
      password: "passwd"

    batch:
      max_bytes: 1048576
      timeout_secs: 3

    healthcheck:
      enabled: false

[jszwedko]

Interesting, nothing is immediately jumping out from your config. Are you sure nothing else is listening on the ports you are binding to? You could use lsof to check.

[hightwoltt]

Here is lsof | grep TCP command output:

vector       1               root    9u     IPv4          570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1               root   10u     IPv4          570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   7 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   7 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   8 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   8 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   9 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   9 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  10 vector-wo root    9u     IPv4    570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  10 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  12 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  12 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  13 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  13 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  14 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  14 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  15 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  15 vector-wo root   10u     IPv4     570656580       0t0       TCP localhost:8686 (LISTEN)