Cannot start emqx single node setup

[aisven]

• [X ] BUG REPORT

---

BUG REPORT

Expected behavior

emqx starts up and accepts connections. The container does not exit.

Actual behavior

The process exits after about 30 seconds without any log output that would indicate the error.

logs:

The first log is the console output. To see that I made a run with the logging gelf configuration in docker-compose.yml commented out.

[sludwig@mqtt mqtt-emqx]$ docker-compose up
Starting mqtt-emqx-vod-dev-c ... done
Attaching to mqtt-emqx-vod-dev-c
mqtt-emqx-vod-dev-c    | node.max_ports=1048576
mqtt-emqx-vod-dev-c    | listener.tcp.external.acceptors=64
mqtt-emqx-vod-dev-c    | listener.ssl.external=58883
mqtt-emqx-vod-dev-c    | listener.ws.external=58083
mqtt-emqx-vod-dev-c    | allow_anonymous=false
mqtt-emqx-vod-dev-c    | listener.ssl.external.acceptors=32
mqtt-emqx-vod-dev-c    | node.process_limit=2097152
mqtt-emqx-vod-dev-c    | node.max_ets_tables=2097152
mqtt-emqx-vod-dev-c    | listener.wss.external=58084
mqtt-emqx-vod-dev-c    | listener.ws.external.acceptors=16
mqtt-emqx-vod-dev-c    | [email protected]
mqtt-emqx-vod-dev-c    | listener.tcp.external=51883
mqtt-emqx-vod-dev-c    | EMQX_LOADED_PLUGINS=emqx_management,emqx_auth_username,emqx_recon,emqx_retainer,emqx_dashboard
mqtt-emqx-vod-dev-c exited with code 1
[sludwig@mqtt mqtt-emqx]$ 

[sludwig@mqtt mqtt-emqx]$ docker exec -ti mqtt-emqx-vod-dev-c /bin/sh
/opt/emqx $ tail -f log/erlang.log.1 

=====
===== LOGGING STARTED Wed Jul 17 21:50:19 UTC 2019
=====
Exec: /opt/emqx/erts-10.3.4/bin/erlexec -boot /opt/emqx/releases/v3.2.0/emqx -mode embedded -boot_var ERTS_LIB_DIR /opt/emqx/erts-10.3.4/../lib -mnesia dir "/opt/emqx/data/mnesia/[email protected]" -config /opt/emqx/data/configs/app.2019.07.17.21.50.21.config -args_file /opt/emqx/data/configs/vm.2019.07.17.21.50.21.args -vm_args /opt/emqx/data/configs/vm.2019.07.17.21.50.21.args -- console
Root: /opt/emqx
/opt/emqx
Starting emqx on node [email protected]
Start http:management listener on 8080 successfully.
Start http:dashboard listener on 18083 successfully.
Start mqtt:tcp listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp listener on 0.0.0.0:51883 successfully.
Start mqtt:ws listener on 0.0.0.0:58083 successfully.
Start mqtt:ssl listener on 0.0.0.0:58883 successfully.
Start mqtt:wss listener on 0.0.0.0:58084 successfully.
EMQ X Broker 3.2.0 is running now!
Eshell V10.3.4  (abort with ^G)
([email protected])1> [sludwig@mqtt mqtt-emqx]$ 

^ here at the end you see that an Eshell promt is open somehow and then after some seconds the process exists

[sludwig@mqtt mqtt-emqx]$ docker exec -ti mqtt-emqx-vod-dev-c /bin/sh
/opt/emqx $ tail -f log/
dummy.txt     emqx.log.1    emqx.log.idx  erlang.log.1  run_erl.log
/opt/emqx $ tail -f log/run_erl.log 
run_erl [2530] Wed Jul 17 21:52:00 2019
Args before exec of shell:
run_erl [2530] Wed Jul 17 21:52:00 2019
argv[0] = sh
run_erl [2530] Wed Jul 17 21:52:00 2019
argv[1] = -c
run_erl [2530] Wed Jul 17 21:52:00 2019
argv[2] = exec "/opt/emqx/bin/emqx" "console"
[sludwig@mqtt mqtt-emqx]$ 

Test code

my docker-compose.yml

version: '3'
services:

  mqtt-emqx-vod-dev-s:
    image: emqx/emqx
    container_name: mqtt-emqx-vod-dev-c
    environment:
      EMQX_LOADED_PLUGINS: "emqx_management,emqx_auth_username,emqx_recon,emqx_retainer,emqx_dashboard"
      EMQX_NAME: emqxvoddev1
      EMQX_HOST: 10.0.3.5
      EMQX_NODE__COOKIE: 9beqyAQGcWQ1FvSiLnf3
      EMQX_LOG__CONSOLE: console
      EMQX_ALLOW_ANONYMOUS: 'false'
      EMQX_LISTENER__TCP__EXTERNAL: 51883
      EMQX_LISTENER__SSL__EXTERNAL: 58883
      EMQX_LISTENER__WS__EXTERNAL: 58083
      EMQX_LISTENER__WSS__EXTERNAL: 58084
      EMQX_LISTENER__API__MGMT: 58080
    logging:
      driver: gelf
      options:
        gelf-address: udp://10.0.5.4:12201
        tag: mqtt-emqx-vod-dev
    ports:
      - "51883:1883"
      - "58883:8883"
      - "58083:8083"
      - "58084:8084"
      - "58080:8080"
    volumes:
      - mqtt-emqx-vod-dev-lib-v:/opt/emqx/lib
      - mqtt-emqx-vod-dev-etc-v:/opt/emqx/etc
#      - mqtt-emqx-vod-dev-log-v:/opt/emqx/log
#      - mqtt-emqx-vod-dev-data-v:/opt/emqx/data
      - ./local/volumes/log:/opt/emqx/log
      - ./local/volumes/data:/opt/emqx/data
      - ./local/users/add_users.sh:/bin/add_users.sh

volumes:
  mqtt-emqx-vod-dev-lib-v:
  mqtt-emqx-vod-dev-etc-v:
#  mqtt-emqx-vod-dev-log-v:
#  mqtt-emqx-vod-dev-data-v:

All these files and directories on the Docker Host belong to root:docker and all of them are also readable and traversable by others. The stuff under local is also writeable by others.

EMQ version

Which branch are you on? If you know the Git revision then add it here as well.

emqx/emqx latest

Docker version

Docker version 18.09.7, build 2d0083d

How docker info?

docker info

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.09.7
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.21.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.777GiB
Name: mqtt.internal.graphicx.io
ID: 3JT3:4WRD:3DSK:565G:BFCZ:E4VO:EVK2:SUMD:MGZT:SV74:QPNE:5I3V
Docker Root Dir: /datadrive/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

System

CentOS 7

Hardware

standard modern Azrue VM

[aisven]

Also, it does not make a difference if I comment out the line with EMQX_NODE__COOKIE: 9beqyAQGcWQ1FvSiLnf3 in docker-compose.yml

[aisven]

While the container is active, docker ps is

[sludwig@mqtt mqtt-emqx]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                             NAMES
7296e871e13e        emqx/emqx           "/usr/bin/docker-ent…"   6 seconds ago       Up 5 seconds        4369/tcp, 5369/tcp, 6369/tcp, 11883/tcp, 18083/tcp, 0.0.0.0:51883->1883/tcp, 0.0.0.0:58080->8080/tcp, 0.0.0.0:58083->8083/tcp, 0.0.0.0:58084->8084/tcp, 0.0.0.0:58883->8883/tcp   mqtt-emqx-vod-dev-c

[aisven]

I came one step further in this analysis. If I revert to using network_mode: host in docker-compose.yml the broker starts up and does not exit after about 30 seconds. (Note that reverting back to using Docker Volumes without mapping to the Docker Host local file system did not make a difference.)

version: '3'
services:

  mqtt-emqx-vod-dev-s:
    image: emqx/emqx
    container_name: mqtt-emqx-vod-dev-c
    environment:
      EMQX_LOADED_PLUGINS: "emqx_management,emqx_auth_username,emqx_recon,emqx_retainer,emqx_dashboard"
      EMQX_NAME: emqxvoddev1
      EMQX_HOST: 10.0.3.5
#      EMQX_NODE__COOKIE: 9beqyAQGcWQ1FvSiLnf3
      EMQX_LOG__CONSOLE: console
      EMQX_ALLOW_ANONYMOUS: 'false'
#      EMQX_LISTENER__TCP__EXTERNAL: 51883
#      EMQX_LISTENER__SSL__EXTERNAL: 58883
#      EMQX_LISTENER__WS__EXTERNAL: 58083
#      EMQX_LISTENER__WSS__EXTERNAL: 58084
#      EMQX_LISTENER__API__MGMT: 58080
#    logging:
#      driver: gelf
#      options:
#        gelf-address: udp://10.0.5.4:12201
#        tag: mqtt-emqx-vod-dev
    network_mode: host
#    ports:
#      - "51883:1883"
#      - "58883:8883"
#      - "58083:8083"
#      - "58084:8084"
#      - "58080:8080"
    volumes:
      - mqtt-emqx-vod-dev-lib-v:/opt/emqx/lib
      - mqtt-emqx-vod-dev-etc-v:/opt/emqx/etc
      - mqtt-emqx-vod-dev-log-v:/opt/emqx/log
      - mqtt-emqx-vod-dev-data-v:/opt/emqx/data
#      - ./local/volumes/log:/opt/emqx/log
#      - ./local/volumes/data:/opt/emqx/data
      - ./local/users/add_users.sh:/bin/add_users.sh

volumes:
  mqtt-emqx-vod-dev-lib-v:
  mqtt-emqx-vod-dev-etc-v:
  mqtt-emqx-vod-dev-log-v:
  mqtt-emqx-vod-dev-data-v:

[Rory-Z]

Specify image as emqx/emqx:v3.2.0 and remove EMQX_HOST to start normal

[PotatoGuo]

@sourcekick Just exactly same as I've met. I can normally run the emqx image without enviroment confiuration EMQX_NODE__NAME(or any configuration), but with this, the container died silently, and the logs are same with what you displayed

[Rory-Z]

@PotatoGuo Can you share your environment configuration?

[PotatoGuo]

@zhanghongtong I just added -e EMQX_NODE__NAME="[email protected]"

[Rory-Z]

@PotatoGuo The IP address in the EMQX_NODE__NAME needs to be consistent with the IP address of the container, or else an exception error will result

[PotatoGuo]

@zhanghongtong Yes, it worked, Thanks!

[Rory-Z]

@PotatoGuo You're welcome :)