DockerResponseServerError { status_code: 500, message: "container create failed (no logs from conmon): conmon bytes \"\": readObjectStart: expect { or n, but found \0, error found in #0 byte o

[jiridanek]

DockerResponseServerError { status_code: 500, message: "container create failed (no logs from conmon): conmon bytes \"\": readObjectStart: expect { or n, but found \0, error found in #0 byte of ...||..., bigger context ...||..." }
stack backtrace:

This seems to be caused by an update of podman or one of its dependencies

• Unpacking crun (101:1.12-0ubuntu22.04+obs65.5) over (0.17+dfsg-1.1) ... -> Unpacking crun (101:1.12-0ubuntu22.04+obs65.13) over (0.17+dfsg-1.1) ...
• Unpacking containers-common (4:1-0ubuntu22.04+obs20.52) ... -> Unpacking containers-common (4:1-0ubuntu22.04+obs20.56) ...
• Setting up conmon (2:2.1.8-0ubuntu22.04+obs16.33) ... -> Setting up conmon (2:2.1.9-0ubuntu22.04+obs17.1) ...
• Setting up crun (101:1.12-0ubuntu22.04+obs65.5) ... -> Setting up crun (101:1.12-0ubuntu22.04+obs65.13) ...
• Setting up containers-common (4:1-0ubuntu22.04+obs20.52) ... -> Setting up containers-common (4:1-0ubuntu22.04+obs20.56) ...

Version of podman itself between last successful and a series of failures has not changed

• Setting up podman (4:4.6.2-0ubuntu22.04+obs81.12) ...

[jiridanek]

https://github.com/skupperproject/skupper-router/actions/runs/7251189221/job/19753018142#step:8:80

[jiridanek]

This seems to be caused by the conmon upgrade, according to

• OCI Runtime Failed on upgrade containers/podman#21024
• conmon 2.1.9 not working with podman 4.8.2 and ubuntu 22.04 containers/conmon#475

[XaratosS]

So is there a fix for it?

[jiridanek]

@XaratosS using podman installed from linuxbrew was the fix for me at that time. What version of podman you use, where is that from, and what are your troubles? Ubuntu 24.04 has podman 4.9.3, so maybe if you need this for Github Actions, that version of podman may work well enough without the need to install anything newer

[XaratosS]

@jiridanek I am running it on Docker. Is Docker not supported anymore?

[jiridanek]

There should not be any problems with Docker, as far as I know. It's just that we tend to prefer podman for CI.

[XaratosS]

Hmmn then it might be a diffrent issue. Maybe I havent set up the right permissions for the token.

[XaratosS]

Sadly the issue is still persistent:

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Docker Version:

docker version
Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:42 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:42 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Edit: It looks like he cant write into the Database:

2024-05-27 13:55:29,613 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 execution node chosen {"type": "projectupdate", "task_id": 18, "state": "execution_node_chosen", "work_unit_id": null, "task_name": "Ansible Playbook update", "execution_node": "awx-1"}
2024-05-27 13:55:29,620 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 waiting {"type": "projectupdate", "task_id": 18, "state": "waiting", "work_unit_id": null, "task_name": "Ansible Playbook update"}
2024-05-27 13:55:29,695 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 pre run {"type": "projectupdate", "task_id": 18, "state": "pre_run", "work_unit_id": null, "task_name": "Ansible Playbook update"}
2024-05-27 13:55:29,699 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 preparing playbook {"type": "projectupdate", "task_id": 18, "state": "preparing_playbook", "work_unit_id": null, "task_name": "Ansible Playbook update"}
2024-05-27 13:55:29,721 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 running playbook {"type": "projectupdate", "task_id": 18, "state": "running_playbook", "work_unit_id": null, "task_name": "Ansible Playbook update"}
2024-05-27 13:55:29,728 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 work unit id received {"type": "projectupdate", "task_id": 18, "state": "work_unit_id_received", "work_unit_id": "NQo9qoKS", "task_name": "Ansible Playbook update"}
2024-05-27 13:55:29,736 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 work unit id assigned {"type": "projectupdate", "task_id": 18, "state": "work_unit_id_assigned", "work_unit_id": "NQo9qoKS", "task_name": "Ansible Playbook update"}
WARNING 2024/05/27 13:55:30 Could not read in control service: read unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
WARNING 2024/05/27 13:55:30 Could not close connection: close unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
2024-05-27 13:55:30,244 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 post run {"type": "projectupdate", "task_id": 18, "state": "post_run", "work_unit_id": "NQo9qoKS", "task_name": "Ansible Playbook update"}
2024-05-27 13:55:30,245 INFO     [072c8656] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 18
2024-05-27 13:55:30,307 INFO     [072c8656] awx.analytics.job_lifecycle projectupdate-18 finalize run {"type": "projectupdate", "task_id": 18, "state": "finalize_run", "work_unit_id": "NQo9qoKS", "task_name": "Ansible Playbook update"}
2024-05-27 13:55:30,311 WARNING  [072c8656] awx.main.dispatch project_update 18 (failed) encountered an error (rc=None), please see task stdout for details.
2024-05-27 13:55:31,359 WARNING  [-] awx.main.commands.run_callback_receiver Error in events bulk_create, will try indiviually, error: PostgreSQL text fields cannot contain NUL (0x00) bytes
2024-05-27 13:55:31,362 INFO     [-] awx.main.commands.run_callback_receiver Database Error Saving individual Event uuid=eddb48dd-b320-4a00-a87e-668277908afb try=1, error: PostgreSQL text fields cannot contain NUL (0x00) bytes

[jiridanek]

@XaratosS are you even using skupper at all? or did you just google for a docker error message and found the issue here?

[XaratosS]

@jiridanek I figured out the error. I had to switch versions. 24.4.0 was not working on Docker. I had to go down to 24.3.1. This one worked fine. I remembered it later on. Sorry for all the work I put you through.

[jiridanek]

24.6.0 was not working on Docker. I had to go down to 24.3.1.

@XaratosS these are versions of what, exactly?

[XaratosS]

Oh yeah I was talking about awx. I think I am defenetly in the wrong project...