conmon 2.1.9 not working with podman 4.8.2 and ubuntu 22.04

[dmdx86]

root@nas:~/conmon-2.1.8# cd ../conmon-2.1.9/
root@nas:~/conmon-2.1.9# make install
<snip>
root@nas:~/conmon-2.1.9# podman run --rm -it ubi9-init /bin/bash
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Same error on any other container, new or existing.

It is OK on 2.1.8

root@nas:~/conmon-2.1.9# cd ../conmon-2.1.8/
root@nas:~/conmon-2.1.8# make install
<snip>
root@nas:~/conmon-2.1.8# podman run --rm -it ubi9-init /bin/bash
[root@e9ded0c5b94a /]# 
exit
root@nas:~/conmon-2.1.8# 

I compile and build podman/conmon/crun/buildah, do not use the versions in the Ubuntu repositories. Everything was good up until I updated conmon this morning.

This is trivial for me to reproduce, LMK if anything else is needed from me or if I am doing something dumb to cause this.

root@nas:~/conmon-2.1.8# podman version
Client:       Podman Engine
Version:      4.8.2
API Version:  4.8.2
Go Version:   go1.20.3
Built:        Thu Dec 14 16:53:00 2023
OS/Arch:      linux/amd64

[flyingfishflash]

I logged to say the same. I'm on archlinux, but all of my containers now fail to start with the same exact error.

Reverting back to 2.1.8 with podman 4.8.2 works as expected.

Client:       Podman Engine
Version:      4.8.2
API Version:  4.8.2
Go Version:   go1.21.5
Git Commit:   aa546902fa1a927b3d770528565627d1395b19f3-dirty
Built:        Wed Dec 13 17:07:26 2023
OS/Arch:      linux/amd64

[CameronNemo]

Indeed it appears the release is broken. This is the results I get:

$ podman run --rm -it docker.io/alpine:edge sh
Error: runc: runc create failed: chdir c�Z: no such file or directory: OCI runtime attempted to invoke a command that was not found

And then in the working directory I see ctl and winsz sockets...

Everything works fine with conmon 2.1.8.

[martinetd]

hit the same problem when trying to check my patch just now...
The problem was introduced in a recent static analysis fix ; the free is incorrect if the value comes straight from options.

I've fixed it in #476

[PeterUpfold]

I can also reproduce this issue on Arch.

$ podman run -t ubuntu echo "Hello"

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Downgrading to 2.1.8 causes the issue to go away.

[urbenlegend]

Can confirm that applying #476 on top of Arch's 2.1.9 package seems to have fixed the issue. My containers now all properly start up again.

[urbenlegend]

Indeed it appears the release is broken. This is the results I get:

$ podman run --rm -it docker.io/alpine:edge sh
Error: runc: runc create failed: chdir c�Z: no such file or directory: OCI runtime attempted to invoke a command that was not found

And then in the working directory I see ctl and winsz sockets...

Everything works fine with conmon 2.1.8.

Today I noticed ctl and winsz in my root directory. Do these belong to podman? I was a bit freaked out when I saw these files.

[giuseppe]

Today I noticed ctl and winsz in my root directory. Do these belong to podman? I was a bit freaked out when I saw these files.

conmon creates them

[CameronNemo]

@giuseppe y'all might want to tag a bugfix release or at least update the 2.1.9 release notes (if that is that possible) to mention that it is broken. Seems that less-than-careful downstreams like NixOS are not aware of this bug.

[haircommander]

should be fixed in 2.1.10 now

[d03j]

conmon creates them

Is this new? I also noticed them today. I would swear they did not use to be there before.

[martinetd]

They're not supposed to be in your root directory (just some run dir that is used by podman as per the path in options); the bug just made them appear in whatever current directory was used at the time because the path was incorrectly freed and reused for something else.

You can safely delete them.

[jwhitaker-a365]

hey @lsm5 , any chance of getting 2.1.10 into obs kubic? :)

[AntonioSun]

Problem still exist in Mac? Can someone double-check please?

As I just installed yesterday and it is still failing for me on mac:

$ podman run -t ubuntu echo "Hello"
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob sha256:005e2837585d0b391170fd9faf2e0c279d64ba0eb011cda8dedf28cb5839861e
Copying config sha256:da935f0649133cbea2f5ad83db14bf782aa5ee9ad17cd609253e3750201a9298
Writing manifest to image destination
Error: preparing container 6dc501b868f5994fbde0decaa223c0275d7cb67448310bff3c3352dbd576d6f3 for attach: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

$ brew info podman
==> podman: stable 4.8.2 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.8.2 (191 files, 56.7MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-12-31 at 15:17:20
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/podman.rb
License: Apache-2.0 and GPL-3.0-or-later

$ uname -srm
Darwin 23.1.0 arm64

$ sw_vers
ProductName:		macOS
ProductVersion:		14.1.2
BuildVersion:		23B92