Recursive env-var replacement not performed for Image=... of the [Container] section

[codedump]

Issue Description

Generally, the systemd unit files have a fairly powerful ability of supporting (more) general configuration by means of env-files or variables. For instance:

[Service]
Type=oneshot
Environment="MESSAGE=jinkies"
ExecStart=sh -c "echo message: $MESSAGE"

Will result in an output like:

...
Aug 15 09:54:31 idefix sh[207592]: message: jinkies

This also works recursively:

[Service]
Type=oneshot
Environment="MESSAGE=jinkies from ${PERSON}"
Environment="PERSON=Velma"
ExecStart=sh -c "echo message: $MESSAGE"

this logs:

...
Aug 15 10:01:04 idefix sh[208571]: message: jinkies from Velma

The first part of this behavior, i.e. 1-st level replacement, extends to host-related parameters of the podman systemd.unit files, e.g. the Image=... key:

[Service]
Environment=IMAGE=alpine:latest

[Container]
Image=${IMAGE}
Exec=echo "Hello"

this will result in:

Aug 15 09:57:37 idefix cert-test[208166]: Hello

But the recursive deplacement fails:

[Service]
Environment=IMAGE=alpine:${IMAGE_TAG}
Environment=IMAGE_TAG=latest

[Container]
Image=${IMAGE}
Exec=echo "Hello"

Result:

Aug 15 10:03:18 idefix cert-test[208802]: Error: parsing reference "alpine:${IMAGE_TAG}": invalid reference format

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a file /etc/containers/systemd/test-recursive-fail.container with the following contents:

[Service]
Environment=IMAGE_TAG=latest
Environment=ALPINE_IMAGE=alpine:${IMAGE_TAG}

[Container]
Image=${ALPINE_IMAGE}
Exec=sh -c "echo Hello"

2. Create a testing file /etc/containers/systemd/test-recursive-work.container like this:

[Service]
Environment=ALPINE_IMAGE=alpine:latest

[Container]
Image=${ALPINE_IMAGE}
Exec=sh -c "echo Hello"

3. Reload systemd to ingest the files systemctl daemon-reload

4. Verify that the ...fail.container file fails, while the ...work.container behaves as expected:

$ systemctl start test-recursive-fail.service
Job for cert-test.service failed because the control process exited with error code.
See "systemctl status test-recursive-fail.service" and "journalctl -xeu test-recursive-fail.service" for details.
$ systemctl start test-recursive-work.service
$

and that the systemd reflects the reason for the failure as:

$ journalctl -xeu test-recursive-fail.service
...
Aug 15 10:05:10 idefix test-recursive-fail[209112]: Error: parsing reference "alpine:${IMAGE_TAG}": invalid reference format
...

Describe the results you received

Describe the results you received

Describe the results you expected

(For the results, check the journalctl output as demonstrated above.)

podman info output

Podman version is 5.1.1 on Fedora CoreOS.

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[eriksjolund]

This also works recursively:

[Service]
Type=oneshot
Environment="MESSAGE=jinkies from ${PERSON}"
Environment="PERSON=Velma"
ExecStart=sh -c "echo message: $MESSAGE"

this logs:

...
Aug 15 10:01:04 idefix sh[208571]: message: jinkies from Velma

I tried the example on Fedora CoreOS 40.20240728.1.1 which has systemd 255.10
but I don't get the same result.

1. sudo useradd test
2. sudo machinectl shell --uid test
3. mkdir -p ~/.config/systemd/user
4. Create the file ~/.config/systemd/user/example.service with the contents

   [Service]
   Type=oneshot
   Environment="MESSAGE=jinkies from ${PERSON}"
   Environment="PERSON=Velma"
   ExecStart=sh -c "echo message: $MESSAGE"
   

5. systemctl --user daemon-reload
6. Run the command

   journalctl -q --user -xeu example.service | grep message:
   

   The following output is printed

   Aug 12 18:06:35 fcos sh[746598]: message: jinkies from ${PERSON}
   

[codedump]

Aug 12 18:06:35 fcos sh[746598]: message: jinkies from ${PERSON}

Yeah, sorry about that. This line:

ExecStart=sh -c "echo message: $MESSAGE"

Should actually read like this (mind the curly braces):

ExecStart=sh -c "echo message: ${MESSAGE}"

The fact that systemd will require the curly {} when substituting a variable for a part of a string is apparently documented behavior.

I took shortcuts while describing the bug and attempted to write a "cleaner" unit file from scratch, and I wasn't careful enough to pay attention to the braces. (I would've thought it wouldn't make a difference anyway, since inside sh -c "..." it's actually a shell... thing... isn't it? Then that's supposed to be a "real" env var with pure sh-ish substitution, right? I'm not sure why the {} matter in this case, but apparently they do. I'm guessing systemd is eager to do the substitution before executing the command; and if it can't, it possibly escapes the $MESSAGE substring. Or so (?!))

In any case, the point here is the recursive replacement of ${PERSON} within ${MESSAGE}.

[eriksjolund]

Should actually read like this (mind the curly braces):

ExecStart=sh -c "echo message: ${MESSAGE}"

Yes that fixed it.
I tried once more but now with curly braces. The journalctl now shows

Aug 16 06:32:03 fcos sh[748413]: message: jinkies from Velma

[rhatdan]

Take a look at the generated systemd unit and see if the error is present there?

[codedump]

Take a look at the generated systemd unit and see if the error is present there?

It's... complicated :-p

Short answer first: expansion into .service happens as it should.

Long answer: take for example this .container:

[Service]
Environment=ALPINE_IMAGE=alpine:${IMAGE_TAG}
Environment=IMAGE_TAG=latest

[Container]
Image=${ALPINE_IMAGE}
Exec=sh -c "echo Hello Kitty"

It gets expanded into this .service, which looks exactly as it should:

...
ExecStart=/usr/bin/podman run ... -d ${ALPINE_IMAGE} sh -c "echo Hello Kitty"      

Trying to better understand systemd's behavior I wrote this .service instead of the original one; the main difference is that we're not sub-shelling the echo anymore, we let systemd invoke it directly:

[Service]
Type=oneshot
Environment=PERSON="Velma"
Environment=MESSAGE="jinkies from ${PERSON}"
ExecStart=echo "message: ${MESSAGE}"

Result:

Aug 16 18:48:09 idefix echo[227478]: message: jinkies from ${PERSON}

Oops. Not good.

Let's try a different one:

ExecStart=sh -c "export"

This gives:

...
Aug 16 18:48:58 idefix sh[227614]: export MESSAGE="jinkies from \${PERSON}"
Aug 16 18:48:58 idefix sh[227614]: export PERSON="Velma"
...

So... yep.

Apparently it never was supposed to work. Systemd never did the substitution. More than that, it actually did escape the $-sign in the message, showing pretty much the opposite of any substitution intention.

That it worked with the sh -c "..." invocation is purely incidental. It actually did work because systemd translated ${...} to \${...}. But this is a shell-thing. (To understand this shell-ism, try calling FOO=hello sh -c "echo $FOO" on a regular terminal command line: it will not work. You need to actually invoke it as FOO=hello sh -c "echo \$FOO". That's because at the time at which the command line is being constructed -- i.e. in the parent shell -- $FOO is not yet known. It only becomes known to the sh subshell as it's being invoked. Therefore \$FOO needs to be escaped in order to not be replaced by an empty string right at the beginnig.)

But what about the "well documented behavior"?

There's a part of the systemd.service manpage that says:

   [...] Use "${FOO}" as part of a word, or as a word of its own, on the
   command line, in which case it will be erased and replaced by the
   exact value of the environment variable [...]
   Use "$FOO" as a separate word on the command line, in
   which case it will be replaced by the value of the environment
   variable split at whitespace, [...]

But that doesn't mean what I thought it means (that's what I get for blindly trusting stackoverflow, hah!). And indeed, the systemd.exec manpage actually says:

    Environment=
        ...
        Variable expansion is not performed inside the strings and
        the "$" character has no special meaning.

My original statement that this was "apparently documented behavior" was inadvertently a blatant lie.

So I'm apparently the podman quadlet behaves as expected after all. It's systemd that does funny things.

I'm closing this as it's not a "bug" of podman (although this kind of behavior would be cool). Feel free to reopen if you think it's worth pursuing, but it feels outside the scope of podman.