pull --all-tags : confusing documentation

[edsantiago]

podman pull --all-tags CI test is flaking today due to the quay outage.

Question: is there any way to force --all-tags to respect the aliases in registries.conf?

Issue: docs need improvement because "docker.io" is misleading and wrong and confusing:

IMPORTANT: When using the all-tags flag, Podman does not iterate over the search registries in the containers-registries.conf(5) but always uses docker.io for unqualified image names.

[Luap99]

For reference this is the used registries.conf: https://github.com/containers/podman/blob/main/test/registries-cached.conf

containers-registries.conf(5) says:

Redirection and mirrors are currently processed only when reading a single image, not when pushing to a registry nor when doing any other kind of lookup/search on a on a registry. This may change in the future.

Given --all-tags must lookup all tags first then pull image the tag list still reads from quay.io but I don't know why it is that way. It certainly breaks the usage of this in restricted environments.

cc @mtrmac

Also trying locally with debug logs shows this very clearly

DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?scope=repository%3Alibpod%2Ftestdigest_v2s2%3Apull&service=quay.io 
DEBU[0001] Increasing token expiration to: 60 seconds   
DEBU[0001] GET https://quay.io/v2/libpod/testdigest_v2s2/tags/list 
DEBU[0001] Looking up image "quay.io/libpod/testdigest_v2s2:20200210" in local containers storage 
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ... 
DEBU[0001] reference "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210" does not resolve to an image ID 
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ... 
DEBU[0001] reference "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210" does not resolve to an image ID 
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ... 
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0001] Attempting to pull candidate quay.io/libpod/testdigest_v2s2:20200210 for quay.io/libpod/testdigest_v2s2:20200210 
DEBU[0001] parsed reference into "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210" 
Trying to pull quay.io/libpod/testdigest_v2s2:20200210...
DEBU[0001] Copying source image //quay.io/libpod/testdigest_v2s2:20200210 to destination image [overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210 
DEBU[0001] Using registries.d directory /etc/containers/registries.d 
DEBU[0001] Trying to access "127.0.0.1:60333/libpod/testdigest_v2s2:20200210" 

[mtrmac]

Yes, mirrors are not used for listing tags.

It’s … consistent? The mirror code is designed to work well with partial mirrors, where important images are mirrored and pulling the others falls back to the primary. But in such an environment, listing tags from the mirror returns unexpected results.

Really, this is more of an implementation accident (and delivering a minimal viable mirroring implementation) than a well-considered careful decision.

We could add the feature, and an option to registries.conf to opt into listing tags from mirrors. But then again, I don’t know who should ever use podman pull --all-tags. Other users of “list tags” are likely to be mirroring tools and the like who care about contacting the primary vs. a mirror, and know the details of the setup; such users can trigger a tag listing of the mirror explicitly, if that’s what they want.

---

For the test, would it be practical to run pul --all-tags directly from the mirror location? It does affect what is being tested, I don’t know whether that’s acceptable.

[edsantiago]

For the test, would it be practical to run pull --all-tags directly from the mirror location?

Tentative yes, but there seems to be a gap in the passing of --tls-verify:

$ bin/podman pull --tls-verify=false --all-tags 127.0.0.1:60333/libpod/testdigest_v2s2
Error: pinging container registry 127.0.0.1:60333: Get "https://127.0.0.1:60333/v2/": http: server gave HTTP response to HTTPS client
$ ^-a
...works fine...

(Side note: for reasons I can't explain, my brain was convinced that search would not work on a local registry. It does! I've made myself a TODO subitem on my protect-us-from-quay-outages list)

[Luap99]

For the test, would it be practical to run pul --all-tags directly from the mirror location? It does affect what is being tested, I don’t know whether that’s acceptable.

The challenge here is that we must keep the tests working with and without the local registry setup so we add some complexity into the test to allow that. That said given quay.io flakes are bad enough it seems certainly worth it to do.

[edsantiago]

We already have a mechanism for conditionalizing e2e tests such that they work in and out of CI:

podman/test/e2e/pull_test.go

Lines 25 to 32 in 734c4b9

	// As of 2024-06 all Cirrus tests run using a local registry where
	// we get 404. When running in dev environment, though, we still
	// test against real registry, which returns 401
	expect := "quay.io/libpod/ibetthisdoesnotexistfr: unauthorized: access to the requested resource is not authorized"
	if UsingCacheRegistry() {
	expect = "127.0.0.1:60333/libpod/ibetthisdoesnotexistfr: manifest unknown"
	}
	Expect(session).Should(ExitWithError(125, "initializing source docker://ibetthisdoesnotexistfr:random: reading manifest random in "+expect))

I hate it, really hate the hardcoding and special-casing, hate the fact that some code paths go untested. I choose to accept this grossness for now as a pragmatic tradeoff. I hope some day we can reevaluate.

[mtrmac]

For the test, would it be practical to run pull --all-tags directly from the mirror location?

Tentative yes, but there seems to be a gap in the passing of --tls-verify:

Huh. From a quick read of the code (without testing) this seems to be wired correctly, I don’t immediately see where the information is lost.

---

(Side note: for reasons I can't explain, my brain was convinced that search would not work on a local registry. It does! )

There are two APIs: /v1/search, which does a server-side search (possible at scale) and is not implemented by docker/distribution at all; and /v2/_catalog, which lists everything and searches client-side (unreasonable for large-scale hosted registries) and is implemented by docker/distribution.

The two are basically distinct code paths, testing one does not test the other. (And it’s up to the registry being contacted which one is used; we prefer /v1/search.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago what do you want to do with this one?

[edsantiago]

This is beyond my ability to fix, unless we decide to go with a documentation-only note. @Luap99 @mtrmac

[rhatdan]

I would be fine with just documenting this.

[mtrmac]

If --tls-verify is not working, we should definitely fix that part.