Image is not deleted from the catalog in onprem

[YevheniiSemendiak]

When you delete the last tag of image, the image itself should not be displayed in neuro image ls.
This holds only for cluster deployments, where do not use managed image registry.

@anayden you told we have enabled garbage collector, but I didn't found where is it.
Could you elaborate, please?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

[YevheniiSemendiak]

Trying to reproduce, pushed image:test-garbage-collector:v1 image to onprem-poc cluster and removed it, awaiting the GC to process (does it once in a day).

[YevheniiSemendiak]

The issue is still relevant:

@zubenkoivan, could you please check the garbage collector?

[zubenkoivan]

Hey @YevheniiSemendiak . This is not related to garbage collection. This feature is not supported by private docker registry instances. Please see the issue below. There is a PR that might solve it once merged.

Issue:
distribution/distribution#2314

PR:
distribution/distribution#3195

[anayden]

Cronjob was setup by Alin.

BUT, looking at our chat, I see neither of us verified the files are actually deleted. So it might be, the job is running, but is not doing anyting.

[YevheniiSemendiak]

From the issues and PR Ivan mentioned, its seen that the blobs are actually deleted by the garbage-collect command, but the repositories without manifests (distribution/distribution#3195). So, we are blocked by the uplink.

[bigtfromaz]

I am prepping to develop a basic open-source package primarily to list and remove tags and images when a repository admin no longer wants them available to the repository's consumers. I ran into this issue while prototyping my intended HTTP conversations in Postman.

In my findings we see that after an image is deleted from the repository using the Docker-Content-Digest, the repository is still listed in _catalog. The problem is the HTTP DELETE conversation corrupted the registry datastore.

When a repository is in this state, and some user or script issues a docker pull imagename command using the default tag, or any tag, they receive this message:

   `Error response from daemon: manifest for myhost.net/hello-world-old:latest not found: 
     manifest unknown: manifest unknown` 

This message indicates a corrupt registry datastore because the response should be NOT FOUND.

When a utility like this API comes in from the side and manipulates the database, it should never corrupt the database. In my opinion, remediating this should be a high priority.