Docker image update policy

[iamFIREcracker]

Hey,

My team is using a fairly old version of Neo4j (4.2.8 to be precise) and all of us got a bit surprised when we realized that the image that we had been using for almost a year, had recently been updated:

Last pushed 13 days ago by doijanky

Even more surprising was that the newer version (or one that got "updated" sometime in between last summer and 13 days ago) was now shipping Debian 11, which unfortunately requires the host environment to run a more recent version of docker (see: #318). Note: the 4.2.8 version that we have, cached inside our CI/CD environment, runs Debian 10, and we never run into any permissions issue with it.

Now, we are working things out internally to try and switch to a newer version of docker on the host, and to a newer version of neo4j; however, I also wanted to understand more of how are the Neo4j docker images getting updated. From where I stand it seems as if you were treating all the published tags as rolling versions, i.e. like latest, where pulling it today or in one month might give different results; however, I question if this is what the different engineering teams are actually expecting, especially given that quite a big change (like the upgrade of the underlying OS) could be snuck into a tag rewrite and potentially set your whole CI/CD pipeline on fire.

I know we can refer to images by their digest, and that's what we will ultimately end up doing if we realized we could not trust a specific version, like 4.2.8, to stay immutable; however, if version 4.2.8 today is different from version 4.2.8 from one week ago, how could I then (1) be notified about the update, and (2) audit the new version to confirm it's compatible with our deployment environment?

Thanks,
M.

[jennyowen]

Neo4j is a supported, official docker image. As such, the people at docker automatically rebuild our images when there are changes and updates to our base images.
At some point in the last year, debian updated their default debain slim to be bullseye rather than buster. This prompted an update of our base image openjdk:11-jdk-slim, which in turn caused Neo4j images to be rebuilt.

We maintain a list of supported Neo4j versions and their image tags for the people at docker:
https://github.com/docker-library/official-images/blob/master/library/neo4j
The listed images are rebuilt and updated whenever the base images are changed. This allows the Neo4j to get security updates as soon as they are available.

The parts of this update process we control are:

• Which Neo4j versions are in the supported list
• The contents of the Neo4j image that is NOT the openjdk base.

Updating the Neo4j image contents

All the Neo4j image source code can be found here:
https://github.com/neo4j/docker-neo4j-publish

It is extremely rare that we change the contents of an image after publishing, and even then it will only be a change to the docker entrypoint scripts around Neo4j and never any part of the database itself.
When the docker image is built, it will download Neo4j from our server and this means the Neo4j docker image will always be running the same source code as a regular Neo4j installation.

Which Neo4j versions should get updates

Currently all supported major, minor and patch versions of Neo4j will automatically receive security updates from docker.
The list of supported Neo4j versions and the support road map can be found at:
https://neo4j.com/developer/kb/neo4j-supported-versions/

When Neo4j 5.0.0 is released, we are only going to list the latest patch version of a major.minor version as supported in docker. This will apply retroactively to all existing Neo4j releases.
For example, of all the 4.2.X releases, only the latest patch will be marked to receive security updates in docker.

Neo4j versions not listed at https://github.com/docker-library/official-images/blob/master/library/neo4j are still available for use, they just do not get the updates to the base image.

[jennyowen]

For your question about getting notifications about when an image is updated, I don't know. Those updates are done by docker, not Neo4j, you would have to ask docker support about how to monitor a particular tag for changes.

[jennyowen]

From where I stand it seems as if you were treating all the published tags as rolling versions, i.e. like latest, where pulling it today or in one month might give different results;

Yes I suppose that is true. Although, I would argue that this is true for all docker tags. Docker tags are mutable by design. Unless you manage the image yourself (or specify which image you want by digest) you do risk unexpected changes.

For neo4j:4.2.8, the promise is that it will ALWAYS have only neo4j version 4.2.8, there will never be forced updates of Neo4j, but the various undocumented parts of the image that support neo4j are indeed subject to change.

[iamFIREcracker]

We maintain a list of supported Neo4j versions and their image tags for the people at docker:
https://github.com/docker-library/official-images/blob/master/library/neo4j
The listed images are rebuilt and updated whenever the base images are changed. This allows the Neo4j to get security updates as soon as they are available.

Thanks for the pointer -- docker-library/official-images#1864, in particular, in which I could read the rationale behind having all the point versions to be rebuilt.

For your question about getting notifications about when an image is updated, I don't know. Those updates are done by docker, not Neo4j, you would have to ask docker support about how to monitor a particular tag for changes.

I am not sure what to think about this, honestly. I, as a end user, should not have to worry about who published the image into the docker registry: was it the engineering team? was it a bot/automatic process? Don't care. If docker.hub is an official distribution channel for neo4j, then you are ultimately responsible for everything happening on it (in a way it becomes part of your product), and if already published versions are expected to be rebuilt, even after months from their initial publication, I somehow think this should be clearly stated somewhere.

From where I stand it seems as if you were treating all the published tags as rolling versions, i.e. like latest, where pulling it today or in one month might give different results;

Yes I suppose that is true. Although, I would argue that this is true for all docker tags. Docker tags are mutable by design. Unless you manage the image yourself (or specify which image you want by digest) you do risk unexpected changes.

If you think about it, it's all about expectations and what people can do vs should do:

• If you do semantic versioning, you will not expect a breaking change in a patch release
  • Still, nothing prevents publishers from pushing a new patch version of a npm module and introduce a breaking change with that -- potentially breaking all the downstream systems
• When working with scm systems like git, It's a common practice not to rewrite the commit history
  • Still you are free do it, if really want / have to, but that will most likely catch people off-guard
• The fact that docker tags are mutable does not imply that all the docker tags have to be mutable
  • It's fine for tags like 4 or 4.4 to be considered mutable; I am not so sure about 4.4.12

Also, talking about digests: are the digests for any image that gets re-built, kept around, forever? Or will those disappear the moment the tag is re-built?

For neo4j:4.2.8, the promise is that it will ALWAYS have only neo4j version 4.2.8, there will never be forced updates of Neo4j, but the various undocumented parts of the image that support neo4j are indeed subject to change.

This works well in theory; in practice, these undocumented parts are part of the image anyway, and users will inevitably rely on them, directly or indirectly (e.g. without realizing it, we were relying on the fact that Neo4j was running on Debian 10, and when it switched to 11, it all stopped to work).

So, to recap:

• All the neo4j docker tags are to be considered mutable
  • This is so that security patches from upstream (i.e. base image) will automatically land to neo4j users
  • Changes at the service level (i.e. neo4j) are not expected
  • Changes at the OS level on the other, are to be expected
• How can one implement reproducible deployment
  • You could link to a specific digest
    • Though we have to confirm if digest will be kept around after the same image version gets re-built
  • You cache images internally
    • It requires a private registry
      • That's what we are doing this already

Once we agree on / confirm the above, I think there won't be anything else left discussing.

I will be honest: I would rather have neo4j (and other services) not to to automatically re-build old point versions, but it's just my personal opinion of course, and I don't expect you do anything with it. From where I stand, at least now I know what to expect, and can act accordingly.

Thanks,
M.