bulk-patch CLI for local files/git repositories

[schrej]

One of the missing pieces of tooling in the GitOps space is a good way of applying changes to local git repositories. When managing large clusters, or many smaller ones, it can be difficult to apply changes in a controlled way, especially when performing bulk-edits on multiple resources across different clusters.

Performing such operations on a cluster (or multiple) directly, e.g. from a CI pipeline, is much easier. Actions are performed against the API, ensuring that they are syntactically correct. kubectl even comes with several commands like patch and set to help with such tasks.

Changes in a git repository that should then get synced to a cluster are more tricky. They need to be syntactically correct so that they can get applied. Of course there are tools that can help to validate resources and even dry-run them against a cluster, which can be embedded into a CI pipeline. When doing small, manual changes, this is perfectly sufficient. If a mistake was made, it is easy to fix in a subsequent or amended commit.

But for automated changes, nobody wants to fix problems in a merge request because indentation is off by one space due to different formatting of files, all of which were modified using some arcane sed command in a script that was duplicated 30 times for different tasks.
Changes need to be valid when they are committed by automation, the (optional) Pull Request should just be there for approval.

I am looking for a tool that helps with performing such changes. Since I haven't found any* so far, here is what I've come up with:

• A system similar to Kustomzie patches, because a lot of people are familiar with them.
• Instead of combining base resources with these patches, and directly applying them to a cluster, patches are applied to the base resources in-place. This way they can be committed to git, which is what we want.

Additional features to make it really useful:

• Safeguards that validate the current state of a file and help to prevent accidental changes. One simple example would be to avoid downgrading a semver by accident.
• Patches that can take input parameters to make them re-usable, e.g. as part for runbooks, or just to use them in a CI pipeline.
• Bonus: All of these features are available as a CLI, as well as a decent Go library that allows to wrap it with custom tooling.

• I have found fluxcd/image-automation-controller of course, but that only has a very limited subset of that functionality.

---

For context: we're managing a fleet of clusters through cluster-api, and provide some standard components on top. All is controlled through one large repository, containing separate configuration for all clusters.

With Kustomzie we could also use composition to control versions on a global level. But we do not want to do that for several reasons:

• we need to be able to control our clusters individually in order to remain flexible when solving production issues for individual customers
• we need to gradually roll out changes to reduce the blast radius of faulty configuration
• Kustomzie makes it more difficult to track whats going on, estimate the impact of changes and where specific configuration values are coming from. At small scale this is manageable, but with 100+ clusters it becomes increasingly difficult.