Handling Secrets with GitOps

[chris-short]

Handling secrets with GitOps is becoming a frequent question and starting to become a problem. Some secrets solutions like sops and external-secrets are in various states of decay.

Luckily, this is in alpha and I feel like this will be our future answer: https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html

Despite being alpha, the Big 3 cloud providers (AWS, Azure, GCP and bonus, Hashicorp Vault) all have providers.

[niklasmtj]

Super excited about the CSI driver. Even when I'm pretty new to the whole k8s world this looks like a great solutions. Used it a couple weeks back in combination with AWS Parameter Store to mount these as ENVs and syncing them as Kubernetes Secrets. Not in production yet but getting our feet wet with it already. The integration worked pretty flawlessly (after reading the docs a little more closely 😅 ).
IIRC one thing that was still a little fiddly was the updating of the ENVs in the Pods after updating the values in Parameter Store. But hey some of the parts are still in alpha. Really looking forward to see how this project is coming along!

[PG2000]

just curious about why you have used the AWs Parameter Store instead of the secrets manager?
Do you had some requirements for that or did you just want to give it a try?

[niklasmtj]

We just wanted to give it a try for a quick prototype deployment

[chris-short]

I too am seeing this turn towards the CSI Secrets Driver by the community. It's the beginning of the turn but, it's definitely heading that way it seems.

[mateuszpruchniak]

When it comes to keeping secrets in GitOps, I see two approaches:

• fully compliant with GitOps principles especially "Versioned and Immutable" so keeping securely in Git
• "hybrid" use desired state of the system expressed declaratively and use an external vault for secrets, in the case of Azure we have Azure Key Vault Provider for Secrets Store CSI Driver which allows for the integration of an Azure key vault as a secrets store with an Azure Kubernetes Service (AKS) cluster via a CSI volume - but we lose immutability which we have in the first option and we need to implement an additional process of provisioning of secrets to Azure Key Vault for every environment.

When you talk about secrets, do you mean passwords, API keys, ..?
Encryption keys are a separate topic in my opinion.

[strantalis]

Typically from what I have seen it's mostly passwords, api keys, ssh keys or even something like kerberos keytabs.

[chris-short]

I see passwords stored in git as a bad thing. I know you can encrypt the file but, it's like having an encrypted database compromised but, you're doing it on purpose. It's just a matter of time until something/someone/some government/some competitor/etc. figures out how to decrypt your publicly exposed encrypted database. I don't know about you but whenever I hear my password is compromised on some site/service, even in an encrypted database where the keys weren't compromised, I still rotate my password.

Yes, this means something isn't in git, I acknowledge that, but it's literally the keys to the kingdom, sitting there, publicly available to any passer by and that fact shouldn't be diminished.

[mateuszpruchniak]

@chris-short I agree.
I think the best way is to keep secrets and keys in external vaults (Azure Key Vault, HashiCorp Vault, ... ) and access to them via a CSI volume. In the case of Financial Institutions, we have a requirement that we have to keep encryption keys in vaults compliant with FIPS 140-2 Level 2.
But application configuration can be store in Git to achieve versioning and immutability of environment.

[strantalis]

This is just my opinion but I prefer not to keep any secret data in git as I feel like its potentially prone to exposure. Especially if local git hooks aren't configured properly. I guess an example of this is making sure a file is ansible vaulted before commiting (Something I have done 😞 ).

We currently just upgraded to using the external-secrets operator and have been happy with it so far. It's nice how you can now scope secret stores at the cluster level and the namespace level.

Something else we have started looking at is the ArgoCD Vault Plugin. Which has more backends now than just vault. This tool is nice if you have custom resources that do not yet have the ability to reference a secret.

[moritzjohner-form3]

External Secrets Operator maintainer here, just saw this was mentioned in Kubecon/GitOps track this morning. Feel free to connect, you can find me @moolen and @gusfcarvalho in KubeCon running around, we do a external secrets operator hour at Container Solutions booth (booth S106) during coffee breaks 4-5pm.

[...] external-secrets are in various states of decay

@chris-short It has been rewritten in go, is well tested and just recently beta was announced 🌞

[knelasevero]

I think that KES (Kubernetes External Secrets - old GoDaddy project donated for reference) presence in the org is now causing some confusion, since it is deprecated. However, it is deprecated in favour of the new project that @moolen mentioned above, growing fast for quite a while now.

More context around the deprecation and all that: external-secrets/kubernetes-external-secrets#864

A few numbers:

ESO website is having around 2500 weekly users. Recently reached 1.1k stars. 105M image pulls. It is present on artifacthub, operatorhub and other channels. It is actively being used by big fin tech enterprises in prod with hundreds of secrets. 665 members on k8s external-secrets slack. From the big Secret Providers, all are stable (gcp, aws, vault) except from azure that is on beta (pending on some infra and specific auth e2e tests there, and more usage).

Digital Ocean had a very interesting rundown of both projects recently (CSI Driver and ESO): https://www.youtube.com/watch?v=EW25WpErCmA

Which basically shows that both projects have their place in different situations. SSCSID has more of a place when you need to explicitly avoid k8s native secrets. I would say ESO in any other situation. (Some users use both simultaneously..)

Edit: just to be clear, we are talking about https://github.com/external-secrets/external-secrets and not KES.

[knelasevero]

Hey all (@scottrigby, @todaywasawesome, @murillodigital, @cdavisafc, @chris-short, @stefanprodan, @christianh814),

The blog that is being written to guide users in this space around gitops secret management just came to our attention: https://docs.google.com/document/d/1hLou9-jIeK0oLIkg8lSeskMqfm7qAMzayZ9b5AlxH6Y/edit

We wanted to check if we (@moolen, @gusfcarvalho, and me) could participate and make some suggestions there. Would not hurt to involve Secret Store CSI Driver folks here as well (@aramase, @ritazh) and have a good representation of both projects. I am an ESO maintainer, but I am also a SSCSID user, and I think that, even though these projects have huge overlap, they deliver most value when we understand their differences in detail, and when we are aware that we for sure won't have a silver bullet for gitops secret management. I think that best practices recommendations would be awesome with that in mind.

What do you think?

[ritazh]

Thanks for including us! Will review and provide feedback in the doc.

[aramase]

cc @tam7t

[chris-short]

I wrote this in DevOps'ish last week:

I'll preface this by saying this is my opinion. It's is not the opinion of the CNCF GitOps Working Group or OpenGitOps. This is solely my opinion. Universally, it's a bad idea to check secrets into git. Whether they're encrypted or not that shouldn't really matter. They're still secrets and, in my opinion, encrypted or not, secrets shouldn't live in git. A shared password safe is better than git. Even better an external secret store so you could utilize the Kubernetes Secrets Store CSI Driver. To me, it comes down to a few important things.

1. Unix philosophy: I think secrets management tooling is probably better at managing secrets than a distributed version control system. Just saying.
2. You need active secrets management in production: That means regular rekeying, monitoring of existing keys, and the auditing and logging such a service provides. This could run in cluster but, chances are you're using a Secrets Store CSI Driver compatible service somewhere in your enterprise.
3. Quantum computing (or time): eventually, that blob of encrypted text is going to fall flat on its face against quantum computing or a skilled adversary. Adversarial governments already posses the power to decrypt many things if they dedicate the resources to it. There's no reason to expose what is essentially a very important database to the general public because, like with all things, given enough time and energy, whatever protection you did have, will fall.
4. I'll say it again, there's no reason to expose what is essentially a very important database to the general public.

GitOps says everyone in the git pool. But, just like some folks still think the best way to run a database is on bare metal, with no abstractions on compute (latency matters). In my opinion, the best way to store your secrets is some place off cluster in a secret management service or on cluster in a secret manager that's highly available. Either way, it should never be publicly accessible.

[scottrigby]

We created this issue during today's meeting:

• Add content: "handling Secrets with GitOps" website#88

This is just to get the ball rolling moving this discussion into an actual work product. For now, moved to website repo, in case we want for example a blog post. We can either move that create another issue from here to the documents repo if we want a lasting document there.

This will allow us to collaborate on the idea. We can also of course continue discussing here, but let's try to make something from this 😄