feat(Chart)!: 📦 add separated chart for CRDs

[darkweaver87]

What does this PR do?

This PR introduces a separate chart used by the main one to install CRDs.
The dependency between the main chart and the CRDs one will be added once CRDs one will be released.

Motivation

#1141
#1209

More

• Yes, I updated the tests accordingly
• Yes, I updated the schema accordingly
• Yes, I ran make test and all the tests passed

[mloiseleur]

1. The CI needs to be tested, with a tag on your fork.
2. The documented CRDs upgrade process should still work. I see (at least) two options:
   a. Keep and copy crd files
   b. Update CRD upgrade instruction to use them in the new directory

[darkweaver87]

Tested on my own account as seen with you 👍

[mloiseleur]

Add the CRDs as templates within the chart like in the cert-manager chart

This morning, I have explored this option. I have encountered two major issues:

1. On install: It fails to deploy dashboard IngressRoute (CRDs are not loaded at this stage)
2. On upgrade: It fails to deploy CRD: invalid ownership on metadata and annotations

I haven't found a way to switch CRDs from current Helm CRD management to Helm Template management without failure. Instructions can be provided to patch resources beforehand, and thus get around this failure. Maybe it's possible to deploy IngressRoute in post-install hook, but should we really do this ?

We are really fighting the tide, here.

This use case of selectively picking CRDs is specific. It's clearly not the common use case.

I think we are left with two options on this:

1. Option Standalone CRD Chart:
   • The common use case works is not modified (current Chart stays as it is today)
   • Users has to opt-in to use this new Chart
   • There is no link between them.
   • Like karpenter, we provide two Charts, the current one with CRDs and a new one with only CRDs.
2. Option Documentation:
   • Instructions in the README on how to install CRDs separately
   • Users with specifics needs on CRDs deploy them separately and can install the Chart without CRDs after

@jnoordsij @darkweaver87 : Wdyt ?

[mloiseleur]

Additional notes:

• Helm maintainers have started to work on Helm v4, see here.
• CRDs management are discussed in the roadmap.
  Helm maintainers have also detailed why it's working this way in HIP-0011.