Trouble with custom resources within helm charts

[khaled]

I'm using the following helm chart in my config:

  kubernetes.helm.releases.pgoperator = {
    chart = kubenix.lib.helm.fetch {
      repo = "https://opensource.zalando.com/postgres-operator/charts/postgres-operator";
      chart = "postgres-operator";
      version = "1.10.1";
      sha256 = "sha256-cwJuDpXyVjchTqKgy+q4UA4L/U0p37Cn3fJIkHqOlxM=";
    };
    includeCRDs = true;
  };

The chart contains CRDs and also happens to contain a CR whose type is defined by one of those CRDs. When the config is evaluated, I get the following error:

       error: The option `kubernetes.api.resources."acid.zalan.do"' does not exist. Definition values:
       - In `/nix/store/vl3x28s0pa5i8nk07pl8jb751p2dasng-source/modules/helm.nix':
           {
             v1 = {
               OperatorConfiguration = {
                 pgoperator-postgres-operator = {
                   _type = "merge";
           ...

Any way around this? Do I first need to generate Nix module options from the CRDs?

Thanks & BTW kubenix is super awesome :)

[adrian-gierakowski]

You can define a dummy resource with the right name and version to prevent the validation error. You will not get any validation on that resource or special merging rules but it’s better than nothing. I’ll dig out an example tomorrow since I’ve been using this method at work.

[khaled]

I was able to get past the error by defining an option like so:

  options.kubernetes.api.resources."acid.zalan.do".v1.OperatorConfiguration = lib.mkOption {
    type = with lib.types; attrsOf anything;
  };

The problem is that the OperatorConfiguration resource defined within the helm chart doesn't actually get rendered into the build output (though the CRD for it does).

[khaled]

BTW, I'm thinking about adding an additional package that can generate options given a helm chart. So far I've got some logic that'll take a chart and produce from it a json schema (a swagger structure with just the definitions). From here it looks like I can leverage most of the existing generate-k8s logic.

Thoughts?

[hall]

Hey @khaled, thanks for raising this issue! I've been doing the same as Adrian and defining dummy resources. A proper fix to parse the CRD then validate the CR would be nice but I've spent no time at all thinking through that one so it might not even be possible.

Regarding validating the helm chart values, I think that'd be great! Doubly so if you can spit out a decent error message. I don't have much thought around implementation at this point though so I'm happy to review whatever approach you have (and others/myself can always iterate on it down the line as needed).

[khaled]

The root problem was #35, so I'm going to close this one out. I opened a separate issue to to track generating types from CRDs.