Add helm charts for training operator

[ChenYi015]

What this PR does / why we need it:

Add helm charts for training operator.

Which issue(s) this PR fixes :
Fixes #1197

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

/hold for review

[coveralls]

Pull Request Test Coverage Report for Build 10958395047

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 10937611143:	0.0%
Covered Lines:	66
Relevant Lines:	66

---

💛 - Coveralls

[andreyvelich]

This is great, thank you for this @ChenYi015!
We have been discussing before that we might want to add helm charts only for Training V2. @kubeflow/wg-training-leads any thoughts ?

Additionally, how we can make sure that kustomize manifests and helm charts are maintained and in sync ?

[tenzen-y]

This is great, thank you for this @ChenYi015! We have been discussing before that we might want to add helm charts only for Training V2. @kubeflow/wg-training-leads any thoughts ?

Additionally, how we can make sure that kustomize manifests and helm charts are maintained and in sync ?

We need to pay massive maintaining costs to keep both HelmChart and kustomize. Hense, I do not think that this is maintainable. Based on my OSS experience, we often implement complex and unmaintainable conversion script. But, in this repository, I think that is not available based on community size.

[ChenYi015]

Agree. It is not a good idea to maintain a script or a tool to do sync between helm charts and kustomize.

[andreyvelich]

I think, we should discuss it in one of our upcoming Training WG calls.
I can see that today majority of CNCF projects have Helm charts and many organizations using it as a primary tool for their deployment.
Although I agree that Kustomize is much more lightweight and easier to use, if we want to see more adoption on Training V2, we might want to have Helm charts.

Maybe we can take a look at some examples on how users combine Helm and Kustomize together, for example: https://trstringer.com/helm-kustomize/.

[tenzen-y]

I think, we should discuss it in one of our upcoming Training WG calls. I can see that today majority of CNCF projects have Helm charts and many organizations using it as a primary tool for their deployment. Although I agree that Kustomize is much more lightweight and easier to use, if we want to see more adoption on Training V2, we might want to have Helm charts.

Maybe we can take a look at some examples on how users combine Helm and Kustomize together, for example: https://trstringer.com/helm-kustomize/.

I actually maintain kustomize and Helm Chart in the Kueue repository. But the synchronization way was only a conversion shell script.

[terrytangyuan]

+1 to huge maintenance overhead from my experience maintaining Argo's helm charts for a vendor.

[terrytangyuan]

Although I agree that Kustomize is much more lightweight and easier to use, if we want to see more adoption on Training V2, we might want to have Helm charts.

One thing we can do is to have automation for maintenance, reduce parameterization, and have the Chart templates stay as close to the raw manifests as possible. Users can do their last-mile modification via kustomize's built-in helm chart support