NVIDIA GPU Operator Constantly Resynchronizing in ArgoCD

[carlmes]

When building a brand new cluster using these settings:

• AWS with OpenShift Envioironment
• OpenShift version 4.17
• Control Plane Instance Type: m6a.4xlarge
• After cluster is provisioned, run bootstrap.sh and choose: 6) rhoai-stable-2.13-aws-gpu

The GPU Operator in the ArgoCD is constantly going out of sync every couple of seconds:

Looking at the sync error, we see that Argo is trying to add these lines:

But the GPU Operator seems to be removing them.

Also, the monitoring console in OpenShift appears to be broken in this release:

It's probably just a new update within the NVIDIA Operator that must be incorporated into our kustomize templates in this project. Source code for the lines that will not synchronize are at:

ai-accelerator/components/operators/gpu-operator-certified/operator/components/console-plugin/consoleplugin.yaml

Line 17 in 39a2cf4

displayName: 'Console Plugin NVIDIA GPU Template'

[carlmes]

Making a note that this failed twice in a row with 2 brand new 4.17 clusters, but appears to not be an issue in earlier versions such as 4.15 and 4.16 during enablement testing.