[LOCAL] Sync issue with Prometheus stack

[blucas]

Issue Type

Bug

Description

When running make ci I came across an issue in deployment (or re-deployment) of the prometheus stack. The issue is that the stack cannot be synced by argocd (see error message below):

➜ argocd app get argocd/kube-prometheus-stack 
Name:               argocd/kube-prometheus-stack
Project:            default
Server:             https://kubernetes.default.svc/
Namespace:          prometheus
URL:                http://localhost:57996/applications/kube-prometheus-stack
Repo:               http://git-http-backend/git/k8s-kurated-addons
Target:             HEAD
Path:               addons/kube-prometheus-stack/
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        OutOfSync from HEAD (f344f91)
Health Status:      Missing

CONDITION                                           MESSAGE  LAST TRANSITION
SyncError                                           Failed sync attempt to f344f918e340d4e6b7629a0b0af13ab3d27aa535: one or more synchronization tasks completed unsuccessfully, reason: CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes,resource mapping not found for name: "kube-prometheus-stack-kube-prometheus" namespace: "prometheus" from "/dev/shm/2512263151": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first (retried 5 times).  2023-03-07 08:17:05 -0500 EST

Apparently this is an issue with just about any resource, but there is a prometheus helmchart issue related to this in the prometheus github repo: