Central Vali ManagedResource cannot be skipped.

[vlvasilev]

What happened:
Since this PR the gardenlet does not install the Vali.
Instead, install a ManagedResource and leave the job to Gardener Resource Manager(GRM)
.
When custom storage is set via gardenlet configuration removes the resources.gardener.cloud/ignore annotation from the managed resource no matter if it has to resize or not the central PVC.

Why this is happening?
In order to resize the PVC one has to scale down first the Vali Statefulset to zero.
But because now we are using ManagedResource the StatefulSet control is done by the GRM.
So if we scale the sts from the gardenlet to zero then the GRM will scale it immediately to 1.
In order to work around this we annotate the ManagedResource(mr) with resources.gardener.cloud/ignore=true, thus the GRM is skipping the Vali mr from a reconciliation.
In the kind local setup the resizing of the PVC is not available.
This is an edge case on a seed where the Vali PVC's StorageClass does not allow disk resizing.
In such cases, the custom disk configuration from the gardenlet config should be removed.
But after that, there is no logic to remove the resources.gardener.cloud/ignore, and Vali is never again reconciled.
In order to remove the annotation, in this case, Rafael decided to remove the annotation even when the new and the old PV size are the same.

What is the side effect?
So when storage is specified when a Vali Deployer is created the underlying mr will be unannotated.

What is the impact of the side effect?
At the time this issue is opened it is impossible to debug the central Vali by skipping it ManagedResource.
Any changes to Vali will be immediately overwritten.
Fortunately, the shoot Valis is not affected because the creation of the shoot Vali Deployer does not include passing custom storage and this storage resizing function is skipped.

What is the solution?
To rewrite the Vali Deployer to not use a ManagedResource.

What you expected to happen:
When Vali ManagedResource is annotated with resources.gardener.cloud/ignore manually it stays until the annotation is removed again manually.

How to reproduce it (as minimally and precisely as possible):

make kind-up

make gardener-up

$ k -n garden annotate mr vali resources.gardener.cloud/ignore=true

See what is the default Vali's default PV and add the same value here

logging:
  enabled: true
  vali:
    enabled: true
  garden:
    storage: "100Gi"

make gardener-up

Now you will see that the annotation is removed.

Anything else we need to know:
This inconvenience interferes only with the central Vali debugging so this is not actually a bug.
This is only information for the Logging Team.
This is why I posted the issue here.

Environment: