409 Conflict when updating StatefulSet

[Fahani]

Hello,

I am trying to update the replicas and a couple env variables for a StatefulSet. This is the code

$statefulSet = $this->kubernetesCluster->getStatefulSetByName($statefulSetName, 'alloy');
$currentReplicas = $statefulSet->getReplicas();
$velocity = (int)$statefulSet->getAnnotations()['velocity'];
$replicasNeeded = $this->calculateTheNeededReplicaPerStatefulSet($velocity, $depth);

// If the current replicas are the same as the one we need, nothing to do.
if ($currentReplicas === $replicasNeeded) {
    continue;
}
// We update the starting point for the new replica group
$envPosition = 0;
$envVars = $statefulSet->getAttribute('spec.template.spec.containers.0.env');
foreach ($envVars as $envVar) {
    if ($envVar['name'] === 'START_FROM_GLOBAL_POSITION') {
        break;
    }
    $envPosition++;
}
// After trying different options, sleep for one second was what it worked for me to perform multiple
// actions over the same StatefulSet
sleep(2);
$statefulSet
    ->setAttribute(
        "spec.template.spec.containers.0.env.$envPosition",
        [
            'name' => 'START_FROM_GLOBAL_POSITION',
            'value' => "$minGlobalPositionProcessedPerStatefulSets[$statefulSetName]"
        ]
    )
    ->update();

// We scale the replicas
sleep(2);
$statefulSet->setReplicas($replicasNeeded)->update();

// We update the info about the size of the consumer group
$envPosition = 0;
$envVars = $statefulSet->getAttribute('spec.template.spec.containers.0.env');
foreach ($envVars as $envVar) {
    if ($envVar['name'] === 'CONSUMER_SIZE_GROUP') {
        break;
    }
    $envPosition++;
}
sleep(2);
$statefulSet
    ->setAttribute(
        "spec.template.spec.containers.0.env.$envPosition",
        ['name' => 'CONSUMER_SIZE_GROUP', 'value' => "$replicasNeeded"]
    )
    ->update();
} catch (Throwable $e) {
 $this->logger->error($e->getMessage(), [$statefulSetName]);
}

I am using sleeps between updates of those three attributes hoping they will get rid of the error but I still getting some of them:

Client error: `PUT https://kubernetes.default.svc/apis/apps/v1/namespaces/alloy/statefulsets/bulkprocessor?pretty=1` resulted in a `409 Conflict` response:
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Operation cannot be f (truncated...)

I also tried $statefulSet->refresh() between updates. I tried as well doing an update of the two env vars and the replicas in the same update but it didn't work, that's why I am doing 3 updates.

Is there a better way to accomplish this?

Thank you

[Fahani]

Hello @rennokki! Tagging you here since you probably know this from the top of your head.

Thank you 🙏