[Question]: Microsoft.Azure.DevOps.Pipelines.Agent VM extension releases

[rybal06]

Describe your question

It looks like this VM extension in Azure (required when using VMSS agents) is out-of-date. This is added automatically by Azure DevOps when using VMSS agent pools.

In the settings block; we see the agent version as 3.236.0 (released February 2024)

{
  "isPipelinesAgent": true,
  "agentFolder": "/agent",
  "agentDownloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.236.0/vsts-agent-linux-x64-3.236.0.tar.gz",
  "enableScriptDownloadUrl": "https://vstsagenttools.blob.core.windows.net/tools/ElasticPools/Linux/16/enableagent.sh"
}

We are using VMSS with ephemeral agents, so it is not practical to install a previous version of the agent and then wait for it to update each time the VMSS instance is replaced by Azure DevOps reimage operations.

Questions:

1. When and how is the Microsoft.Azure.DevOps.Pipelines.Agent VM extension updated to reference the latest GA version of the agent package?

2. How can we become aware of when the VM extension is updated to use a newer agent version?

3. Is there any particular reason that the extension is still referencing 3.236.0 (awaiting QA, bug found,etc)?

Versions

VM extension version: 1.26
Agent version: 3.236.0
Ubuntu 20.04 LTS

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Operation system

Ubuntu 20.04LTS

Version controll system

No response

Azure DevOps Server Version (if applicable)

No response

[elessard-genetec]

I've observed the same behavior, and in my case this caused our pipelines to start failing since we are depending on a feature that was added in the Agent version 3.240.1.

Looks like Azure DevOps service updated the VMSS extension of our VMSS in the past 24 hours, and the new VMSS extension is actually configured to download an older version of the Agent (previously, Agent version 3.246.0 was downloaded, at least for the past couple of weeks).

[rybal06]

I've observed the same behavior, and in my case this caused our pipelines to start failing since we are depending on a feature that was added in the Agent version 3.240.1.

Looks like Azure DevOps service updated the VMSS extension of our VMSS in the past 24 hours, and the new VMSS extension is actually configured to download an older version of the Agent (previously, Agent version 3.246.0 was downloaded, at least for the past couple of weeks).

Thank you for sharing that information; we had extended outages on 2024/11/05 and 2024/11/04 where our entire agent pool went offline for an hour while under heavier load. Checking our pipeline history, version 3.246.0 was being used during the timeframe of our outages.

I tried recreating issues and even built a load testing pipeline which spawned hundreds of jobs and could not re-create the outage a few days later. Checking those pipeline runs, the agent version had been downgraded to 3.236.0.

I wonder if the service team found an issue in 3.246.0 and reverted all customers to 3.236.0?

[elessard-genetec]

I wonder if the service team found an issue in 3.246.0 and reverted all customers to 3.236.0?

I'm a bit surprised to hear that, from my point of view our pipelines using our VMSS agent pool were on version >3.240.1 for at least a month, since we've been relying on a feature introduced in that version since then.

[angaaruriakhil]

We're seeing the same thing. With the new version of the extension, it's using the older version of the pipelines agent. I'm quite interested in why this downgrade has happened.

[ggeorge-pros]

I would add that I'd prefer a way to disable the extension completely. The extension does not support arm64 architectures even though the agent runtime supports it, because the extension configuration is specific to amd64.

We have to create a custom agent startup script for our Kubernetes based agents, and it makes no sense why we can't choose to do that for VMSS agents, especially elastic ones.

[tkasparek]

Hello, thank you for reporting the issue. We did not revert to 3.236.0 so it's definitely not intended and it's 100% a bug somewhere. I will take a look.

[tkasparek]

Update: I've managed to root cause the issue. I will roll out the fix on Monday.

[tkasparek]

The fix is being rolled out and issue should disappear today.