Add support for Azure DevOps API-based datasource

[jessehouwing]

What would you like Renovate to be able to do?

Renovate has support for Azure-DevOps built-in tasks and the Azure DevOps Marketplace's public tasks.

But it doesn't support tasks that were installed directly into an Azure DevOps Organization (tfx build tasks upload) nor does it support extensions that are privately listed on the marketplace.

Renovate also won't be too helpful for people using Team Foundation Server or Azure DevOps Server, since these might not even have all tasks available, nor won't they have the same list of built-in tasks available to them.

Each instance of Azure DevOps however, has a set of APIs that one can query to provide grab the data needed directly from the instance. Below is a set of APIs to query to get all the information required to give proper advice

If you have any ideas on how this should be implemented, please tell us here.

• Have a way to configure the PAT to access the marketplace and the Azure DevOps Organization URI
• Potentially support Azure DevOps Server
• Query all tasks installed in an organization to get the task name's and ids (GET https://dev.azure.com/jessehouwing/_apis/distributedtask/tasks/)
  • Tasks can be identified by:
    • task.name
    • task.id
    • task.contributionIdentifier.task.name
    • task.contributionIdentifier.task.id
  • The built-in tasks can be identified by "serverOwned":true
• Query each task by id to get a list of versions (GET https://dev.azure.com/jessehouwing/_apis/distributedtask/tasks/{{task.Id}}?allversions=true
• Use the ResourceArea's API to find the ExtensionManagement API (GET https://dev.azure.com/jessehouwing/_apis/resourceareas/)
• Fetch the extention manifest from the marketplace API (GET https://extmgmt.dev.azure.com/jessehouwing/_apis/extensionmanagement/installedextensionsbyname/jessehouwing/jessehouwing-vsts-variable-tasks?api-version=7.1-preview.1&assetTypes=Microsoft.VisualStudio.Services.Manifest) using the contribution identifier task.contributionIdentifier.
• Grab the repository data (repository.uri) from the fetched manifest file (GET https://jessehouwing.gallerycdn.vsassets.io/extensions/jessehouwing/jessehouwing-vsts-variable-tasks/3.0.10/1670319234705/Microsoft.VisualStudio.Services.Manifest)

Ideally Renovate should also suggest:

• Tasks renames. These can be found by looking for a task with the same task.id, a higher task.version, but a different task.name.
• Removing deprecated tasks. These can be found by looking at the task.deprecated: true.
• Making it configurable to suggest preview versions task.preview: true.

If RenovateBot would support Private Extensions, then a few new issues pop-up, since private extensions can publish the same tasks (same task.name), even with the same version. In that case a task must be disambiguated by its task.id or by using the long name task.contributionIdentifier.task.name.

The current mechanism, with statically generated files, would allow suggestions for Azure DevOps Server in the case of a non-marketplace connected server instance where the admin would have to download and install the extension manually.

We should probably make the desired sources configurable, as renovatebot might expose the names of internal extensions and tasks through calling out to GitHub.

[jessehouwing]

@JamieMagee I found the old Azure Pipelines issue and for most of the problems with the current extension, I think I know the solution and have documented the correct APIs to use above.

• Related: Support for Azure Pipelines Tasks #15818

[bdovaz]

Related renovatebot/azure-devops-marketplace#42

If it is not fixed it is very confusing because it creates a PR for you to upgrade to a version that does not exist in your organization's instance yet since from what @jessehouwing comments not all instances are upgraded at the same time nor do you know when it will be upgraded.

That detail is beyond the user's control, i.e. Microsoft decides when to update each instance in the Azure DevOps Server (Cloud).

Is there at least some workaround @jessehouwing that can be done with package rules (consume the organization's json and check if that version exists or not as a precondition)? Or is it a too advanced case for package rules?

[mathisgauthey]

Looking for the same thing, have you found a workaround ?

[bdovaz]

@jessehouwing to advance this feature could not be divided into several iterations? Because otherwise I see that it will never be addressed?

The basic thing is that the data source changes from that static source in that repository to use the API you mention so that at least the PRs are already reliable in that sense and the version to which you are going to update really exists in your organization.

The manager is in beta so it makes sense to first cover the most common and basic cases and then extend to more advanced cases: Azure DevOps Server, GitHub + Azure DevOps, task renames, task deprecations, task previews, ...

[bdovaz]

@jessehouwing how can we get this issue moving? I wouldn't mind starting to work on at least the basics (use the API instead of the static source) through a PR but it doesn't even have its relevance with its corresponding issue yet, it's just a mere discussion....

[jessehouwing]

It was moved to a discussion, because the core team wasn't picking this up. I could build this in C# or Pwsh, but not easily in the right language for RenovateBot.

I'd say, get started, make us happy.

[bdovaz]

@jessehouwing PR created #32966

[RDalziel]

Am i right in following this that this will also be able to cover custom extensions which have been published internally?

[jessehouwing]

Yes, the above proposal would also work with private extensions.

[mathisgauthey]

Hey there, I see that #32966 was merged.

Is there any documentation on how to implement the changes on our side so that it uses https://BASE_URL/WORKSPACE/_apis/distributedtask/tasks/ to find tasks versions and not https://github.com/renovatebot/azure-devops-marketplace ?

[mathisgauthey]

Regarding the condition I was saying, maybe it has to do with the fact that I do not have it configured:

https://docs.renovatebot.com/configuration-options/#hostrules

That is used here:

renovate/lib/modules/datasource/azure-pipelines-tasks/index.ts

Line 36 in bddef47

const { token } = hostRules.find({

@mathisgauthey can you try that?

Indeed, I don't have the following :

{
  "platform": "azure",
  "endpoint": "https://BASE_URL/ORG_NAME/",
  "token": "process.env.RENOVATE_TOKEN",
  "azure-pipelines": {
    "enabled": true
  },
  "repositories": [
    "Project/RepoName"
  ],
  "prHourlyLimit": 0,
  "baseBranches": [
    "main"
  ],
  "packageRules": [
    {
      "matchDatasources": [
        "azure-pipelines-tasks"
      ],
      "extractVersion": "^(?<version>\\d+)"
    }
  ]
}

I am not using hostRules as I use renovate entirely for tasks pipeline version, hence no need for an added package manager.

How can I run my config locally when cloning the renovate-bot locally ? It asks for a github token which I don't have, even thought I added a config.js file with my settings from above.

Edit : I managed to roll up a dev environment fast enough to test it locally with a PAT token from Azure, I can only assume you are right :

So what do you think, maybe he should get the token from GlobalConfig.get('token') instead of trying to find hostRules ?

Or is it bad practice ? I'm fairly new (30m in) to typescript and Renovate bot, hence the question.

[bdovaz]

@mathisgauthey I think that if you put in your renovate.json the following should get a token:

{
  "hostRules": [
    {
      "matchHost": "azure.com",
      "hostType": "azure-pipelines-tasks",
      "token": "PAT"
    }
  ]
}

https://docs.renovatebot.com/configuration-options/#hostrules

[mathisgauthey]

@mathisgauthey I think that if you put in your renovate.json the following should get a token:

{
  "hostRules": [
    {
      "matchHost": "azure.com",
      "hostType": "azure-pipelines-tasks",
      "token": "PAT"
    }
  ]
}

https://docs.renovatebot.com/configuration-options/#hostrules

Managed to make it work using :

{
  "hostRules": [
    {
      "matchHost": "BASE_URL_OF_AZURE_SELF_HOSTED_INSTANCE",
      "hostType": "azure-pipelines-tasks",
      "token": "PAT"
    }
  ]
}

And yes everything works fine. Thanks a lot for your contribution !

But wouldn't it be nice if we would instead take the token directly from the "token": "process.env.RENOVATE_TOKEN", and allow not to set hostRules ?

[bdovaz]

@mathisgauthey I was suggested to change it to hostrules because it may not exist that token and also because it is more powerful and configurable as far as I can see: #32966 (comment)

[mathisgauthey]

I see, then maybe we should plan on updating the doc so that the new behavior for azure DevOps is known by users. I set to work on this in the following days.

[RDalziel]

I've managed to get this going too with the host rules described above.

One thing I think I've noticed is that some of our pipelines have used to the contribution id to refer to the task rather than the name. Would people be comfortable with the checks to go against both the name and the contribution ids of the task ? 🤔

[jessehouwing]

The old file based system looks at a combination of names: Extension.contribution.name Extension.contrikution.id Name Id Those are the 4 officially supported ways to name your tasks in yaml.

…

On Thu, Jan 16, 2025, 08:47 Richard Dalziel ***@***.***> wrote: I've managed to get this going too with the host rules described above. One thing I think I've noticed is that some of our pipelines have used to the contribution id to refer to the task rather than the name. Would people be comfortable with the checks to go against both the name and the contribution ids of the task ? 🤔 — Reply to this email directly, view it on GitHub <#24820 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA724S7JXDYQ7ANOJFRMTQD2K5PZ7AVCNFSM6AAAAABRDZHY72VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBVGE3DOMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RDalziel]

Apologies - just realised this is all in the OP!

[bdovaz]

@RDalziel @jessehouwing now that I have made this first functional version, it is a matter of improving it with the initial checklist of @jessehouwing that has created in this thread. I would like you to give me a hand if possible, thanks.

[bdovaz]

The end result should be to remove that fallback that checks against the static json that in this first version I didn't want to remove just in case.