This document provides a detailed procedure for backing up your Azure DevOps (ADO) organization or project using the Azure DevOps Backup Tool from Solidify.
- Determining the correct Azure DevOps project to house our backup infrastructure
- Creating a git repository to hold our backup pipeline resources
- Creating our first backup pipeline
- Storing pipeline variables and secrets in Variable Groups
- Advanced config file usage + examples for various scenarios
- Running our backup pipeline and assessing the results
- Troubleshooting and support requests
Before creating your backup pipeline, select the appropriate Azure DevOps project to house your backup infrastructure. Consider the following guidelines:
- Dedicated Backup Project: Create a dedicated project in Azure DevOps to isolate the backup pipelines, repositories, and related resources from your development activities. This helps in managing access control and ensures that backup-related activities do not interfere with your other projects.
- Permissions: Ensure that the project has the necessary permissions for accessing the resources you want to back up. Administrators and pipeline contributors should have access to this project.
- Naming Conventions: Use clear and descriptive names for the project to easily identify its purpose, such as
Backup-InfrastructureorProject-Backup.
After selecting the appropriate Azure DevOps project, the next step is to create a Git repository to store the resources required for the backup pipeline.
Navigate to your Azure DevOps project, go to the Repos section, and create a new repository. Name it something descriptive like Backup-Pipelines.
Our repository should now be initialized with an empty README:
With your Git repository ready, you can now create your first backup pipeline.
In Azure DevOps, go to the Pipelines section and create a new pipeline:
When asked Where is your code?, select Azure Repos Git (YAML):
Under Select Repository, choose the repository you created in the previous step (Backup-Pipelines):
Under Configure your pipeline, choose Starter pipeline:
You can now design your pipeline. You can bring in the tasks provided by Azure DevOps Backup Tool by clicking Show assistant and then searching for Azure DevOps Backup Tool in the assistant column.
This will bring you into the task configuration. Here you you can view and change how the backup pipeline behaves, where it should look for components to back up, as well as API credentials for Azure DevOps:
Below is a basic YAML configuration for a backup pipeline. In order to make the setup process as simple as possible, please copy the below code block in its entirety and paste into your YAML pipeline text editor:
trigger: none
workspace:
clean: all
variables:
- group: backup-pipeline
- name: workspace
value: '$(System.DefaultWorkingDirectory)/MigrationWorkspace'
- name: System.Debug
value: false # Set to "true" to enable system diagnostics for troubleshooting or support requests
pool:
vmImage: windows-latest
steps:
- task: ado-backup-tool-export@1
displayName: 'ADO Backup Tool: Export'
inputs:
source: '$(sourceUrl)'
sourceOrgName: '$(sourceOrg)'
sourceProject: '$(sourceProject)'
sourceUsername: '$(sourceUsername)'
sourcePAT: '$(migrationToken)'
onPrem: false
workspace: '$(workspace)'
useCustomConfigurations: false
customConfigurationPath: '$(System.DefaultWorkingDirectory)/custom-configs'
resourceAreaPath: true
resourceArtifact: true
resourceAuditLog: true
resourceBoard: true
resourceDeploymentGroup: true
resourceEnvironment: true
resourceGit: true
resourceGitBranchPolicy: true
resourceIterationPath: true
resourcePipeline: true
resourcePullRequest: true
resourceQuery: true
resourceTeam: true
resourceTestplan: true
# resourceTFS: true # Uncomment if there are any TFVC repositories in the source Project
resourceVariableGroup: true
resourceWiki: true
resourceWorkItem: true
env:
SYSTEM_ACCESSTOKEN: $(system.accesstoken)
- task: ArchiveFiles@2
continueOnError: true
condition: succeededOrFailed()
inputs:
rootFolderOrFile: '$(workspace)'
includeRootFolder: false
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)\$(Build.BuildNumber)-export.zip'
replaceExistingArchive: false
- task: PublishPipelineArtifact@1
condition: succeededOrFailed()
continueOnError: true
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)\$(Build.BuildNumber)-export.zip'
artifact: 'Export'
publishLocation: 'pipeline'Optionally, you may comment our or delete any of the resource* inputs if those components are not desired to be backed up.
The resulting pipeline should now look like this:
Now click Save:
In the Save dialogue, click Save:
We are now done with creating our Backup Pipeline. Before we run it, we must specify some parameters to let the pipeline know our API credentials, where our resources are, etc.
We will use Variable Groups to store our pipeline parameters, which will enable us to use the backup pipeline in a dynamic fashion.
Under the Pipelines -> Library menu, add a new Variable Group:
Name the variable group "backup-pipeline" (must match the variable group name in the YAML definition of the backup pipeline). You can change the name at any time as long as your consistently update the variable group name in the YAML definition of the pipeline. Enter the same variables as shown in the screenshot below (replace the values with your own scenario) and click Save.
The necessary variables for the Backup Job are as follows:
migrationToken(The Personal Access Token to be used by the backup job)sourceOrg(The source Azure DevOps organization name)sourceProject(The source ADO Project name)sourceUrl(The URL of the ADO organization)sourceUsername(The email address of the user account to be used by the backup job. Must be the same account as the owner of themigrationToken)
Optionally, you may opt to store your Backup Job variables inside the YAML build pipeline definition itself, or in an Azure KeyVault (use the Link secrets from an Azure key vault as variables setting).
You can extend the functionality of ADO Backup Tool by using custom configuration files. These files will override the default configuration of the underlying application behind the pipeline task, and enable you to change the behavior on a per-component basis.
If you are simply demoing the backup functionality or otherwise have no use for custom configuration files, you can skip ahead to step 6.
Common use cases for using custom config files are:
- Filtering git repositories by name (wildcard supported)
- Filtering build/release pipelines by name (wildcard supported)
- Filtering work items based on a WIQL query
- Overriding default git credentials for git repositories and wikis
Edit your back up pipeline again. You will need to supply the useCustomConfigurations and customConfigurationPath parameters in the task configuration, like this:
useCustomConfigurations: true
customConfigurationPath: '$(System.DefaultWorkingDirectory)/custom-configs'Your pipeline should now look like this:
Save the pipeline before continuing with the next step.
You provide custom configuration files by placing them in the same git repository as your backup/restore pipeline. The naming convention of the configuration files is always "config-[type of component]-[export/import].json.
You can find additional information about the usage of custom configuration files here: https://github.com/solidify/azure-devops-backup-tool-docs/blob/main/custom-configs.md.
You can view templates for individual configuration files for each ADO component here https://github.com/solidify/azure-devops-backup-tool-docs/tree/main/config-templates.
You can find documentation and usage samples with example configuration files for each ADO component here: https://github.com/solidify/azure-devops-backup-tool-docs/tree/main/adapter-docs
As an exercise, let us create a custom configuration file which will let us granularly filter which Work Items to export in the Backup Pipeline.
Start by creating a new folder inside the git repository named Backup-Pipelines:
The new folder name should be custom-configs and the residing file name should be config-workitem-export.json. This will create a configuration file which will affect only Work Items during the backup phase (and not the restore phase):
Paste the following file contents into the new file, and replace ORGNAME with your Azure DevOps organization name, and PROJECTNAME with your target project to be backed up:
{
"configuration":
{
"source": "https://dev.azure.com/ORGNAME",
"target": "D:\\a\\1\\s\\MigrationWorkspace",
"batchSize": 50,
"skip": 0,
"projects": [
{
"name":"PROJECTNAME",
"query":"Select * From WorkItems Where [System.TeamProject] = 'PROJECTNAME' AND [System.CreatedDate] < '2024-08-08T00:00:00.0000000' Order By [Id] Asc"
}
],
"SkipAlreadyDownloadedRevisions": true,
"apiVersion": "7.0"
}
}The resulting file should look like this:
Save the new file by committing it to the version control history:
We are now set up to use our custom configuration file.
Let us verify that the pipeline has successfully picked up our new custom configuration file.
Go ahead and edit the pipeline, and set the System.Debug property to true, like in the below screenshot:
Now, save and run the pipeline. Inspect the log for the ADO Backup Tool: Export task and verify that the task is picking up the correct configuration file. The configuration should appear just like you entered it in the file custom-configs/config-workitem-export.json:
If the configuration as reported by the pipeline is identical to the configuration you specified earlier, then everything is now set up correctly, and you can continue to step 6.
Finally, let us run our newly created Backup Job and verify that the results are as expected.
From the pipeline menu under Pipelines, click Run pipeline:
In the Run pipeline dialogue, simply click Run:
After clicking Run, you will be taken to the Run results screen. Simply wait here until the pipeline finishes or times out (you might need to refresh your browser tab). You may also need to authorize the pipeline to access all of the necessary variable groups.
If you ever need to get back to the Run results screen, simply navigate to Pipelines -> Backup-Pipelines -> (click the latest Run) from anywhere within Azure DevOps.
The pipeline should finish without errors.
From the results screen of the run, you can do either of the following from the screen:
- Click on the Job to view the log file.
- Click on the build artifact to view the components that were successfully backed up. We also require you to send us the build artifact in order to troubleshoot any issues.
From the Artifacts menu, in order to download and inspect the build artifact, simply click the downward-facing arrow to the left of "Export" -> click the 3 dots to the right -> Download artifacts. This will download a .zip-file to your computer which contains the backed up resources.
In the event that the ADO Backup Tool fails, you will receive an error message similar to this one at the bottom of the task log:
Click anywhere in the log to bring the focus to the log file area, then press ctrl+F to search the log file. Enter the search string ERR]. All relevant errors should become highlighted, and you can use the search panel to navigate between them:
If you are making a support request, please make a note of each error and forward all relevant error messages to us via e-mail.
For a more comprehensive guide on troubleshooting the ADO Backup Tool and submitting support requests, please read this article: https://github.com/solidify/azure-devops-backup-tool-docs/blob/main/troubleshooting.md.