CIPD and 3pp for chromium dependencies

[TOC]

What is CIPD?

CIPD stands for "Chrome Infrastructure Package Deployment".
Its code and docs live within the luci-go project.
Chromium uses CIPD to avoid checking large binary files into git, which git does not handle well.
gclient supports CIPD packages in the same way as git repositories. They are specified in DEPS and updated via gclient sync.
You can browse Chromium's CIPD repository online.

What is 3pp?

3pp stands for "Third Party Packages" which allows uniform cross-compiliation, version tracking and archival for third-party software packages for distribution via CIPD.
The code and docs live within the recipe module support_3pp.
By specifying a 3pp package, you can define how to build certain artifacts and where to upload to CIPD. Then our packagers will do the rest for you.

Why use CIPD & 3pp?:

CIPD is our solution to storing large binary blobs directly in git, which git is not good at.
Building these packages with 3pp is beneficial because:
- It makes how each binary file was created clear and verifiable.
- It avoids the need for maintaining a list of ACLs for uploading to CIPD.

Adding a new 3pp package

1. Set up a new directory for your dependency

You'll first want somewhere in the repository in which your dependency will live. For third-party dependencies, this should typically be a subdirectory of //third_party. You'll need to add the same set of things to that directory that you'd add for a non-CIPD dependency -- OWNERS, README.chromium, etc.

For example, if you want to add a package named sample_cipd_dep, you might create the following:

  third_party/
    sample_cipd_dep/
      LICENSE
      OWNERS
      README.chromium

For more on third-party dependencies, see adding_to_third_party.md.

2. Set up the 3pp subdirectory

The 3pp subdirectory will store all the 3pp related files, including a 3pp spec (3pp.pb), as well as scripts, patches and/or tools to build the software from source. It should be placed directly under the package directory.

Staying with the example from above, the sample_cipd_dep directory may be like the following.

Note that among the files in 3pp subdirectory, the 3pp.pb is always required. The rest are optional, depending on how the 3pp.pb is specified.

  third_party/
    sample_cipd_dep/
      LICENSE
      OWNERS
      README.chromium
      3pp/
        3pp.pb  # REQUIRED
        bootstrap.py
        fetch.py
        install.sh
        install_win.sh
        patches/
          0001-foo.patch

2.1 The file `3pp.pb` (Required)

3pp.pb is a text proto specified by the spec.proto schema. It is broken up into two main sections:

create: allows you to specify how the package software gets created, and allows specifying differences in how it's fetched/built/tested on a per-target basis. See here for more details.
upload: contains some details on how the final result gets uploaded to CIPD. See here for more details.

Staying with the example from above, the file sample_cipd_dep/3pp/3pp.pb may be like the following:

create {
  source {
    url {
      download_url: "https://some_url_link/foo.zip"
      version: "1.0.0"
      extension: ".zip"
    }
    patch_version: "cr0"
    unpack_archive: true
  }
}

upload {
  pkg_prefix: "tools"
  universal: true
}

While the above example could meet most of the use case, 3pp.pb is capable of handling more complicated use case like the following:

# create section that is shared by linux-.* and mac-.* platforms
create {
  platform_re: "linux-.*|mac-.*"
  source {
    git {
      repo: "<one_git_repo>"
      tag_pattern: "v%s",

      # Fixed to 3.8.x for now.
      version_restriction: { op: LT val: "3.9a0"}
    }
    patch_dir: "patches"
  }
  build {
    # Can also leave as blank since the script name defaults to "install.sh"
    install: "install.sh"
  }
}

# create section that is specific to linux-.* platforms
create {
  platform_re: "linux-.*"
  build {
    dep: "<dep_foo>"
    dep: "<dep_bar>"

    tool: "<tool_foo>"
  }
}

# create section that is specific to linux-arm.* and linux-mips.* platforms
create {
  platform_re: "linux-arm.*|linux-mips.*"
  build {
    tool: "<tool_bar>"
  }
}

# create section that is specific to windows-*
create {
  platform_re: "windows-.*"
  source { script { name: "fetch.py" } }
  build {
    install: "install_win.sh"
  }
}

upload { pkg_prefix: "tools" }

2.2 The file `install.sh` (Optional)

When the build message is specified in 3pp.pb, the file specified in "build.install" (Default to "install.sh") will be run to transform the source into the built product.

Staying with the example from above, the file sample_cipd_dep/3pp/install.sh may be like the following.

Note that during the build stage, the 3pp directory and all its dependent 3pp directories (i.e. the tool and dep from the build message in 3pp.pb) will be copied to a different directory. So commands in install.sh should not refer to files that are outside of these directories.

#!/bin/bash
# Copyright 2021 The Chromium Authors
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

set -e
set -x
set -o pipefail

# An auto-created directory whose content will ultimately be uploaded to CIPD.
# So the commands below should output the built product to this directory.
PREFIX="$1"

# Commands to transform the source into the built product and move it to $PREFIX
./configure
make install

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
cp -a bin_foo "$SCRIPT_DIR/bootstrap.py" "$PREFIX"

2.3 The file `fetch.py` (Optional)

When specifying the source in 3pp.pb, it is possible to use a custom catch-all script to probe for the latest version and obtain the latest sources. A simple example can be like the following:

Note that this python script should be python3-compatible.

#!/usr/bin/env python3
# Copyright 2023 The Chromium Authors
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

import argparse
import json
import os
import urllib


def do_latest():
  print(urllib.urlopen('some_url/master/VERSION').read().strip())


def get_download_url(version, platform):
  target_os, target_arch = platform.split('-')
  ext = '.zip' if target_os == 'windows' else '.tar.gz'
  partial_manifest = {
    'url': ['download_url1', 'download_url2'],
    'ext': ext,
  }
  print(json.dumps(partial_manifest))


def main():
  ap = argparse.ArgumentParser()
  sub = ap.add_subparsers()

  latest = sub.add_parser("latest")
  latest.set_defaults(func=lambda _opts: do_latest())

  download = sub.add_parser("get_url")
  download.set_defaults(
    func=lambda _opts: get_download_url(
      os.environ['_3PP_VERSION'], os.environ['_3PP_PLATFORM']
    )
  )

  opts = ap.parse_args()
  opts.func(opts)


if __name__ == '__main__':
  main()

3. Add "3pp" subdirectory to `chromium/src` repo

3pp CQ builders (Presubmit)

The following are the optional CQ builders to run the presubmit check for CLs that have the directory "3pp" in the patchset.

3pp-linux-amd64-packager: For builds on the linux-amd64 platform or universal builds (e.g. to be used by all platforms)

3pp CI builders (Postsubmit)

Once the CLs pass the CQ and get landed, the following CI builders will periodically build all the 3pp packages that match the given platforms and upload any new results to CIPD.

3pp-linux-amd64-packager: For builds on the linux-amd64 platform or universal builds (e.g. to be used by all platforms)

4. Add your CIPD package to DEPS

Once your CIPD package is created by the 3pp CI builders, you can add it to DEPS by adding an entry of the following form to the deps dict:

deps = {
  # ...

  # This is the installation directory.
  'src/third_party/sample_cipd_dep': {

    # In this example, we're only installing one package in this location,
    # but installing multiple package in a location is supported.
    'packages': [
      {
        'package': 'chromium/third_party/sample_cipd_dep',
        'version': 'TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC',
      },
    ],

    # As with git-based DEPS entries, 'condition' is optional.
    'condition': 'checkout_android',
    'dep_type': 'cipd',
  },

  # ...
}

This will result in CIPD package chromium/third_party/sample_cipd_dep at TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC being installed in src/third_party/sample_cipd_dep (relative to the gclient root directory).

Updating a CIPD dependency

To modify a CIPD dependency, follow steps 2 and 3 above, then modify the version listed in DEPS.

Miscellaneous

Create a cipd.yaml file in the old way

While it is strongly suggested to use 3pp infrastructure, there are existing flows that create a cipd.yaml file by a GN template or a script, and upload it to CIPD by builders with custom recipes.

Examples are:

android-androidx-packager
android-sdk-packager

Generating cipd.yaml via GN Template:

The cipd_package_definition template in build/cipd/cipd.gni can be used to create the yaml definition as part of Chromium's normal build process. Declare a target like:

cipd_package_definition("my_cipd_package") {
  package = "path/to/cipd/package"
  description = "Prebuilt test binary."
  install_mode = "copy"
  deps = [ "//path/to:test_binary_target" ]
  sources = [ "//path/to:test_binary_file" ]
}

Permissions in CIPD

You can check a package's ACLs with cipd acl-list:

$ cipd acl-list chromium/third_party/sample_cipd_dep
...

Permissions in CIPD are handled hierarchically. You can check entries higher in the package hierarchy with cipd acl-list, too:

$ cipd acl-list chromium
...

By default, cria/project-chromium-cipd-owners own all CIPD packages under chromium/. If you're adding a package, talk to one of them.

To obtain write access to a new package, ask an owner to run:

$ cipd acl-edit chromium/third_party/sample_cipd_dep -owner user:email@address.com

Troubleshooting

A file maintained by CIPD is missing, and gclient sync doesn't recreate it.

CIPD currently caches installation state. Modifying packages managed by CIPD will invalidate this cache in a way that CIPD doesn't detect - i.e., CIPD will assume that anything it installed is still installed, even if you deleted it. To clear the cache and force a full reinstallation, delete your $GCLIENT_ROOT/.cipd directory.

Note that there is a bug on file where gclient sync does not reset CIPD entries that are changed locally.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cipd_and_3pp.md

cipd_and_3pp.md

CIPD and 3pp for chromium dependencies

What is CIPD?

What is 3pp?

Why use CIPD & 3pp?:

Adding a new 3pp package

1. Set up a new directory for your dependency

2. Set up the 3pp subdirectory

2.1 The file `3pp.pb` (Required)

2.2 The file `install.sh` (Optional)

2.3 The file `fetch.py` (Optional)

3. Add "3pp" subdirectory to `chromium/src` repo

3pp CQ builders (Presubmit)

3pp CI builders (Postsubmit)

4. Add your CIPD package to DEPS

Updating a CIPD dependency

Miscellaneous

Create a cipd.yaml file in the old way

Generating cipd.yaml via GN Template:

Permissions in CIPD

Troubleshooting

Files

cipd_and_3pp.md

Latest commit

History

cipd_and_3pp.md

File metadata and controls

CIPD and 3pp for chromium dependencies

What is CIPD?

What is 3pp?

Why use CIPD & 3pp?:

Adding a new 3pp package

1. Set up a new directory for your dependency

2. Set up the 3pp subdirectory

2.1 The file 3pp.pb (Required)

2.2 The file install.sh (Optional)

2.3 The file fetch.py (Optional)

3. Add "3pp" subdirectory to chromium/src repo

3pp CQ builders (Presubmit)

3pp CI builders (Postsubmit)

4. Add your CIPD package to DEPS

Updating a CIPD dependency

Miscellaneous

Create a cipd.yaml file in the old way

Generating cipd.yaml via GN Template:

Permissions in CIPD

Troubleshooting

2.1 The file `3pp.pb` (Required)

2.2 The file `install.sh` (Optional)

2.3 The file `fetch.py` (Optional)

3. Add "3pp" subdirectory to `chromium/src` repo