rough draft of quick-start/flesh out USAGE.md

DEVELOP.md

@@ -1,9 +1,14 @@
 # Contributing to the RAPIDS devcontainers
 
+## Features
+
 From the official devcontainer [documentation on Features](https://containers.dev/implementors/features/):
 > Development container "Features" are self-contained, shareable units of installation code and development container configuration.
 
-In short, a "feature" is a layer in a Dockerfile which encapsulates a reusable bit of logic executed when building a Docker image.
+In short, a "feature" is a layer in a Dockerfile which encapsulates a reusable
+bit of logic executed when building a Docker image. It is not a docker layer to
+put on top of or copied into other layers. It is the script that creates a
+layer.
 
 This repository defines features to install the following dev tools, compilers, and SDKs:
 
@@ -21,3 +26,9 @@ This repository defines features to install the following dev tools, compilers,
 * [sccache](features/src/sccache/)
 * [devcontainer-utils](features/src/utils/)
 * [rapids-build-utils](features/src/rapids-build-utils/)
+
+These scripts assume that apt utilities are available, and thus only run on debian-based images.
+
+## Base images
+
+Base images are composed in [matrix.yml](./matrix.yml) using [YAML anchors](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/). These get built on Github Actions ([release.yml](./.github/workflows/release.yml) and [test.yml](.github/workflows/test.yml))

README.md

@@ -1,11 +1,25 @@
 # RAPIDS [devcontainers](https://containers.dev/)
 
-This repository contains features and workflows for building development containers to support local dev and CI for NVIDIA [RAPIDS](https://github.com/rapidsai), [CCCL](https://github.com/nvidia/cccl), and [Legate](https://github.com/nv-legate).
+This repository contains features and workflows for building development
+containers to support local dev and CI for NVIDIA
+[RAPIDS](https://github.com/rapidsai), [CCCL](https://github.com/nvidia/cccl),
+and [Legate](https://github.com/nv-legate).
+[Devcontainers](https://containers.dev/) are an open standard for specifying the
+creation and execution of Docker containers for developing a codebase. It's like
+using a docker image to develop, but there's some extra configuration and
+installation that can be done. It also provides some alternative ways of
+composing functionality and configuration that can augment Docker's
+capabilities.
 
-We've chosen to use a monorepo, but it is similar in spirit to the official [devcontainers/features](https://github.com/devcontainers/features) and [devcontainers/images](https://github.com/devcontainers/images) repositories.
+In addition to scripts that set up the devcontainer environment for things like GitHub auth, this repo
+contains reusable scripts to install software in arbitrary containers, aiding in composition and code sharing.
+A "feature" in VSCode terms refers to these installation scripts. The script for each feature runs when
+creating the devcontainer.
 
-### For details on using the RAPIDS devcontainers, see [`USAGE.md`](USAGE.md).
+We've chosen to use a monorepo for the features here, but it is similar in spirit to the official [devcontainers/features](https://github.com/devcontainers/features) and [devcontainers/images](https://github.com/devcontainers/images) repositories.
 
-### For details on contributing to this repository, see [`DEVELOP.md`](DEVELOP.md).
+## For details on using the RAPIDS devcontainers, see [`USAGE.md`](USAGE.md).
 
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+## For details on contributing to this repository, see [`DEVELOP.md`](DEVELOP.md).
+
+## See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags). These tags are used as base images in devcontainers, and aren't really meant to be used directly.

USAGE.md

@@ -1,30 +1,144 @@
 # Using the RAPIDS devcontainers
 
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+- [Using the RAPIDS devcontainers](#using-the-rapids-devcontainers)
+  - [Quick start](#quick-start)
+    - [Using devcontainers in VS Code](#using-devcontainers-in-vs-code)
+    - [Using devcontainers from the terminal](#using-devcontainers-from-the-terminal)
+  - [Available tools in the devcontainer](#available-tools-in-the-devcontainer)
+    - [Generated build scripts](#generated-build-scripts)
+    - [rapids-build-utils](#rapids-build-utils)
+    - [Native build tools - CMake, python builds](#native-build-tools---cmake-python-builds)
+  - [Adding projects: manifest.yaml file](#adding-projects-manifestyaml-file)
+  - [Using the pre-built images](#using-the-pre-built-images)
+    - [Using in `devcontainer.json`](#using-in-devcontainerjson)
+    - [Custom devcontainers](#custom-devcontainers)
+  - [Build caching with `sccache`](#build-caching-with-sccache)
+    - [Build caching with private S3 buckets](#build-caching-with-private-s3-buckets)
+    - [Using GitHub OAuth to issue S3 credentials via Hashicorp Vault](#using-github-oauth-to-issue-s3-credentials-via-hashicorp-vault)
+
+## Quick start
+
+So, you have cloned a repo that you've heard has a devcontainer. You can see the file(s) for yourself by looking
+in your repo's `.devcontainer` folder. You may find a `devcontainer.json` file, or you may find some number of folders, such as `cuda12.0-conda` and `cuda12.0-pip`. If you find folders, these each contain a `devcontainer.json`
+file. These files specify how to create and run a container that leaves you with a good setup to do development.
+
+There are at least 2 ways to consume these devcontainer.json files. VS Code was
+the original home of devcontainers prior to becoming an open specification, and
+it remains a good way to use them. If you prefer not to use VS Code, the
+[devcontainers-cli](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli)
+application will build the devcontainer and allow you to interact with it.
+
+### Using devcontainers in VS Code
+
+[The VS Code
+docs](https://code.visualstudio.com/docs/devcontainers/containers#_quick-start-open-an-existing-folder-in-a-container)
+are the definitive source of information for this topic.
+
+Specifically for RAPIDS repos, we frequently have multiple folders for different
+library configurations. Pay attention to which build environment you need when
+launching your devcontainer. You can switch between them
+
+### Using devcontainers from the terminal
+
+The [devcontainer-cli](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) project
+allows you to run and interact with devcontainers from the terminal. It uses NodeJS, so you need
+to install that in order to run it.
+
+You need to specify the devcontainer.json file and workspace folder to use when starting the devcontainer:
 
-* [Using the pre-built images](#using-the-pre-built-images)
-  * [Using in `devcontainer.json`](#using-in-devcontainerjson)
-* [Build caching with sccache](#build-caching-with-sccache)
+```
+devcontainer up --config .devcontainer/cuda12.0-pip/devcontainer.json --workspace-folder=.
+```
+
+## Available tools in the devcontainer
+
+### Generated build scripts
+
+Several scripts are generated for you based on the contents of manifest.yaml. By
+default, manifest.yaml comes from [the RAPIDS/devcontainers
+repo](https://github.com/rapidsai/devcontainers/blob/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml),
+but you can use your own local manifest.yaml file. Refer to the "Adding
+projects" section.
+
+The generated scripts are:
+* /usr/bin/build-*
+* /usr/bin/clean-*
+* /usr/bin/clone-*
+* /usr/bin/configure-*
+
+Here the * is a placeholder for the project name and the kind of
+build. For example, a project with a python component in manifest.yaml
+will have `build-ucxx-python`.
+
+These scripts may trigger side-effects. For example, build-* scripts are only
+generated for projects that exist in the workspace. If you are working on UCXX,
+which depends on RMM, the default workspace only mounts UCXX and generates
+build-ucxx scripts. If you want RMM build scripts also, you can run `clone-rmm`,
+which will clone RMM into your workspace and generate build scripts for it.
+
+### rapids-build-utils
+
+These are meta-build scripts. They assist with setting up the workspace and reloading things when important configuration
+changes happen. You mostly won't need to worry about these,
+but it's good to be aware of them. They come from the [rapidsai/devcontainers repo](https://github.com/rapidsai/devcontainers/tree/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/bin)
+
+### Native build tools - CMake, python builds
+
+The generated scripts mentioned above will take care of running
+build tools for you. However, if you need to run the build tools
+manually, you can `cd` into your source code folder, which is
+mounted as a subfolder in `/home/coder`.
 
+## Adding projects: manifest.yaml file
+
+The build script generation is controlled with a manifest.yaml file, which by default comes from [the rapidsai/devcontainers
+repo](https://github.com/rapidsai/devcontainers/blob/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml)
+
+If you would like to add your project, you can submit a PR to the rapidsai/devcontainers repo. Before you do that, though, you can
+test it locally. Start by copying manifest.yaml from the rapidsai/devcontainers repo. You can put it anywhere, but let's say we put it in .devcontainer/manifest.yaml.
+
+Now open a devcontainer.json file that you want to work with. These
+will likely live in a .devcontainer subfolder, such as cuda12.0-pip. In this file, add a top-level key with this:
+
+```
+ "containerEnv": {
+   "PROJECT_MANIFEST_YML": "${localWorkspaceFolder}/.devcontainer/manifest.yaml"
+ },
+```
+
+Rebuild or re-open your devcontainer, and you should now see updated
+generated scripts.
 
 ## Using the pre-built images
 
+The choice of using a pre-built image refers to the build/args/BASE entry in devcontainer.json. The pre-built
+images are not meant to be used directly. The rapids-build-utils scripts are installed with a "feature," so
+they won't be present if you directly run a pre-built image with Docker instead of with a devcontainer tool (VS Code or devcontainer-cli)
+
 We publish a [matrix](matrix.yml) of pre-built images to DockerHub to accelerate initializing local devcontainers, GitHub Codespaces, and CI jobs.
 
 The features that comprise the image are noted in the image tags. If no version is defined for a tool or SDK, the image includes the latest available version at image build time.
 
 > **NOTE:** `git`, `git-lfs`, `github-cli`, `gitlab-cli`, `cmake`, `ninja`, `sccache`, and our devcontainer-utils [are included](image/.devcontainer/devcontainer.json#L12-L33) in each pre-built image.
 
+See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+
 ### Using in [`devcontainer.json`](https://containers.dev/implementors/json_reference/#image-specific)
 
 The pre-built images can be used as the `"image"`, or as the base of a Dockerfile in `"build"`, in `devcontainer.json`:
 
 <details><summary>devcontainer.json using pre-built image</summary><pre>{<br/>  "image": "rapidsai/devcontainers:24.02-cpp-llvm16-cuda12.0-nvhpc23.5-ubuntu22.04",<br/>  "hostRequirements": { "gpu": true },<br/>  "workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",<br/>  "workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"<br/>}</pre></details>
 
+### Custom devcontainers
+
 You can also build a custom devcontainer by composing individual features:
 
 <details><summary>devcontainer.json using individual features</summary><pre>{<br/>  "image": "ubuntu:22.04",<br/>  "features": {<br/>    "ghcr.io/rapidsai/devcontainers/features/cmake:24.02": {},<br/>    "ghcr.io/rapidsai/devcontainers/features/ninja:24.02": {},<br/>    "ghcr.io/rapidsai/devcontainers/features/sccache:24.02": {<br/>      "version": "0.5.4"<br/>    }<br/>  },<br/>  "overrideFeatureInstallOrder": [<br/>    "ghcr.io/rapidsai/devcontainers/features/cmake",<br/>    "ghcr.io/rapidsai/devcontainers/features/ninja",<br/>    "ghcr.io/rapidsai/devcontainers/features/sccache"<br/>  ],<br/>  "workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",<br/>  "workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"<br/>}</pre></details>
 
+You can also add libraries or programs on top of a pre-built image using this
+same mechanism. These are not baked into the image, but cached as a docker
+layer.
+
 > **NOTE:** Feature updates published since your most recent image build will invalidate your docker image layer cache, meaning it can take the [devcontainers CLI](https://github.com/devcontainers/cli) longer to initialize containers composed from individual features.
 
 ## Build caching with `sccache`