rough draft of quick-start/flesh out USAGE.md

DEVELOP.md

@@ -1,9 +1,14 @@
 # Contributing to the RAPIDS devcontainers
 
+## Features
+
 From the official devcontainer [documentation on Features](https://containers.dev/implementors/features/):
 > Development container "Features" are self-contained, shareable units of installation code and development container configuration.
 
-In short, a "feature" is a layer in a Dockerfile which encapsulates a reusable bit of logic executed when building a Docker image.
+In short, a "feature" is a layer in a Dockerfile which encapsulates a reusable
+bit of logic executed when building a Docker image. It is not a docker layer to
+put on top of or copied into other layers. It is the script that creates a
+layer.
 
 This repository defines features to install the following dev tools, compilers, and SDKs:
 
@@ -21,3 +26,9 @@ This repository defines features to install the following dev tools, compilers,
 * [sccache](features/src/sccache/)
 * [devcontainer-utils](features/src/utils/)
 * [rapids-build-utils](features/src/rapids-build-utils/)
+
+These scripts assume that apt utilities are available, and thus only run on debian-based images.
+
+## Base images
+
+Base images are composed in [matrix.yml](./matrix.yml) using [YAML anchors](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/). These get built on Github Actions ([release.yml](./.github/workflows/release.yml) and [test.yml](.github/workflows/test.yml))

README.md

@@ -1,11 +1,25 @@
 # RAPIDS [devcontainers](https://containers.dev/)
 
-This repository contains features and workflows for building development containers to support local dev and CI for NVIDIA [RAPIDS](https://github.com/rapidsai), [CCCL](https://github.com/nvidia/cccl), and [Legate](https://github.com/nv-legate).
+This repository contains features and workflows for building development
+containers to support local dev and CI for NVIDIA
+[RAPIDS](https://github.com/rapidsai), [CCCL](https://github.com/nvidia/cccl),
+and [Legate](https://github.com/nv-legate).
+[Devcontainers](https://containers.dev/) are an open standard for specifying the
+creation and execution of Docker containers for developing a codebase. It's like
+using a docker image to develop, but there's some extra configuration and
+installation that can be done. It also provides some alternative ways of
+composing functionality and configuration that can augment Docker's
+capabilities.
 
-We've chosen to use a monorepo, but it is similar in spirit to the official [devcontainers/features](https://github.com/devcontainers/features) and [devcontainers/images](https://github.com/devcontainers/images) repositories.
+In addition to scripts that set up the devcontainer environment for things like GitHub auth, this repo
+contains reusable scripts to install software in arbitrary containers, aiding in composition and code sharing.
+A "feature" in VSCode terms refers to these installation scripts. The script for each feature runs when
+creating the devcontainer.
 
-### For details on using the RAPIDS devcontainers, see [`USAGE.md`](USAGE.md).
+We've chosen to use a monorepo for the features here, but it is similar in spirit to the official [devcontainers/features](https://github.com/devcontainers/features) and [devcontainers/images](https://github.com/devcontainers/images) repositories.
 
-### For details on contributing to this repository, see [`DEVELOP.md`](DEVELOP.md).
+## For details on using the RAPIDS devcontainers, see [`USAGE.md`](USAGE.md).
 
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+## For details on contributing to this repository, see [`DEVELOP.md`](DEVELOP.md).
+
+## See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags). These tags are used as base images in devcontainers, and aren't really meant to be used directly.

USAGE.md

@@ -1,30 +1,203 @@
 # Using the RAPIDS devcontainers
 
-### See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+- [Using the RAPIDS devcontainers](#using-the-rapids-devcontainers)
+  - [Quick start](#quick-start)
+  - [Detailed start](#detailed-start)
+    - [Using devcontainers in VS Code](#using-devcontainers-in-vs-code)
+    - [Using devcontainers from the terminal](#using-devcontainers-from-the-terminal)
+    - [Generated build scripts](#generated-build-scripts)
+    - [rapids-build-utils](#rapids-build-utils)
+    - [Native build tools - CMake, python builds](#native-build-tools---cmake-python-builds)
+    - [Exiting the devcontainer](#exiting-the-devcontainer)
+  - [Adding projects: `manifest.yaml` file](#adding-projects-manifestyaml-file)
+  - [Using the pre-built images](#using-the-pre-built-images)
+    - [Using in `devcontainer.json`](#using-in-devcontainerjson)
+    - [Custom devcontainers](#custom-devcontainers)
+  - [Build caching with `sccache`](#build-caching-with-sccache)
+    - [Build caching with private S3 buckets](#build-caching-with-private-s3-buckets)
+    - [Using GitHub OAuth to issue S3 credentials via Hashicorp Vault](#using-github-oauth-to-issue-s3-credentials-via-hashicorp-vault)
+
+## Quick start
+
+If you really want to get started right away and ignore all the details of how
+devcontainers work, the easiest way is to treat the devcontainer
+like a docker container that you use interactively. Brad Dice wrote [a script
+that wraps the devcontainer
+CLI](https://gist.github.com/bdice/a92d224b3e3b1b387fc18b8095b3bdbd) to do this.
+
+To obtain this script:
+```
+curl -LO https://gist.githubusercontent.com/bdice/a92d224b3e3b1b387fc18b8095b3bdbd/raw/28eb8edc856ae04d4cd83571fea5b894f714f01c/rapids-dev
+chmod +x rapids-dev
+./rapids-dev
+```
+
+This script expects you to have your current directory set to the
+root of a repo that has a .devcontainer folder. You may want to move
+this script to a folder that you place on PATH, such as `~/bin`
+
+Running that command will build the devcontainer and drop you at
+an interactive prompt. You can immediately build your project
+with one of the devcontainers scripts. Type `build-` and hit `<TAB>` to see your options.
+
+Skip ahead to [Available tools in the devcontainer](#available-tools-in-the-devcontainer) to see more options
+for interaction at this prompt.
+
+## Detailed start
+
+So, you have cloned a repo that you've heard has a devcontainer. You can see the file(s) for yourself by looking
+in your repo's `.devcontainer` folder. You may find a `devcontainer.json` file, or you may find some number of folders, such as `cuda12.0-conda` and `cuda12.0-pip`. If you find folders, these each contain a `devcontainer.json`
+file. These files specify how to create and run a container that leaves you with a good setup to do development.
+
+There are at least 2 ways to consume these devcontainer.json files. VS Code was
+the original home of devcontainers prior to becoming an open specification, and
+it remains a good way to use them. If you prefer not to use VS Code, the
+[devcontainers-cli](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli)
+application will build the devcontainer and allow you to interact with it.
+
+### Using devcontainers in VS Code
+
+[The VS Code
+docs](https://code.visualstudio.com/docs/devcontainers/containers#_quick-start-open-an-existing-folder-in-a-container)
+are the definitive source of information for this topic.
+
+Specifically for RAPIDS repos, we frequently have multiple folders for different
+library configurations. Pay attention to which build environment you need when
+launching your devcontainer. You can switch between them by reopening your
+native host (e.g. CMD+SHIFT+P -> "Reopen folder in SSH"), and then re-opening in
+devcontainer (CMD+SHIFT+P -> "Reopen in container"), at which point you'll see a
+prompt to choose a different `devcontainer.json`.
+
+### Using devcontainers from the terminal
+
+The [devcontainer-cli](https://code.visualstudio.com/docs/devcontainers/devcontainer-cli) project
+allows you to run and interact with devcontainers from the terminal. It uses NodeJS, so you need
+to install that in order to run it.
+
+There are wrappers to facilitate working with this CLI. Refer back to the
+[Quick-start section](#quick-start). If you need more flexibility, you can call
+the devcontainer CLI directly.
+
+You need to specify the `devcontainer.json` file and workspace folder to use when starting the devcontainer. Also note that you must manage the 3 steps of the lifecycle yourself:
+
+* bring up the devcontainer
+* run your command (or run bash for interactive prompt)
+* stop and remove the docker container
+
+```
+CONTAINER_JSON=.devcontainer/cuda12.0-pip/devcontainer.json
+devcontainer up --config ${CONTAINER_JSON} --workspace-folder=$(pwd)
+devcontainer exec --config ${CONTAINER_JSON} --workspace-folder=$(pwd) bash
+```
+
+Stopping and removing the docker container is manual. The devcontainer CLI does
+not currently facilitate this in any way. One possible workflow (copied from [aforementioned wrapper](https://gist.github.com/bdice/a92d224b3e3b1b387fc18b8095b3bdbd)):
+
+```
+CONTAINER_ID=$(docker ps --quiet \
+      --filter label=devcontainer.local_folder=$(pwd) \
+      --filter label=devcontainer.config_file=${CONTAINER_JSON})
+num_active_shells=$(docker exec "${container_id}" ps aux | grep -c "/bin/sh")
+if [[ ${num_active_shells} -le 1 ]]; then
+    echo "All devcontainers are closed. Stopping and removing container ${container_id}."
+    docker stop "${container_id}"
+    docker rm "${container_id}"
+fi
+```
+
+### Generated build scripts
+
+Several scripts are generated for you based on the contents of `manifest.yaml`. By
+default, `manifest.yaml` comes from [the RAPIDS/devcontainers
+repo](https://github.com/rapidsai/devcontainers/blob/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml),
+but you can use your own local `manifest.yaml` file. Refer to the "Adding
+projects" section.
+
+The generated scripts are:
+* `/usr/bin/build-*`
+* `/usr/bin/clean-*`
+* `/usr/bin/clone-*`
+* `/usr/bin/configure-*`
+
+Here the `*` is a placeholder for the project name and the kind of
+build. For example, a project with a python component in `manifest.yaml`
+will have `build-cudf-python`.
+
+These scripts may trigger side-effects. For example, `build-*` scripts are only
+generated for projects that exist in the workspace. If you are working on `cudf`,
+which depends on `rmm`, the default workspace only mounts `cudf` and generates
+`build-cudf` scripts. If you want `rmm` build scripts also, you can run `clone-rmm`,
+which will clone `rmm` into your workspace and generate build scripts for it.
+
+### rapids-build-utils
+
+These are meta-build scripts. They assist with setting up the workspace and reloading things when important configuration
+changes happen. You mostly won't need to worry about these,
+but it's good to be aware of them. They come from the [rapidsai/devcontainers repo](https://github.com/rapidsai/devcontainers/tree/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/bin)
+
+### Native build tools - CMake, python builds
+
+The generated scripts mentioned above will take care of running
+build tools for you. However, if you need to run the build tools
+manually, you can `cd` into your source code folder, which is
+mounted as a subfolder in `/home/coder`.
+
+### Exiting the devcontainer
 
-* [Using the pre-built images](#using-the-pre-built-images)
-  * [Using in `devcontainer.json`](#using-in-devcontainerjson)
-* [Build caching with sccache](#build-caching-with-sccache)
+If you are in VS Code and you need to return to your host machine (local or SSH),
+you can run `Dev Containers: Reopen in SSH`.
 
+## Adding projects: `manifest.yaml` file
+
+The build script generation is controlled with a `manifest.yaml` file, which by default comes from [the rapidsai/devcontainers
+repo](https://github.com/rapidsai/devcontainers/blob/branch-24.02/features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml)
+
+If you would like to add your project, you can submit a PR to the rapidsai/devcontainers repo. Before you do that, though, you can
+test it locally. Start by copying `manifest.yaml` from the rapidsai/devcontainers repo. You can put it anywhere, but let's say we put it in .devcontainer/manifest.yaml.
+
+Now open a devcontainer.json file that you want to work with. These
+will likely live in a .devcontainer subfolder, such as cuda12.0-pip. In this file, add a top-level key with this:
+
+```
+ "containerEnv": {
+   "PROJECT_MANIFEST_YML": "${localWorkspaceFolder}/.devcontainer/manifest.yaml"
+ },
+```
+
+Rebuild or re-open your devcontainer, and you should now see updated
+generated scripts.
 
 ## Using the pre-built images
 
-We publish a [matrix](matrix.yml) of pre-built images to DockerHub to accelerate initializing local devcontainers, GitHub Codespaces, and CI jobs.
+The choice of using a pre-built image refers to the build/args/BASE entry in `devcontainer.json`. The pre-built
+images are not meant to be used directly. The rapids-build-utils scripts are installed with a "feature," so
+they won't be present if you directly run a pre-built image with Docker instead of with a devcontainer tool (VS Code or devcontainer-cli)
+
+We publish a [matrix](matrix.yml) of pre-built images to DockerHub to accelerate initializing local devcontainers, GitHub Codespaces, and CI jobs. These use the "feature" scripts to install their components,
+so you can think of them as caching those steps.
 
 The features that comprise the image are noted in the image tags. If no version is defined for a tool or SDK, the image includes the latest available version at image build time.
 
 > **NOTE:** `git`, `git-lfs`, `github-cli`, `gitlab-cli`, `cmake`, `ninja`, `sccache`, and our devcontainer-utils [are included](image/.devcontainer/devcontainer.json#L12-L33) in each pre-built image.
 
+See the list of `rapidsai/devcontainers` tags [on DockerHub](https://hub.docker.com/r/rapidsai/devcontainers/tags).
+
 ### Using in [`devcontainer.json`](https://containers.dev/implementors/json_reference/#image-specific)
 
 The pre-built images can be used as the `"image"`, or as the base of a Dockerfile in `"build"`, in `devcontainer.json`:
 
 <details><summary>devcontainer.json using pre-built image</summary><pre>{<br/>  "image": "rapidsai/devcontainers:24.02-cpp-llvm16-cuda12.0-nvhpc23.5-ubuntu22.04",<br/>  "hostRequirements": { "gpu": true },<br/>  "workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",<br/>  "workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"<br/>}</pre></details>
 
+### Custom devcontainers
+
 You can also build a custom devcontainer by composing individual features:
 
 <details><summary>devcontainer.json using individual features</summary><pre>{<br/>  "image": "ubuntu:22.04",<br/>  "features": {<br/>    "ghcr.io/rapidsai/devcontainers/features/cmake:24.02": {},<br/>    "ghcr.io/rapidsai/devcontainers/features/ninja:24.02": {},<br/>    "ghcr.io/rapidsai/devcontainers/features/sccache:24.02": {<br/>      "version": "0.5.4"<br/>    }<br/>  },<br/>  "overrideFeatureInstallOrder": [<br/>    "ghcr.io/rapidsai/devcontainers/features/cmake",<br/>    "ghcr.io/rapidsai/devcontainers/features/ninja",<br/>    "ghcr.io/rapidsai/devcontainers/features/sccache"<br/>  ],<br/>  "workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",<br/>  "workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind"<br/>}</pre></details>
 
+You can add libraries or programs on top of a pre-built image using this
+same mechanism. These are not baked into the image, but cached as a docker
+layer.
+
 > **NOTE:** Feature updates published since your most recent image build will invalidate your docker image layer cache, meaning it can take the [devcontainers CLI](https://github.com/devcontainers/cli) longer to initialize containers composed from individual features.
 
 ## Build caching with `sccache`