GrootFS

GrootFS

GrootFS is a tool with a command line interface (CLI) that provides filesystem isolation for containers. Isolated filesystems are also called root filesystems (or rootfss). Each Garden container references one rootfs that is mounted as its root mountpoint.

CLI

The command line interface (CLI) implements the Garden's image plugin binary interface and is used as an image plugin by Garden's volumizer. The CLI consists of the following commands:

• InitStore: used to create a new grootfs store. Required before creating any rootfses.
• DeleteStore: deletes a store created by InitStore.
• GenerateVolumeSizeMetadata: TODO
• Capacity: returns store_size_bytes from the config file
• Create: creates a new rootfs and returns information (json document) about it on the standard out. Garden parses that json object and uses its data when building the container config.json.
• Delete: deletes a rootfs, invoked when Garden destroys a container
• Clean: cleans all unused layers from the cache in the store, invoked when the store size reaches a threshold
• List: lists all the images in the store
• Stats: returns store stats

What is a Root Filesystem (rootfs)?

The root filesystem is an overlay filesystem that is mounted under a directory in the GrootFS store. Every rootfs starts with a base image (a tar file, or an OCI image) that consists of layers that are downloaded during rootfs creation. The base image layers are mounted as lower/upper overlay dirs (and are read-only) while the workdir is created by GrootFS on rootfs creation and is writable. Thus containers can only change their writable workdirs but cannot change the base layers.

As base image layers are read-only for containers, they can be shared across different rootfses via simply mounting the upper/lower dirs into different overlay mounts thus optimising disk usage and not downloading layers that are already downloaded. For example, two containers that have their rootfses based on the same base image (such as ubuntu) would have their own rootfses with their own workdirs, but the upper/lower dirs in the overlay mount would be the same. Furthermore, when the second container is created, the layers from the ubuntu base image will not be downloaded (as they have been downloaded when the first container has been created) which also helps for performance.

Note: GrootFS has been designed to be filesystem agnostic, so you can implement it on top of any linux filesystem. Historically the team has been experimenting with aufs, btrfs, overlayfs, xfs and ext4. For more information on the outcomes of those experiments and insight of why we settled on overlayfs on top of xfs, please read this blog post

Command Details

init-store

GrootFS requires a place to create its rootfses. This is achieved by creating a sparse file and loop mounting an xfs filesystem on it. Using a sparse file means that we can quickly generate a huge filesystem that initially takes very little space on the parent filesystem. Of course this also leads to confusion about how much space is actually used. See understanding grootfs store disk usage for a discussion. Beware that talk of reclaiming space only applies to the xfs filesystem within the sparse file. Once a sparse file expands to accommodate more content, it cannot be shrunk again. To do that you would need to copy the sparse file to a new sparse file, and that is not possible with GrootFS and garden. So if a filesystem containing a GrootFS is full because of the size of the sparse file, there is no easy fix.

When the command grootfs --config <CONFIG_FILE> init-store is run (with store set to an appropriate path in the config file), GrootFS creates the following directory structure. E.g. for store: /tmp/store/unprivileged:

/tmp/store/
├── unprivileged/
│   ├── images/
│   ├── l/
│   ├── locks/
│   ├── meta/
│   │   ├── dependencies/
│   │   └── namespace.json
│   ├── projectids/
│   ├── tmp/
│   ├── volumes/
│   └── whiteout_dev
└── unprivileged.backing-store

The unprivileged.backing-store file is the sparse file containing the xfs filesystem. The xfs filesystem it contains was mounted by GrootFS at /tmp/store/unprivileged.

The subdirectories are used as follows:

• images: contains mountpoints of rootfses created by GrootFS
• l: shortname symlinks to volumes
• locks: contains lock files
• meta: contains details about uid/gid mappings and volume sizes
• projectids: used by xfs to manage quotas
• tmp: temporary storage
• volumes: where the volumes (image layers) are stored

Existing Backing Stores

It is possible to Init a store in a location where the backing-store file already exists. If it has an xfs filesystem already in it, that will be mounted. If not, the filesystem will be created first.

Direct IO

The GrootFS store is a loop mounted file. Reads and writes to the xfs filesystem inside the store will be cached by the xfs filesystem. Also the backing file's reads and writes will be cached inside its filesystem. Better performance can be obtained by setting the direct-io flag on the loop device used for the backing file mount. This avoids caching on the loop device, saving memory, and stops implicit syncing to the disk.

You can choose to enable direct IO as an argument to the init-store call.