-
Notifications
You must be signed in to change notification settings - Fork 198
Self hosting outside the Vagrant VM
While the Vagrant VM allows for easier setup of the required dependencies and environment, here is a breakdown of what it takes to turn a regular server into its own version of docs.rs.
Beware: This process is rather rough! Attempts at cleaning it up, automating setup components, etc, would be greatly appreciated!
The commands and package names on this page will assume an Ubuntu server running systemd, but hopefully the explanatory text should give enough information to adapt to other systems.
Docs.rs has a few basic requirements:
- Rust (preferably via
rustup) - Git
- CMake, GCC, G++, and
pkg-config(to build dependencies for crates and docs.rs itself) - OpenSSL, zlib, curl, and
libmagic(to link against) - PostgreSQL
- LXC tools (doc builds run inside an LXC container)
$ curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain nightly
$ source $HOME/.cargo/env
# apt install build-essential git curl cmake gcc g++ pkg-config libmagic-dev libssl-dev zlib1g-dev postgresql lxc-utilsTo help things out later on, we can create a new unprivileged user that will run the server process. This user will own all the files required by the docs.rs process. This user will need to be able to run lxc-attach through sudo to be able to run docs builds, so give it a sudoers file at the same time:
# adduser --disabled-login --disabled-password --gecos "" cratesfyi
# echo 'cratesfyi ALL=(ALL) NOPASSWD: /usr/bin/lxc-attach' > /etc/sudoers.d/cratesfyi(The name cratesfyi is a historical one: Before the site was called "docs.rs", it was called "crates.fyi" instead. If you want to update the name of the user, feel free! Just be aware that the name cratesfyi will be used throughout this document.)
In addition to the LXC container, docs.rs also stores several related files in a "prefix" directory. This directory can be stored anywhere, but the cratesfyi user needs to be able to access it:
# mkdir /cratesfyi-prefix
# chown cratesfyi:cratesfyi /cratesfyi-prefixNow we can set up some required folders. To make sure they all have proper ownership, run them all as cratesfyi:
$ sudo -u cratesfyi mkdir -vp /cratesfyi-prefix/documentations /cratesfyi-prefix/public_html /cratesfyi-prefix/sources
$ sudo -u cratesfyi git clone https://github.com/rust-lang/crates.io-index.git /cratesfyi-prefix/crates.io-index
$ sudo -u cratesfyi git --git-dir=/cratesfyi-prefix/crates.io-index/.git branch crates-index-diff_last-seen(That last command is used to set up the crates-index-diff crate, so we can start monitoring new crate releases.)
To help contain what crates' build scripts can access, documentation builds run inside an LXC container. To create one inside the prefix directory:
# LANG=C lxc-create -n cratesfyi-container -P /cratesfyi-prefix -t download -- --dist ubuntu --release bionic --arch amd64
# ln -s /cratesfyi-prefix/cratesfyi-container /var/lib/lxc
# chmod 755 /cratesfyi-prefix/cratesfyi-container
# chmod 755 /var/lib/lxc(To make deployment simpler, it's important that the OS the container is using is the same as the host! In this case, the host is assumed to be running 64-bit Ubuntu 18.04. If you make the container use a different release or distribution, you'll need to build docs.rs separately inside the container when deploying.)
You'll also need to configure networking for the container. The following is a sample /etc/default/lxc-net that enables NAT networking for the container:
USE_LXC_BRIDGE="true"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE=""
LXC_DOMAIN=""
In addition, you'll need to set the container's configuration to use this. Add the following lines to /cratesfyi-prefix/cratesfyi-container/config:
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
Now you can reload the LXC network configuration, start up the container, and set it up to auto-start when the host boots:
# systemctl restart lxc-net
# systemctl enable [email protected]
# systemctl start [email protected]Now we need to do some setup inside this container. You can either copy all these commands so that each one attaches on its own, or you can run lxc-console -n cratesfyi-container to open a root shell inside the container and skip the lxc-attach prefix.
# lxc-attach -n cratesfyi-container -- apt update
# lxc-attach -n cratesfyi-container -- apt upgrade
# lxc-attach -n cratesfyi-container -- apt install curl ca-certificates binutils gcc libc6-dev libmagic1 pkg-config build-essentialInside the container, we also need to set up a cratesfyi user, and install Rust for it. In addition to the base Rust installation, we also need to install all the default targets so that we can build docs for all the Tier 1 platforms. The Rust compiler installed inside the container is the one that builds all the docs, so if you want to use a new Rustdoc feature, this is the compiler to update.
lxc-attach -n cratesfyi-container -- adduser --disabled-login --disabled-password --gecos "" cratesfyi
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain nightly'
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup target add i686-apple-darwin'
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup target add i686-pc-windows-msvc'
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup target add i686-unknown-linux-gnu'
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup target add x86_64-apple-darwin'
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup target add x86_64-pc-windows-msvc'Now that we have Rust installed inside the container, we can use a trick to give the cratesfyi user on the host the same Rust compiler as the container. By symlinking the following directories into its user directory, we don't need to track a third toolchain.
for directory in .cargo .rustup .multirust; do [[ -h /home/cratesfyi/$directory ]] || sudo -u cratesfyi ln -vs /var/lib/lxc/cratesfyi-container/rootfs/home/cratesfyi/$directory /home/cratesfyi/; doneTo ensure that the docs.rs server is configured properly, we need to set a few environment variables. The primary ones are going into a separate environment file, so we can load them into the systemd service that will manage the server.
Write the following into /home/cratesfyi/.cratesfyi.env. If you have a GitHub access token that the site can use to collect repository information, add it here, but otherwise leave it blank. The variables need to exist, but they can be blank to skip that collection.
CRATESFYI_PREFIX=/cratesfyi-prefix
CRATESFYI_DATABASE_URL=postgresql://cratesfyi:password@localhost
CRATESFYI_GITHUB_USERNAME=
CRATESFYI_GITHUB_ACCESSTOKEN=
RUST_LOG=cratesfyi
Now add the following to /home/cratesfyi/.profile:
export $(cat $HOME/.cratesfyi.env | xargs -d '\n')
export PATH="$HOME/.cargo/bin:$PATH"
export PATH="$PATH:$HOME/docs.rs/target/release"Now we can actually clone and build the docs.rs source! The location of it doesn't matter much, but again, we want it to be owned by cratesfyi so it can build and run the final executable. In addition, we copy the built cratesfyi binary into the container so that it can be used to arrange builds on the inside.
sudo -u cratesfyi git clone https://github.com/rust-lang-nursery/docs.rs.git ~cratesfyi/docs.rs
sudo su - cratesfyi -c 'cd ~/docs.rs && cargo build --release'
cp -v /home/cratesfyi/docs.rs/target/release/cratesfyi /var/lib/lxc/cratesfyi-container/rootfs/usr/local/binNow that we have the repository built, we can use it to set up the database. Docs.rs uses a Postgres database to store information about crates and their documentation. To set one up, we first need to ask Postgres to create the database, and then run the docs.rs command to create the initial tables and content:
sudo -u postgres sh -c "psql -c \"CREATE USER cratesfyi WITH PASSWORD 'password';\""
sudo -u postgres sh -c "psql -c \"CREATE DATABASE cratesfyi OWNER cratesfyi;\""
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- database init"
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build add-essential-files"
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build crate rand 0.5.5"
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- database update-search-index"
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- database update-release-activity"We're almost there! At this point, we've got all the pieces in place to run the site. Now we can set up a systemd service that will run the daemon that will collect crate information, orchestrate builds, and serve the website. The following systemd service file can be placed in /etc/systemd/system/cratesfyi.service:
[Unit]
Description=Cratesfyi daemon
After=network.target postgresql.service
[Service]
User=cratesfyi
Group=cratesfyi
Type=forking
PIDFile=/cratesfyi-prefix/cratesfyi.pid
EnvironmentFile=/home/cratesfyi/.cratesfyi.env
ExecStart=/home/cratesfyi/docs.rs/target/release/cratesfyi daemon
WorkingDirectory=/home/cratesfyi/docs.rs
[Install]
WantedBy=multi-user.target
Enabling and running that will serve the website on http://localhost:3000, so if you want to route public traffic to it, you'll need to set up something like nginx to proxy the connections to it.
If you want to update the Rust compiler used to build crates (and the Rustdoc that comes with it), you need to make sure you don't interrupt any existing crate builds. The daemon waits for 60 seconds between checking for new crates, so you need to make sure you catch it during that window. Since we hooked the daemon into systemd, the logs will be available in its journal. Running journalctl -efu cratesfyi (it may need to be run as root if nothing appears) will show the latest log output and show new entries as they appear. You're looking for a message like "Finished building new crates, going back to sleep" or "Queue is empty, going back to sleep", which indicates that the crate-building thread is waiting.
To prevent the queue from building more crates, run the following:
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build lock"This will create a lock file in the prefix directory that will prevent more crates from being built. At this point, you can update the rustc inside the container and add the rustdoc static files to the database:
lxc-attach -n cratesfyi-container -- su - cratesfyi -c 'rustup update'
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build add-essential-files"Once this is done, you can unlock the queue to allow crates to build again:
sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build unlock"And we're done! New crates will start being built with the new rustc. If you want to rebuild any existing docs with the new rustdoc, you need to manually build them - there's no automated way to rebuild failed docs or docs from a certain rust version yet.
To update the code for docs.rs itself, you can follow a similar approach. First, watch the logs so you can stop the daemon from building more crates. (You can replace the lock command with a systemctl stop cratesfyi if you don't mind the web server being down while you build.)
# journalctl -efu cratesfyi
(wait for build daemon to sleep)
$ sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build lock"Once the daemon has stopped, you can start updating the code and rebuilding:
$ sudo su - cratesfyi -c "cd ~/docs.rs && git pull"
$ sudo su - cratesfyi -c "cd ~/docs.rs && cargo build --release"Now that we have a shiny new build, we need to make sure the service is using it:
# cp -v /home/cratesfyi/docs.rs/target/release/cratesfyi /var/lib/lxc/cratesfyi-container/rootfs/usr/local/bin
# systemctl restart cratesfyiNext, we can unlock the builder so it can start checking new crates:
$ sudo su - cratesfyi -c "cd ~/docs.rs && cargo run --release -- build unlock"And we're done! Changes to the site or the build behavior should be visible now.