Skip to content

Commit

Permalink
Fix building of wheels in monorepo and update docs
Browse files Browse the repository at this point in the history
The main issue is that our previous hack to detect the component name
from the folder no longer works, since it'll now be an unprefixed
"client", so have the Makefile add the prefixing in.

Documentation has been updated as well.
  • Loading branch information
legoktm committed Feb 13, 2024
1 parent 639161e commit 76d35a4
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 20 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ requirements: ## Creates requirements files for the Python projects

.PHONY: build-wheels
build-wheels: ## Builds the wheels and adds them to the wheels subdirectory
./scripts/verify-sha256sum-signature $$(basename ${PKG_DIR})
./scripts/verify-sha256sum-signature securedrop-$$(basename ${PKG_DIR})
./scripts/build-sync-wheels
./scripts/sync-sha256sums $$(basename ${PKG_DIR})
./scripts/sync-sha256sums securedrop-$$(basename ${PKG_DIR})
@echo Done!

.PHONY: test
Expand Down
46 changes: 28 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@

## Updating our bootstrapped build tools

We use [build](https://pypa-build.readthedocs.io/en/latest/) toolchain to build our reproducible wheels.
If we have to update the tool, use the following steps
We use the [build](https://pypa-build.readthedocs.io/en/latest/) toolchain to build our reproducible wheels.
If we have to update the tools, use the following steps

```shell
# Ensure you are running in a cleanly boostrapped virtual environment
Expand Down Expand Up @@ -37,35 +37,31 @@ Make sure that your GPG public key is stored in `pubkeys/`, so CI can verify the

## Updating Python wheels

Maintainers of `securedrop-client` and `securedrop-proxy` must ensure that
the requirements files which are used for build of these packages (`build-requirements.txt`)
using `make requirements` are kept up to date in latest `main` of those repositories.

If new dependencies were added in the `build-requirements.txt` of that
repo that are not in the `wheels` subdirectory for the package in this repository,
then the maintainer needs to do the following (we are taking `securedrop-client` project
as an example):
When adding a new production dependency to a component, new wheels will need to be built
plus updates to `build-requirements.txt`. This should be done after you have updated the
dependencies in the component's `pyproject.toml` and `poetry.lock` files.

### 0. Enable the virtualenv

You can create a fresh virtualenv and install the build tools from our bootstrapped wheels.
Create a fresh virtualenv and install the build tools from our bootstrapped wheels.

```shell
rm -rf .venv
make install-deps
```

Remember that the following steps needs to be done from the same virtual environment.
The following steps needs to be done from the same virtual environment.

### 1. Create updated build-requirements.txt for the project
### 1. Try to update build-requirements.txt for the project

From the `securedrop-builder` directory,
From the `securedrop-builder` directory, run the following, where `<component>`
is what you're trying to update dependencies for, e.g. "client", "proxy", etc.

```shell
PKG_DIR=/home/user/code/securedrop-client make requirements
PKG_DIR=/home/user/code/securedrop-client/<component> make requirements
```

This will create the proper `build-requirements.txt` file in the project directory along with the binary wheel
This will create/update the `build-requirements.txt` file in the project directory along with the binary wheel
hashes from our own Python package index server.

If we are missing any wheels from our cache/build/server, it will let you know with a following message.
Expand Down Expand Up @@ -93,7 +89,10 @@ PKG_DIR=/home/user/code/securedrop-client make build-wheels

This above command will let you know about any new wheels + sources. It will
build/download sources from PyPI (by verifying it against the sha256sums from
the `requirements.txt` of the project).
the `poetry.lock` of the project).

If your package contains compiled code (e.g. C or Rust extensions), it must be
built for all Debian versions we support.

### 3. Commit changes to the wheels directory (if only any update of wheels)

Expand All @@ -105,4 +104,15 @@ git add wheels/
git commit
```

Finally, submit a PR containing the new wheels and updated files.
Submit a PR containing the new wheels and updated files.

### 4. Update build-requirements.txt

After building and committing the new wheels, re-run the command from step 1:

```shell
PKG_DIR=/home/user/code/securedrop-client/<component> make requirements
```

This will update the build-requirements.txt file, commit and open a PR with these
changes. Note that CI will likely fail until the PR from step 3 is merged.

0 comments on commit 76d35a4

Please sign in to comment.