-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace pinned builds with reproducible pinned builds #1686
Changes from 1 commit
dda0ff8
ffb49a0
cef0a4e
d015c62
2930e56
a51ae8c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -93,32 +93,23 @@ git submodule update --init --recursive | |||
Select build instructions below for a [pinned build](#pinned-build) (preferred) or an [unpinned build](#unpinned-build). | ||||
|
||||
> ℹ️ **Pinned vs. Unpinned Build** ℹ️ | ||||
We have two types of builds for Leap: "pinned" and "unpinned." The only difference is that pinned builds use specific versions for some dependencies hand-picked by the Leap engineers - they are "pinned" to those versions. In contrast, unpinned builds use the default dependency versions available on the build system at the time. We recommend performing a "pinned" build to ensure the compiler remains the same between builds of different Leap versions. Leap requires these versions to remain the same, otherwise its state might need to be recovered from a portable snapshot or the chain needs to be replayed. | ||||
We have two types of builds for Leap: "pinned" and "unpinned." A pinned build is a reproducible build with the build environment and dependency versions fixed by the development team. In contrast, unpinned builds use the dependency versions provided by the build platform. Unpinned builds tend to be quicker because the pinned build environment must be built from scratch. Pinned builds, in addition to being reproducible, ensure the compiler remains the same between builds of different Leap major versions. Leap requires the compiler version to remain the same, otherwise its state might need to be recovered from a portable snapshot or the chain needs to be replayed. | ||||
|
||||
> ⚠️ **A Warning On Parallel Compilation Jobs (`-j` flag)** ⚠️ | ||||
When building C/C++ software, often the build is performed in parallel via a command such as `make -j "$(nproc)"` which uses all available CPU threads. However, be aware that some compilation units (`*.cpp` files) in Leap will consume nearly 4GB of memory. Failures due to memory exhaustion will typically, but not always, manifest as compiler crashes. Using all available CPU threads may also prevent you from doing other things on your computer during compilation. For these reasons, consider reducing this value. | ||||
|
||||
> 🐋 **Docker and `sudo`** 🐋 | ||||
If you are in an Ubuntu docker container, omit `sudo` from all commands because you run as `root` by default. Most other docker containers also exclude `sudo`, especially Debian-family containers. If your shell prompt is a hash tag (`#`), omit `sudo`. | ||||
|
||||
#### Pinned Build | ||||
Make sure you are in the root of the `leap` repo, then run the `install_depts.sh` script to install dependencies: | ||||
#### Pinned Reproducible Build | ||||
The pinned reproducible build requires Docker. Make sure you are in the root of the `leap` repo and then run | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Somewhere do you want to mention any docker on X86 architecture would work for production builds (JIT and OC)? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually.. that's curious.. what happens if you try to run this on ARM or such 🤔 There is actually nothing x86 specific with the possible exception that the debian root, leap/tools/reproducible.Dockerfile Line 3 in a51ae8c
being called out via hash might limit this to x86. That might be a compelling reason to use a tag name instead such as debian:buster-20230919 , though using a tag name is more at risk of a malicious switch-a-roo if debian's docker hub is compromised in some way in the future.
But, existing pinned builds did not support anything but x86, so I'm not sure we need to change any documentation around this limitation strictly due to this PR (maybe some other misc README clean up..) |
||||
```bash | ||||
sudo scripts/install_deps.sh | ||||
DOCKER_BUILDKIT=1 docker build -f tools/Dockerfile.reproducible -o . . | ||||
``` | ||||
|
||||
Next, run the pinned build script. You have to give it three arguments in the following order: | ||||
1. A temporary folder, for all dependencies that need to be built from source. | ||||
1. A build folder, where the binaries you need to install will be built to. | ||||
1. The number of jobs or CPU cores/threads to use (note the [jobs flag](#step-3---build) warning above). | ||||
|
||||
> 🔒 You do not need to run this script with `sudo` or as root. | ||||
|
||||
For example, the following command runs the `pinned_build.sh` script, specifies a `deps` and `build` folder in the root of the Leap repo for the first two arguments, then builds the packages using all of your computer's CPU threads: | ||||
This command will take a substantial amount of time because a toolchain is built from scratch. Upon completion, the current directory will contain a built `.deb` and `.tar.gz` (you can change the `-o .` argument to place the output in a different directory). If needing to reduce the number of parallel jobs as warned above, run the command as, | ||||
```bash | ||||
scripts/pinned_build.sh deps build "$(nproc)" | ||||
DOCKER_BUILDKIT=1 docker build --build-arg LEAP_BUILD_JOBS=4 -f tools/Dockerfile.reproducible -o . . | ||||
``` | ||||
Now you can optionally [test](#step-4---test) your build, or [install](#step-5---install) the `*.deb` binary packages, which will be in the root of your build directory. | ||||
|
||||
#### Unpinned Build | ||||
The following instructions are valid for this branch. Other release branches may have different requirements, so ensure you follow the directions in the branch or release you intend to build. If you are in an Ubuntu docker container, omit `sudo` because you run as `root` by default. | ||||
|
This file was deleted.
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you explain
reproducible
in more details? Was the build by old pinned build script not reproducible?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, previous build was not reproducible due to, for example, different paths used during the build (some of these paths get baked in to the final executable), and different time stamps used inside of the
.deb
package. Fixing these doesn't strictly require building inside of a container (for example maybe we could useffile-prefix-map
to fix up the paths in different installations). But doing it in a container makes it easier.(there are other issues too, like the fixup
tweak-deb.sh
does onInstalled-Size
. and maybe some other nuances I'm not recalling quickly off hand)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your explanation.