Support macOS (formerly: unable to build bottlerocket on macos)

[webern]

          Sure, here's the log. Not sure where the error comes from, AFAICS everything should be built in docker and hence machine-agnostic? Not sure though where the local rust part comes in...

  bottlerocket-os/bottlerocket#21 236.6 error: could not compile `sundog`
  bottlerocket-os/bottlerocket#21 236.6 
  bottlerocket-os/bottlerocket#21 236.6 Caused by:
  bottlerocket-os/bottlerocket#21 236.6   process didn't exit successfully: `/usr/bin/rustc --crate-name sundog --edition=2018 api/sundog/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C metadata=f1dda1c0b15e0c90 -C extra-filename=-f1dda1c0b15e0c90 --out-dir /home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps --target x86_64-bottlerocket-linux-gnu -C linker=/usr/bin/x86_64-bottlerocket-linux-gnu-gcc -L dependency=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps -L dependency=/home/builder/.cache/release/deps --extern apiclient=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libapiclient-cee067d451d7b166.rlib --extern constants=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libconstants-37ede6ace73bb42f.rlib --extern datastore=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libdatastore-dbb12debb9ff125d.rlib --extern http=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libhttp-68066c2bc6086886.rlib --extern log=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/liblog-56fb0fccdab81be0.rlib --extern model=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libmodel-0c570bc5f68d5a14.rlib --extern serde=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde-0fe5639f2e2024f0.rlib --extern serde_json=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde_json-43d9398d57cbcc08.rlib --extern simplelog=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsimplelog-192c7c07a4f82e7a.rlib --extern snafu=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsnafu-8194cd9e9fb48158.rlib --extern tokio=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libtokio-0c8ac197f1b09636.rlib -Cprefer-dynamic -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Clink-arg=-Wl,-z,relro,-z,now -L native=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/build/ring-e73751118ec72435/out` (signal: 9, SIGKILL: kill)
  bottlerocket-os/bottlerocket#21 236.6 warning: build failed, waiting for other jobs to finish...
  bottlerocket-os/bottlerocket#21 372.4 error: Bad exit status from /var/tmp/rpm-tmp.t16z5j (%build)
  bottlerocket-os/bottlerocket#21 372.4 
  bottlerocket-os/bottlerocket#21 372.4 RPM build errors:
  bottlerocket-os/bottlerocket#21 372.4     Bad exit status from /var/tmp/rpm-tmp.t16z5j (%build)
  bottlerocket-os/bottlerocket#21 ERROR: executor failed running [/bin/sh -c rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/${PACKAGE}.spec]: exit code: 1
  ------
   > [rpmbuild 7/7] RUN --mount=source=.cargo,target=/home/builder/.cargo     --mount=type=cache,target=/home/builder/.cache,from=cache,source=/cache     --mount=type=cache,target=/home/builder/rpmbuild/BUILD/sources/models/src/variant,from=variantcache,source=/variantcache     --mount=type=cache,target=/home/builder/rpmbuild/BUILD/sources/logdog/conf/current,from=variantcache,source=/variantcache     --mount=source=sources,target=/home/builder/rpmbuild/BUILD/sources     rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/os.spec:
  ------
  executor failed running [/bin/sh -c rpmbuild -ba --clean       --undefine _auto_set_build_flags       rpmbuild/SPECS/${PACKAGE}.spec]: exit code: 1


  --- stderr
  BuildAttempt: Failed to execute command: 'docker build . --network none --target package --tag buildsys-pkg-os-x86_64-d4ae984f528b --build-arg PACKAGE=os --build-arg ARCH=x86_64 --build-arg GOARCH=amd64 --build-arg VARIANT=aws-k8s-1.24 --build-arg VARIANT_PLATFORM=aws --build-arg VARIANT_RUNTIME=k8s --build-arg VARIANT_FAMILY=aws-k8s --build-arg VARIANT_FLAVOR= --build-arg REPO=default --build-arg SDK=public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.29.0 --build-arg TOOLCHAIN=public.ecr.aws/bottlerocket/bottlerocket-toolchain-x86_64:v0.29.0 --build-arg NOCACHE=3534030891 --build-arg TOKEN=d4ae984f528b'
[cargo-make] ERROR - Error while executing command, exit code: 101
[cargo-make] WARN - Build Failed.

Originally posted by @pat-s in bottlerocket-os/bottlerocket#2685 (comment)

[webern]

Arnaldo's question: bottlerocket-os/bottlerocket#2685 (comment)

Hey, @pat-s I read the logs and this is interesting (I formatted a little the output to make it more readable)

`/usr/bin/rustc --crate-name sundog --edition=2018 \
api/sundog/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat \
  --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 \
  -C metadata=f1dda1c0b15e0c90 -C extra-filename=-f1dda1c0b15e0c90 \
  --out-dir /home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps --target x86_64-bottlerocket-linux-gnu \
  -C linker=/usr/bin/x86_64-bottlerocket-linux-gnu-gcc \
  -L dependency=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps \
  -L dependency=/home/builder/.cache/release/deps \
  --extern apiclient=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libapiclient-cee067d451d7b166.rlib \
  --extern constants=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libconstants-37ede6ace73bb42f.rlib \
  --extern datastore=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libdatastore-dbb12debb9ff125d.rlib \
  --extern http=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libhttp-68066c2bc6086886.rlib \
  --extern log=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/liblog-56fb0fccdab81be0.rlib \
  --extern model=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libmodel-0c570bc5f68d5a14.rlib \
  --extern serde=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde-0fe5639f2e2024f0.rlib \
  --extern serde_json=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libserde_json-43d9398d57cbcc08.rlib \
  --extern simplelog=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsimplelog-192c7c07a4f82e7a.rlib \
  --extern snafu=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libsnafu-8194cd9e9fb48158.rlib \
  --extern tokio=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/deps/libtokio-0c8ac197f1b09636.rlib \
  -Cprefer-dynamic -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Clink-arg=-Wl,-z,relro,-z,now \
  -L native=/home/builder/.cache/x86_64-bottlerocket-linux-gnu/release/build/ring-e73751118ec72435/out` (signal: 9, SIGKILL: kill)

The process was killed, probably because of limits on the VM that docker starts in MAC, @webern, I remember you had problems with docker in MAC, did you ever experienced something like this?

[webern]

When I had problems with building Bottlerocket on macOS, it was a long time ago, on x86_64. At that time it was a few simple bash commands that weren't available on the default bash version. (Also it was painfully slow due to filesystem virtualization).

This issue appears to be different. As far as I can tell, all we have to go on is (signal: 9, SIGKILL: kill), which isn't much.

[mikn]

I would bump this as a colleague of mine tested building twoliter from scratch, including a set of variants and it did work.
Our polyglot toolchain management would be very happy if there was also a OSX ARM64 binary release of Twoliter so we do not have to compile it from scratch for our poor Darwin users. ❤️

[mikn]

Another alternative that would work well for us would be to update the published crate: https://crates.io/crates/twoliter

[cbgbt]

Hey @mikn, thanks for the feedback!

I'm a little surprised that twoliter builds on MacOS succeeded. We have a somewhat odd Linux-specific performance optimization that works around limitations in docker/BuildKit.

As a quick explanation of that -- we use BuildKit to isolate and drive builds that twoliter performs. Docker is somewhat opinionated on how it isolates its build context -- the most common way that the build context is populated is by copying all of the files in the build context and sending them to the docker daemon. This is a pretty big time expenditure as the number and size of files in the build context grows.

To get around copying everything, pipesys uses two perhaps more-niche Linux/Unix features:

• Abstract Unix domain sockets -- Unix domain sockets are typically "namespaced" on the filesystem, ie each socket gets a file. On Linux, an abstract Unix domain socket instead lives in the "abstract" namespace within the kernel. Very useful if your processes are filesystem isolated but still live within the same network namespace. Unfortunately though, I don't believe MacOS supports this feature
• Sending file descriptors over the unix domain socket: Once we have our "abstract" socket, we share references to build contents by simply sending a file descriptor for the build directory into the docker build environment (with new solutions introduced to keep the contents immutable and otherwise isolate builds from the network). I believe this works on MacOS, just adding the context to understand why we need the abstract sockets.

While twoliter itself should compile for MacOS, the pipesys server utility which handles this optimization is conditionally compiled and will panic on execution.

We chatted out-of-band about this a bit, but perhaps there's Linux VMs involved in the mix on your end somewhere?

Despite that, I'm interested in publishing twoliter to crates.io. I cut #419 to track that!

[bittermandel]

Hi @cbgbt 👋🏻 I'm the famous colleague.

And you are correct, I'm running everything from within a Colima VM which runs Ubuntu as I could not get it to run properly directly on MacOS. I switched over to building from an x86_64 machine directly instead as my desktop is much stronger and that is what we will run on in the end, but I'd be happy to provide whatever information you might need if you decide to add support for MacOS in the end 🙂

I got stuck at the following error, it happens for all the packages.

  --- stderr
  thread 'main' panicked at tools/buildsys/src/builder.rs:99:10:
  Failed to obtain current UID: Os { code: 2, kind: NotFound, message: "No such file or directory" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: failed to run custom build command for `kata-containers-static v0.1.0 (/Users/jonathangrahl/projects/molnett/src/platform/base/bottlerocket/packages/kata-containers-static)`

Caused by:
  process didn't exit successfully: `/Users/jonathangrahl/projects/molnett/src/platform/base/bottlerocket/target/x86_64/debug/build/kata-containers-static-fd5dc308adaef1c6/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=BUILDSYS_ARCH
  cargo:rerun-if-env-changed=BUILDSYS_EXTERNAL_KITS_DIR
  cargo:rerun-if-env-changed=BUILDSYS_OUTPUT_GENERATION_ID
  cargo:rerun-if-env-changed=BUILDSYS_PACKAGES_DIR
  cargo:rerun-if-env-changed=BUILDSYS_ROOT_DIR
  cargo:rerun-if-env-changed=BUILDSYS_STATE_DIR
  cargo:rerun-if-env-changed=TLPRIVATE_SDK_IMAGE
  cargo:rerun-if-changed=Cargo.toml
  cargo:rerun-if-changed=/Users/jonathangrahl/projects/molnett/src/platform/base/bottlerocket/build/external-kits/external-kit-metadata.json
  cargo:rerun-if-changed=kata-containers-static.spec