Build toolchains on macOS

[axelson]

This is not close to being ready yet, but I wanted to create the PR to begin discussions since Slack history is not preserved.

The original plan (which is currently commented out) was to build all the mac toolchains with the ./build_release.sh script. However that results in the following error (can be viewed in full at https://app.circleci.com/pipelines/github/nerves-project/toolchains/110/workflows/25aba439-358a-4505-8aef-042561fb7fb0/jobs/6376):

==> nerves
cc -c -O2 -Wall -Wextra -Wno-unused-parameter -std=c99 -D_GNU_SOURCE -o /Users/distiller/project/nerves_toolchain_armv7_nerves_linux_gnueabihf/_build/target_dev/lib/nerves/obj/port.o src/port.c
cc /Users/distiller/project/nerves_toolchain_armv7_nerves_linux_gnueabihf/_build/target_dev/lib/nerves/obj/port.o  -o /Users/distiller/project/nerves_toolchain_armv7_nerves_linux_gnueabihf/_build/target_dev/lib/nerves/priv/port
Compiling 41 files (.ex)
Generated nerves app
==> nerves_toolchain_armv7_nerves_linux_gnueabihf
Resolving Nerves artifacts...
  Resolving nerves_toolchain_armv7_nerves_linux_gnueabihf
  => Trying https://github.com/**************/toolchains/releases/download/v1.4.2/nerves_toolchain_armv7_nerves_linux_gnueabihf-mac_arm-1.4.2-6A1DC92.tar.xz
     Status 404 Not Found
  => Trying https://github.com/**************/toolchains/releases/download/v1.4.2/nerves_toolchain_armv7_nerves_linux_gnueabihf-mac_arm-1.4.2-6A1DC92D79541A1165C0CC8A4F0AEA7CFD446A050628F87D9458501D63848B64.tar.xz
     Status 404 Not Found
  => no_result
==> nerves
==> nerves_toolchain_ctng
Compiling 1 file (.ex)
Generated nerves_toolchain_ctng app
Can't find a mac_arm_defconfig fragment. Check that one exists.
** (Protocol.UndefinedError) protocol String.Chars not implemented for %IO.Stream{device: :standard_io, line_or_bytes: :line, raw: true} of type IO.Stream (a struct)
    (elixir 1.10.4) lib/string/chars.ex:3: String.Chars.impl_for!/1
    (elixir 1.10.4) lib/string/chars.ex:22: String.Chars.to_string/1
    lib/nerves/artifact.ex:26: Nerves.Artifact.build/2
    lib/mix/tasks/compile.nerves_package.ex:31: Mix.Tasks.Compile.NervesPackage.run/1
    (mix 1.10.4) lib/mix/task.ex:330: Mix.Task.run_task/3
    (mix 1.10.4) lib/mix/tasks/compile.all.ex:76: Mix.Tasks.Compile.All.run_compiler/2
    (mix 1.10.4) lib/mix/tasks/compile.all.ex:56: Mix.Tasks.Compile.All.do_compile/4
    (mix 1.10.4) lib/mix/tasks/compile.all.ex:27: anonymous fn/2 in Mix.Tasks.Compile.All.run/1

The main error seems to be Can't find a mac_arm_defconfig fragment. Check that one exists., which is correct. But since mac toolchains have been built from this repository before (e.g. nerves_toolchain_aarch64_nerves_linux_gnu-darwin_arm-1.4.2-3D80D46.tar.xz and nerves_toolchain_aarch64_nerves_linux_gnu-darwin_x86_64-1.4.2-3D80D46.tar.xz) something is probably incorrect in the CI setup. But I'm not sure what is wrong in the CI setup since build_release.sh and all-configs.sh do not appear to be configurable (although perhaps specific environment variables are needed).

Also I'm not sure if the 404's are expected when trying to download prebuilt toolchains, but it would make sense to skip those checks since we're trying to build the toolchains.

Any ideas on things to try?

[fhunleth]

Thanks for doing this!!! Don't worry about building the Darwin aarch64 toolchains. I don't think we'll be able to do that unless CircleCI has ARM-based Macs. Building the x86_64 toolchains for OSX seems like a really helpful improvement for now.

[axelson]

Thanks for doing this!!!
You're welcome!

Don't worry about building the Darwin aarch64 toolchains. I don't think we'll be able to do that unless CircleCI has ARM-based Macs. Building the x86_64 toolchains for OSX seems like a really helpful improvement for now.

Hmm, it might be possible:

It is possible to build Apple Silicon/Universal binaries using the Xcode 12.0.0-beta image as Apple provides both the Intel (x86_64) and Apple Silicon (arm64) toolchains in this release. Cross-compiling Apple Silicon binaries on Intel hosts has an additional overhead and as a result compilation times will be longer than native compilation for Intel.
Running or testing Apple Silicon apps natively is currently not possible as CircleCI build hosts are Intel-based Macs. Binaries will need to be exported as artifacts for testing apps locally.
From: https://circleci.com/docs/2.0/testing-ios/#apple-silicon-support

Although I'll focus on the x86_64 toolchains for now.

[fhunleth]

@axelson I bumped the version to trigger an OSX build.

[axelson]

Making progress! Build is now 2 hours in 🎉

Is this error something to be worried about?

result: ["CT_CONFIG_VERSION=\"3\"", "CT_EXPERIMENTAL=y",
 "CT_LOCAL_TARBALLS_DIR=\"${CT_TOP_DIR}/../dl\"",
 "CT_TARBALLS_BUILDROOT_LAYOUT=y", "CT_PREFIX_DIR=\"../x-tools/${CT_TARGET}\"",
 "CT_EXTRA_CFLAGS_FOR_BUILD=\"-I/usr/local/opt/gettext/include\"",
 "CT_EXTRA_LDFLAGS_FOR_BUILD=\"-L/usr/local/opt/gettext/lib -lintl\"",
 "CT_ARCH_X86=y", "CT_ARCH_64=y", "CT_TARGET_VENDOR=\"nerves\"",
 "CT_KERNEL_LINUX=y", "CT_LINUX_V_4_14=y", "CT_LIBC_MUSL=y",
 "# CT_CC_GCC_SJLJ_EXCEPTIONS is not set", "CT_CC_LANG_CXX=y"]
desired: ["CT_EXPERIMENTAL=y", "CT_LOCAL_TARBALLS_DIR=\"${CT_TOP_DIR}/../dl\"",
 "CT_ARCH_X86=y", "CT_ARCH_64=y", "CT_KERNEL_LINUX=y", "CT_LINUX_V_4_14=y",
 "CT_LIBC_MUSL=y", "# CT_CC_GCC_SJLJ_EXCEPTIONS is not set"]
** (EXIT from #PID<0.94.0>) an exception was raised:
    ** (ArgumentError) argument error
        (stdlib 3.13) :ets.lookup(:elixir_config, :no_warn_undefined)
        (elixir 1.10.4) src/elixir_config.erl:8: :elixir_config.get/1
        (elixir 1.10.4) lib/module/checker.ex:95: Module.Checker.merge_no_warn_undefined/1
        (elixir 1.10.4) lib/module/checker.ex:82: Module.Checker.undefined_and_deprecation_warnings/2
        (elixir 1.10.4) lib/module/checker.ex:9: Module.Checker.verify/2
        (elixir 1.10.4) lib/module/parallel_checker.ex:179: anonymous fn/5 in Module.ParallelChecker.spawn_checkers/1

It's not causing the build to stop, and it's a bit weird since it's an error within elixir. It's in the output twice so far.

[axelson]

https://app.circleci.com/pipelines/github/nerves-project/toolchains/127/workflows/939ae590-8ced-4f75-95e9-4cf0209cb549/jobs/6655

hdiutil: create failed - No space left on device

Hmmm, maybe need to increase the disk size somehow. Or free up space on already built toolchains.

[fhunleth]

Is this error something to be worried about?

result: ["CT_CONFIG_VERSION=\"3\"", "CT_EXPERIMENTAL=y",
 "CT_LOCAL_TARBALLS_DIR=\"${CT_TOP_DIR}/../dl\"",
 "CT_TARBALLS_BUILDROOT_LAYOUT=y", "CT_PREFIX_DIR=\"../x-tools/${CT_TARGET}\"",
 "CT_EXTRA_CFLAGS_FOR_BUILD=\"-I/usr/local/opt/gettext/include\"",
 "CT_EXTRA_LDFLAGS_FOR_BUILD=\"-L/usr/local/opt/gettext/lib -lintl\"",
 "CT_ARCH_X86=y", "CT_ARCH_64=y", "CT_TARGET_VENDOR=\"nerves\"",
 "CT_KERNEL_LINUX=y", "CT_LINUX_V_4_14=y", "CT_LIBC_MUSL=y",
 "# CT_CC_GCC_SJLJ_EXCEPTIONS is not set", "CT_CC_LANG_CXX=y"]
desired: ["CT_EXPERIMENTAL=y", "CT_LOCAL_TARBALLS_DIR=\"${CT_TOP_DIR}/../dl\"",
 "CT_ARCH_X86=y", "CT_ARCH_64=y", "CT_KERNEL_LINUX=y", "CT_LINUX_V_4_14=y",
 "CT_LIBC_MUSL=y", "# CT_CC_GCC_SJLJ_EXCEPTIONS is not set"]
** (EXIT from #PID<0.94.0>) an exception was raised:
    ** (ArgumentError) argument error
        (stdlib 3.13) :ets.lookup(:elixir_config, :no_warn_undefined)
        (elixir 1.10.4) src/elixir_config.erl:8: :elixir_config.get/1
        (elixir 1.10.4) lib/module/checker.ex:95: Module.Checker.merge_no_warn_undefined/1
        (elixir 1.10.4) lib/module/checker.ex:82: Module.Checker.undefined_and_deprecation_warnings/2
        (elixir 1.10.4) lib/module/checker.ex:9: Module.Checker.verify/2
        (elixir 1.10.4) lib/module/parallel_checker.ex:179: anonymous fn/5 in Module.ParallelChecker.spawn_checkers/1

It's not causing the build to stop, and it's a bit weird since it's an error within elixir. It's in the output twice so far.

I've been ignoring this error. It means that there might be an inconsistency between the configuration is processed and then resaved. I had thought that it had a nicer error message, but apparently not.

[fhunleth]

https://app.circleci.com/pipelines/github/nerves-project/toolchains/127/workflows/939ae590-8ced-4f75-95e9-4cf0209cb549/jobs/6655

hdiutil: create failed - No space left on device

Hmmm, maybe need to increase the disk size somehow. Or free up space on already built toolchains.

I see the message "Not cleaning up work directory since CI build" and I don't remember why we did that. It seems like the work directory could be cleaned up. That certainly should save a lot of space.

[axelson]

Any ideas where else we could free up space? I guess I could try a build locally and then interactively check where the disk space is used.

[fhunleth]

See #105 for a rebased version. It's slightly improved, but still doesn't work. CircleCI's new build duration limits affect this now.