Switch to using .dylib format for iOS binary modules

[freakboy3742]

Fixes #176.

The iOS App Store is rejecting apps that include binary components in the "Resources" folder. They will apparently only accept dynamically loaded components if they're (a) packaged as Frameworks, and (b) contained in the Frameworks folder.

This PR (which is is substantially built on the work of @alibeyram, @landrybr and team) allows for this usage pattern. It doesn't change how the support package is distributed; it relies on the project using the support package to move the .dylibs into their final location.

It incorporates changes that:

• alter the iOS module compilation format to .dylib
• sets the extension for iOS modules to .dylib
• installs a bootstrap loader to look in the Frameworks folder for .dylib modules.

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[freakboy3742]

At this point, the code requires the use of the template provided by @alibeyram: beeware/briefcase-iOS-Xcode-template#22

There's one more change required - dylibs need some name munging to avoid collisions when two submodules use the same binary name.

[alibeyram]

Thanks for the credit. Let me tag @landrybr who has done most of this.

@freakboy3742 I have a question about the AppleFrameworkFinder. Have you tested it with app_packages? I think sys.meta_path.append( will and I used sys.meta_path.insert(0, to go around the issue I am gonna mention.

For example

from . import _imaging as core

Given there are other meta_path that are priority they will try to load the _imaging.cpython-310-iphonesimulator.dylib which will not be available in . (current directory)

Regarding the name collisions, I think we can assume all the binaries (.dylib) are at the top directory in the packages. If you are ok with this assumption I can quickly modify both template and the AppleFrameworkFinder

[freakboy3742]

Thanks for the credit. Let me tag @landrybr who has done most of this.

@freakboy3742 I have a question about the AppleFrameworkFinder. Have you tested it with app_packages? I think sys.meta_path.append( will and I used sys.meta_path.insert(0, to go around the issue I am gonna mention.

I'm still in the process of testing; from what testing I've done so far, the only difference between insert(0, ...) and append(...) is whether the standard Python loader is tried first or last (and thus whether an attempt to load a pure-python package has to try 3 non-existent binary dylibs before succeeding).

However, I'll make sure testing deep packages is part of my testing before I finalise this patch.

[alibeyram]

Ok seems like I made some silly error. append seems to work fine.

[mhsmith]

installs a bootstrap loader to look in the Frameworks folder for .dylib modules

Regardless of where it's loaded from, the module's __file__ should be set to its "original" location within the Python directory tree, because this attribute is often used to find other files with relative paths. In my experience on Android, the fact that the __file__ doesn't actually exist has never caused a problem. But the attribute does need to be set before any code in the module is executed, i.e. between the "create" and "exec" stages in the importer.

There's one more change required - dylibs need some name munging to avoid collisions when two submodules use the same binary name.

The example I encountered on Android was "utils", which can be found at cytoolz.utils, h5py.utils and zmq.backend.cython.utils.

I didn't keep a note of the module name, but I also encountered a case in TensorFlow where one Python module was dynamically linked against another one. With ELF binaries, such links are resolved using the DT_NEEDED and DT_SONAME values compiled into the binaries, so the filename they're loaded from doesn't matter. But I don't know whether Mach-O has a similar mechanism.

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

[freakboy3742]

installs a bootstrap loader to look in the Frameworks folder for .dylib modules

Regardless of where it's loaded from, the module's __file__ should be set to its "original" location within the Python directory tree, because this attribute is often used to find other files with relative paths. In my experience on Android, the fact that the __file__ doesn't actually exist has never caused a problem. But the attribute does need to be set before any code in the module is executed, i.e. between the "create" and "exec" stages in the importer.

Noted - thanks for the heads up.

I didn't keep a note of the module name, but I also encountered a case in TensorFlow where one Python module was dynamically linked against another one. With ELF binaries, such links are resolved using the DT_NEEDED and DT_SONAME values compiled into the binaries, so the filename they're loaded from doesn't matter. But I don't know whether Mach-O has a similar mechanism.

It does; I'll need to dig into what RPATHs need to be rewritten to support this.

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

That's a good idea - unfortunately, it's taken me most of the day to confirm that the answer is... no. The Xcode app validator doesn't appear to differentiate between a bare dylib, and a symlink to a bare dylib.

[alibeyram]

For all of the above issues, it's also worth checking whether Apple allows you to load a module from the Frameworks folder via a symlink in a different location.

We tried that first and it did not work.

[alibeyram]

@freakboy3742 let me know if you decided to go with

framework_name = "_".join(fullname.split("."))

to solve the collision issue so I can update the template with the correct bash script.

[freakboy3742]

@alibeyram I've just opened beeware/briefcase-iOS-Xcode-template#23. In addition to the framework naming change, I needed to make some tweaks that I wanted to make around how the script executes on repeated passes, plus some simplifications on how the processing step is invoked to optimise compilation time.

[freakboy3742]

By way of proof - I've pushed up an update to Travel Tips to the App Store. It's past initial validation and been accepted for TestFlight; now the wait begins to see if human reviewers like it.

Next step is to confirm that third-party libraries can be compiled and used.

[mhsmith]

Next step is to confirm that third-party libraries can be compiled and used.

You mentioned today that this means all third-party libraries would have to be recompiled into new wheels containing dylibs, which couldn't be loaded by older versions of the support package which don't have the custom finder. Is this just because of their filenames? If so, what if the new wheels kept on using the .so suffix, and then Briefcase renamed them at the same time as it moved them into frameworks?

[alibeyram]

It's not just a rename. The validator will not accept the .so (shared object) architecture. You can run otool to see the differences between so and dylib

[mhsmith]

I understand that, but the question is, how do we release the new wheel files into the repository without breaking all the builds of existing apps using old Briefcase versions? Those users may be happy enough to have a working app even if they can't release it on the App Store, so it's not like those old versions of Briefcase were useless to them, and we shouldn't force them to upgrade without warning.

[alibeyram]

I see your point now.

[freakboy3742]

Four possible solutions to the backwards incompatibility problem:

1. The nuclear option: Delete all the old packages, and start from scratch.
2. Create a new Anaconda repo for "new style" dylib packages, and update Briefcase to use that repo.
3. Anaconda.org provides an option for "labels"; this is effectively a tagging mechanism, but the tags are each exposed as a separate PyPI endpoint. By default the "main" tag appears on https://pypi.anaconda.org/beeware/simple, but if we were to add a dylib label to the new packages, the repo https://pypi.anaconda.org/beeware/label/dylib/simple would only serve dylib-compatible packages. As long as we don't tag new packages with main, we should be safe.
4. Use a different binary tag. All the wheels are currently tagged ios_12_0, and are multi-platform wheels. If we switch to using separate iphoneos_12_0 and iphonesimulator_12_0 wheels, and do the merge in Briefcase, it means creating wheels becomes a lot simpler (as there's no post-processing step required). For this to be the best option, we'd need to be able to produce multi-arch iphonesimulator wheels in a single pass - I think this might be possible, but I need to do more tinkering.

[mhsmith]

When we talked about this the other day, we agreed that the old version of the support package probably would be able to load dylibs if they were called .so, but the .so -> MH_BUNDLE / .dylib -> MH_DYLIB mapping is so well established on Apple platforms that deviating from it would cause confusion.

So we concluded the best option was "4. Use a different binary tag." And we were inclined towards using separate wheels for each of the 3 ABIs, because:

• There's a huge variety of architecture detection code in build scripts, especially for non-Python packages, and most of it assumes there is only one architecture being compiled. We don't want to have to patch all of these projects one at a time to make them aware of Apple's multi-architecture compiler.

• It would give the app build tool maximum flexibility in whether to build a single-ABI or multi-ABI app. We should look beyond just Briefcase here, because we're attempting to define a standard for all Python-on-iOS packaging, and it would be great if it was adopted by Kivy and other projects as well, so we could combine our efforts and grow the selection of available packages more rapidly.

As for the format of the tag, I suggest we use the same format as is already used by macOS, Linux and Android, with the architecture after the version number. In which case, it makes sense to treat "simulator" as part of the architecture:

• ios_12_0_arm64
• ios_12_0_arm64_simulator
• ios_12_0_x86_64_simulator

[mhsmith]

we were inclined towards using separate wheels for each of the 3 ABIs

The downside of this is that Briefcase would have to run pip multiple times, and either merge the results together, or store them side by side.

However, all of that could be avoided if Briefcase just knew whether it was building for a physical or simulated device, because iOS currently only supports one physical architecture, and there is in practice only one useful simulator architecture – the architecture of the build machine.

[alibeyram]

Any downside to doing option 3 and changing the anaconda tag going forward for future versions of the briefcase? I see the benefits of option 4 specifically if other projects (e.g. Kivy) want to use this approach as well

[freakboy3742]

Any downside to doing option 3 and changing the anaconda tag going forward for future versions of the briefcase? I see the benefits of option 4 specifically if other projects (e.g. Kivy) want to use this approach as well

There's no specific downside to introducing a new tag - in fact, it might be a good idea to do in addition to option 4 to avoid confusion over which packages are available. The bigger question is whether there's a downside to having a single wheel, vs multiple per-platform (or per-architecture) wheels. I'm continuing to poke around in this space to get a working end-to-end toolchain that can build binary wheels with crossenv, then deploy them with Briefcase.

[KodaKoder]

The merge is very nice! just one thing trying to compile for Mac Catalyst is failing
Do you plan to add the support?

[freakboy3742]

@KodaKoder See #117.

It's not on our immediate roadmap, but if someone wants to add support, I'll take a look at a patch.

[KodaKoder]

Signing the libs raise this error

https://d.pr/i/LdU5EQ

[freakboy3742]

@KodaKoder Thanks for the report. What did you do to reproduce this error? In particular, are you using the new Xcode template and Briefcase version?

[KodaKoder]

I just imported the framework and stlib following the guide
https://github.com/beeware/Python-Apple-support/blob/main/USAGE.md
and changed the script to sign dylib instead of so

[freakboy3742]

Ah - that guide will need to be updated.

As the guide says at the top, the officially supported mechanism for using this package is with Briefcase. The new template (with new signing scripts) is beeware/briefcase-iOS-Xcode-template#23