clang-14, llvm-14: restore functionality on macOS 15 (Xcode 16) #25919
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Sep 25, 2024
This is a cherry-pick of llvm/llvm-project@73e15b5. This enables lldb-14 to be built by Xcode ≥ 15, making it possible to build on macOS ≥ 14, and macOS 13 with Xcode 15. References: https://trac.macports.org/ticket/70779
markmentovai
force-pushed
the
clang-14_xcode-16
branch
from
a8b36c8
to
2b4d892
Compare
|
@markmentovai I am seeing the same warnings here as with clang-15 as discussed here #25918 (comment) |
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 26, 2024
This applies a1d4865, which addressed this for clang-16–18 for https://trac.macports.org/ticket/68640, to clang-14, to fix a bug observed while running clang-14 as a linker driver on macOS 15. When running as a linker driver, clang provides ld with clang’s own LTO plugin by invoking ld with `-lto_library ${PREFIX}/libexec/llvm-15/lib/libLTO.dylib`. Upon receipt of these arguments, Xcode ld currently loads the plugin by re-`execve`ing itself with DYLD_LIBRARY_PATH=${PREFIX}/libexec/llvm-15/lib in effect, causing dyld to prefer libLTO.dylib in that directory over the @rpath/libLTO.dylib that ld requests to load via a Mach-O load command. With DYLD_LIBRARY_PATH in effect, dyld can potentially use any other module in the same directory to satisfy any other Mach-O load command. In this case, the directory contained both libc++.1.dylib and libc++abi.dylib from clang, and dyld used these to replace the libraries of the same name ordinarily provided by the OS in /usr/lib (via the dyld shared cache). This is undesirable in general, but occurred silently on macOS < 15. It became noticeable on macOS 15 because system libraries that depend on libc++abi.dylib now reference symbols that are present in the system’s version, but not in clang’s, causing messages such as dyld[60301]: symbol '__ZnwmSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. dyld[60301]: symbol '__ZdlPvSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. repeated for every system library that references those symbols. These messages warn of potential run-time bugs due to dyld intentionally mis-resolving the missing symbols. As clang should not seek to replace the system’s libc++ with its own in system libraries, this was a latent problem even pre-macOS 15. The workaround moves clang’s libc++ libraries to a new subdirectory, ${PREFIX}/libexec/llvm-15/lib/libc++, where they will not be found or used even with DYLD_LIBRARY_PATH set to ${PREFIX}/libexec/llvm-15/lib as it is when clang runs the linker. This is an observable change in the installed clang-14 package, but the revision is not being bumped in this commit because this change is being merged atomically in macports#25919 with another change that updates clang-14’s revision. References: https://trac.macports.org/ticket/70779
markmentovai
force-pushed
the
clang-14_xcode-16
branch
from
2b4d892
to
f7c8132
Compare
|
I converted this and #25918 to a draft, because they shouldn’t be merged until #25919 (comment) can be addressed. More information at #25918 (comment). |
This applies a1d4865, which addressed this for clang-16–18 for https://trac.macports.org/ticket/68640, to clang-14, to fix a bug observed while running clang-14 as a linker driver on macOS 15. When running as a linker driver, clang provides ld with clang’s own LTO plugin by invoking ld with `-lto_library ${PREFIX}/libexec/llvm-14/lib/libLTO.dylib`. Upon receipt of these arguments, Xcode ld currently loads the plugin by re-`execve`ing itself with DYLD_LIBRARY_PATH=${PREFIX}/libexec/llvm-14/lib in effect, causing dyld to prefer libLTO.dylib in that directory over the @rpath/libLTO.dylib that ld requests to load via a Mach-O load command. With DYLD_LIBRARY_PATH in effect, dyld can potentially use any other module in that same directory to satisfy any other Mach-O load command. In this case, the directory contained both libc++.dylib and libc++abi.dylib from clang, and dyld used these to replace the libraries of the same name ordinarily provided by the OS in /usr/lib (via the dyld shared cache). This is undesirable in general, but occurred silently on macOS < 15. It became noticeable on macOS 15 because system libraries that depend on libc++abi.dylib now reference symbols that are present in the system’s version, but not in clang’s, causing messages such as dyld[60301]: symbol '__ZnwmSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. dyld[60301]: symbol '__ZdlPvSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. repeated for every system library that references those symbols. These messages warn of potential run-time bugs due to dyld intentionally mis-resolving the missing symbols. As clang should not seek to replace the system’s libc++ with its own in system libraries, this was a latent problem even pre-macOS 15. The workaround moves clang’s libc++ libraries to a new subdirectory, ${PREFIX}/libexec/llvm-14/lib/libc++, where they will not be found or used even with DYLD_LIBRARY_PATH set to ${PREFIX}/libexec/llvm-14/lib as it is when clang runs the linker. This also contains a fix for the LC_INSTALL_NAMEs of libc++.dylib and libc++abi.dylib, which should have been recorded as @rpath-relative, but due to an error in the Portfile’s handling of CMAKE_INSTALL_NAME_DIR, were instead recorded using absolute paths rooted at ${PREFIX}/libexec/llvm-14/lib. When the libraries were moved elsewhere, they tripped MacPorts’ check for linking errors due to libc++.dylib’s dependency on libc++abi.dylib at a path where it was no longer installed. Discussion at macports#25918 (comment). This is an observable change in the installed clang-14 package, but the revision is not being bumped in this commit because this change is being merged atomically in macports#25919 with another change that updates clang-14’s revision. References: https://trac.macports.org/ticket/70779
This is a cherry-pick of: llvm/llvm-project@c57c7b7 llvm/llvm-project@7939ce3 This enables clang-14 to be built by Xcode 16, making it possible to build on macOS 15, and macOS 14 with Xcode 16. It also enables clang-14 to be built by clang-18 and newer. References: https://trac.macports.org/ticket/70779
markmentovai
force-pushed
the
clang-14_xcode-16
branch
from
f7c8132
to
49df5ce
Compare
|
Should be all fixed now. No library overrides, unresolved symbols, or .dylibs with bad install names or attempting to load other .dylibs at nonexistent paths. |
cjones051073
pushed a commit
that referenced
this pull request
Sep 27, 2024
This applies a1d4865, which addressed this for clang-16–18 for https://trac.macports.org/ticket/68640, to clang-14, to fix a bug observed while running clang-14 as a linker driver on macOS 15. When running as a linker driver, clang provides ld with clang’s own LTO plugin by invoking ld with `-lto_library ${PREFIX}/libexec/llvm-14/lib/libLTO.dylib`. Upon receipt of these arguments, Xcode ld currently loads the plugin by re-`execve`ing itself with DYLD_LIBRARY_PATH=${PREFIX}/libexec/llvm-14/lib in effect, causing dyld to prefer libLTO.dylib in that directory over the @rpath/libLTO.dylib that ld requests to load via a Mach-O load command. With DYLD_LIBRARY_PATH in effect, dyld can potentially use any other module in that same directory to satisfy any other Mach-O load command. In this case, the directory contained both libc++.dylib and libc++abi.dylib from clang, and dyld used these to replace the libraries of the same name ordinarily provided by the OS in /usr/lib (via the dyld shared cache). This is undesirable in general, but occurred silently on macOS < 15. It became noticeable on macOS 15 because system libraries that depend on libc++abi.dylib now reference symbols that are present in the system’s version, but not in clang’s, causing messages such as dyld[60301]: symbol '__ZnwmSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. dyld[60301]: symbol '__ZdlPvSt19__type_descriptor_t' missing from root that overrides /usr/lib/libc++abi.dylib. Use of that symbol in /usr/lib/swift/libswiftCore.dylib is being set to 0xBAD4007. repeated for every system library that references those symbols. These messages warn of potential run-time bugs due to dyld intentionally mis-resolving the missing symbols. As clang should not seek to replace the system’s libc++ with its own in system libraries, this was a latent problem even pre-macOS 15. The workaround moves clang’s libc++ libraries to a new subdirectory, ${PREFIX}/libexec/llvm-14/lib/libc++, where they will not be found or used even with DYLD_LIBRARY_PATH set to ${PREFIX}/libexec/llvm-14/lib as it is when clang runs the linker. This also contains a fix for the LC_INSTALL_NAMEs of libc++.dylib and libc++abi.dylib, which should have been recorded as @rpath-relative, but due to an error in the Portfile’s handling of CMAKE_INSTALL_NAME_DIR, were instead recorded using absolute paths rooted at ${PREFIX}/libexec/llvm-14/lib. When the libraries were moved elsewhere, they tripped MacPorts’ check for linking errors due to libc++.dylib’s dependency on libc++abi.dylib at a path where it was no longer installed. Discussion at #25918 (comment). This is an observable change in the installed clang-14 package, but the revision is not being bumped in this commit because this change is being merged atomically in #25919 with another change that updates clang-14’s revision. References: https://trac.macports.org/ticket/70779
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 27, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.0.tcl port group because the port group enclosed the value in quotes, but the Portfile omitted them. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group was not setting CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in macports#25918, and llvm-14 in macports#25919. A more comprehensive explanation of the change appears at macports#25918 (comment).
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 27, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.0.tcl port group because the port group enclosed the value in quotes, but the Portfiles omitted them. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group was not setting CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in macports#25918, and llvm-14 in macports#25919. A more comprehensive explanation of the change appears at macports#25918 (comment).
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 27, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.1.tcl port group because the port group enclosed the value in quotes, but the Portfiles omitted them. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group was not setting CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in macports#25918, and llvm-14 in macports#25919. A more comprehensive explanation of the change appears at macports#25918 (comment).
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 27, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.1.tcl port group because the port group enclosed the value in quotes, but the Portfiles omitted the quotes. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group was not setting CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in macports#25918, and llvm-14 in macports#25919. A more comprehensive explanation of the change appears at macports#25918 (comment).
markmentovai
added a commit
to markmentovai/macports-ports
that referenced
this pull request
Sep 27, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.1.tcl port group because the port group encloses the value in quotes, but the Portfiles omitted the quotes. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group does not set CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in macports#25918, and llvm-14 in macports#25919. A more comprehensive explanation of the change appears at macports#25918 (comment).
12 tasks
cjones051073
pushed a commit
that referenced
this pull request
Sep 28, 2024
The Portfiles for these llvm versions were mishandling CMAKE_INSTALL_NAME_DIR in configure.pre_args-delete, failing to delete the intended variable set by the _resources/port1.0/group/cmake-1.1.tcl port group because the port group encloses the value in quotes, but the Portfiles omitted the quotes. CMAKE_INSTALL_RPATH was being mishandled in the same way, but because these Portfiles specify cmake.install_rpath as empty, the port group does not set CMAKE_INSTALL_RPATH at all, so the configure.pre_args-delete entry for that variable can be removed from the Portfiles altogether. These Portfiles’ configure.pre_args-replace item for CMAKE_SYSTEM_PREFIX_PATH has been updated to reflect the value actually set by the port group. For llvm-16 and later, these Portfile updates should not cause any meaningful change to the build output, so these Portfiles’ `revision` fields are not updated. This same change would have impact to llvm-15 and earlier. llvm-15 was handled more carefully in #25918, and llvm-14 in #25919. A more comprehensive explanation of the change appears at #25918 (comment).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a cherry-pick of:
llvm/llvm-project@c57c7b7 llvm/llvm-project@7939ce3
This enables clang-14 to be built by Xcode 16, making it possible to build on macOS 15, and macOS 14 with Xcode 16. It also enables clang-14 to be built by clang-18 and newer.
References: https://trac.macports.org/ticket/70779
Description
Type(s)
Tested on
macOS 15.0 24A335 arm64
Xcode 16.0 16A242d
Verification
Have you
port lint --nitpick?sudo port test?sudo port -vst install?@cjones051073