[icu] --clean-after-build option removes needed files for dependents

[ghost]

Describe the bug
Specifying --clean-after-build removes files other dependents may need in order to build properly.

Environment

• OS: Windows 10
• Compiler: MSVC 2019

To Reproduce
Steps to reproduce the behavior:

1. ./vcpkg install qt5-base:<triplet> --clean-after-build
2. qt5-base fails to build until icu is removed and then installed without the --clean-after-build flag

Expected behavior
The needed headers or libraries should be present to allow dependents to build correctly.

Failure logs
If I recall correctly it is an undefined symbol error. qt5-base is where it was detected. There may be others but they have not been researched or tested.

[StarGate-One]

1. --clean-after-build deletes all [vcpkg-root]\buildtrees\[port]\[triplet]-[dbg/rel] folders and files (leaving the logs) and all [vcpkg-root]\packages\[port]_[triplet] folders and files along with the downloaded source file upon completion of each port built.
2. If the mumble script needs any of the files within the forementioned directories/sub-folders, then do not include the --clean-after-build parameter when invoking vcpkg
3. Exactly which headers/libraries are being deleted as I am I am unable to replicate the issue:
   New vcpkg instance as of 21:16 EST on 2021/01/06...

.\vcpkg.exe install --clean-after-build qt5-base:x64-windows
The following packages will be built and installed:

• brotli[core]:x64-windows -> 1.0.9
• bzip2[core]:x64-windows -> 1.0.8#1
• double-conversion[core]:x64-windows -> 3.1.5
• egl-registry[core]:x64-windows -> 2020-02-20
• freetype[brotli,bzip2,core,png,zlib]:x64-windows -> 2.10.4
• harfbuzz[core]:x64-windows -> 2.7.2#1
• icu[core]:x64-windows -> 67.1#7
• libjpeg-turbo[core]:x64-windows -> 2.0.5#2
• libpng[core]:x64-windows -> 1.6.37#14
• libpq[core,openssl,zlib]:x64-windows -> 12.2#10
• openssl[core]:x64-windows -> 1.1.1h#4
• pcre2[core]:x64-windows -> 10.35#2
  qt5-base[core]:x64-windows -> 5.15.1#3
• ragel[core]:x64-windows -> 6.10-3
• sqlite3[core]:x64-windows -> 3.33.0
• zlib[core]:x64-windows -> 1.2.11#9
• zstd[core]:x64-windows -> 1.4.5#1
  Additional packages (*) will be modified to complete this operation.

Starting package 1/17: brotli:x64-windows
Building package brotli[core]:x64-windows...

building/compiling/linking/installing ports ... .... ...

Installing package zstd[core]:x64-windows...
Installing package zstd[core]:x64-windows... done
Elapsed time for package zstd:x64-windows: 14.33 s
Starting package 17/17: qt5-base:x64-windows
Building package qt5-base[core]:x64-windows...
-- Figuring out qt target mkspec. Target arch x64
-- Target mkspec set to: win32-msvc
-- Host mkspec not set. Qt's own buildsystem will try to figure out the host system
-- Downloading http://download.qt.io/official_releases/qt/5.15/5.15.1/submodules/qtbase-everywhere-src-5.15.1.tar.xz...
-- Extracting source D:/Git/vcpkg/test/downloads/qtbase-everywhere-src-5.15.1.tar.xz
-- Applying patch patches/winmain_pro.patch
-- Applying patch patches/windows_prf.patch
-- Applying patch patches/qt_app.patch
-- Applying patch patches/gui_configure.patch
-- Applying patch patches/icu.patch
-- Applying patch patches/xlib.patch
-- Applying patch patches/egl.patch
-- Applying patch patches/zstdd.patch
-- Applying patch patches/Qt5BasicConfig.patch
-- Applying patch patches/Qt5PluginTarget.patch
-- Applying patch patches/create_cmake.patch
-- Applying patch patches/Qt5GuiConfigExtras.patch
-- Using source at D:/Git/vcpkg/test/buildtrees/qt5-base/src/5.15.1-63a9328040.clean
-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst...
-- Using msys root at D:/Git/vcpkg/test/downloads/tools/msys2/c809757c94447846
-- Configuring x64-windows-dbg
-- Configuring x64-windows-dbg done
-- Configuring x64-windows-rel
-- Configuring x64-windows-rel done
-- Building x64-windows-dbg
-- Fixing makefile installation path x64-windows-dbg
-- Installing x64-windows-dbg
-- Package x64-windows-dbg done
-- Building x64-windows-rel
-- Fixing makefile installation path x64-windows-rel
-- Installing x64-windows-rel
-- Package x64-windows-rel done
-- Installing: D:/Git/vcpkg/test/packages/qt5-base_x64-windows/share/qt5-base/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package qt5-base[core]:x64-windows... done
Installing package qt5-base[core]:x64-windows...
Installing package qt5-base[core]:x64-windows... done
Elapsed time for package qt5-base:x64-windows: 9.723 min

Total elapsed time: 21.54 min

The package qt5-base:x64-windows provides CMake targets:

find_package(Qt5AccessibilitySupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::AccessibilitySupport Qt5::AccessibilitySupport Qt::AccessibilitySupportPrivate Qt5::AccessibilitySupportPrivate)

find_package(Qt5Bootstrap CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::Bootstrap Qt5::Bootstrap Qt::BootstrapPrivate Qt5::BootstrapPrivate)

find_package(Qt5Concurrent CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::Concurrent Qt5::Concurrent Qt::ConcurrentPrivate Qt5::ConcurrentPrivate)

find_package(Qt5Core CONFIG REQUIRED)
# Note: 2 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::Core Qt5::Core Qt::WinMain Qt5::WinMain)

find_package(Qt5DBus CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::DBus Qt5::DBus Qt::DBusPrivate Qt5::DBusPrivate)

find_package(Qt5DeviceDiscoverySupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::DeviceDiscoverySupport Qt5::DeviceDiscoverySupport Qt::DeviceDiscoverySupportPrivate Qt5::DeviceDiscoverySupportPrivate)

find_package(Qt5EdidSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::EdidSupport Qt5::EdidSupport Qt::EdidSupportPrivate Qt5::EdidSupportPrivate)

find_package(Qt5EventDispatcherSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::EventDispatcherSupport Qt5::EventDispatcherSupport Qt::EventDispatcherSupportPrivate Qt5::EventDispatcherSupportPrivate)

find_package(Qt5FbSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::FbSupport Qt5::FbSupport Qt::FbSupportPrivate Qt5::FbSupportPrivate)

find_package(Qt5FontDatabaseSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::FontDatabaseSupport Qt5::FontDatabaseSupport Qt::FontDatabaseSupportPrivate Qt5::FontDatabaseSupportPrivate)

find_package(Qt5Gui CONFIG REQUIRED)
# Note: 9 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::Gui Qt5::Gui Qt::GuiPrivate Qt5::GuiPrivate)

find_package(Qt5Network CONFIG REQUIRED)
# Note: 1 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::Network Qt5::Network Qt::NetworkPrivate Qt5::NetworkPrivate)

find_package(Qt5OpenGL CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::OpenGL Qt5::OpenGL Qt::OpenGLPrivate Qt5::OpenGLPrivate)

find_package(Qt5OpenGLExtensions CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::OpenGLExtensions Qt5::OpenGLExtensions Qt::OpenGLExtensionsPrivate Qt5::OpenGLExtensionsPrivate)

find_package(Qt5PlatformCompositorSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::PlatformCompositorSupport Qt5::PlatformCompositorSupport Qt::PlatformCompositorSupportPrivate Qt5::PlatformCompositorSupportPrivate)

find_package(Qt5PrintSupport CONFIG REQUIRED)
# Note: 1 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::PrintSupport Qt5::PrintSupport Qt::PrintSupportPrivate Qt5::PrintSupportPrivate)

find_package(Qt5Sql CONFIG REQUIRED)
# Note: 3 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::Sql Qt5::Sql Qt::SqlPrivate Qt5::SqlPrivate)

find_package(Qt5Test CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::Test Qt5::Test Qt::TestPrivate Qt5::TestPrivate)

find_package(Qt5ThemeSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::ThemeSupport Qt5::ThemeSupport Qt::ThemeSupportPrivate Qt5::ThemeSupportPrivate)

find_package(Qt5VulkanSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::VulkanSupport Qt5::VulkanSupport Qt::VulkanSupportPrivate Qt5::VulkanSupportPrivate)

find_package(Qt5Widgets CONFIG REQUIRED)
# Note: 1 target(s) were omitted.
target_link_libraries(main PRIVATE Qt::Widgets Qt5::Widgets Qt::WidgetsPrivate Qt5::WidgetsPrivate)

find_package(Qt5WindowsUIAutomationSupport CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::WindowsUIAutomationSupport Qt5::WindowsUIAutomationSupport Qt::WindowsUIAutomationSupportPrivate Qt5::WindowsUIAutomationSupportPrivate)

find_package(Qt5Xml CONFIG REQUIRED)
target_link_libraries(main PRIVATE Qt::Xml Qt5::Xml Qt::XmlPrivate Qt5::XmlPrivate)

[ghost]

We were working around this issue by installing icu before installing qt5-base without the option, as you described. If you can't reproduce the error with other ports that require it, then it might be related to the x64-windows-static-md community triplet. We could continue working around the issue. Obviously, keeping Qt component files around after the libraries are built uses a lot of space and it would be nice to avoid that.

[StarGate-One]

Tomorrow I will look at the mumble powershell script handling the vcpkg, ports and features installation to see if I can spot something. I did see during my cursory look that each port is being installed with a vcpkg install for each port. It is possible a prerequisite port is being installed with default/core only features then later on the same prerequisite port is required needing an optional feature in addition to the default/core features. In this case, vcpkg will fail unless the vcpkg install command line includes the --recurse option. This basically tells vcpkg to remove the port then reinstall if with what ever features are currently installed plus the additional features needed now. Of course putting all the ports on the same vcpkg install command line, vcpkg will compute the best order to install the ports with all required features. Or if you study each port, features and prerequisite ports features you could order them in the for each powershell loop and accomplish the same results but would require more maintenance than just letting vcpkg take care of it.

[ghost]

It appears our script might have been installing icu individually as expected. We were also doing the same thing with mdnsresponder. I have revised the script and am testing it now.

[StarGate-One]

@ZeroAbility

• I wrote a cmd/bat windows batch file to run x64-windows-static-md 2 different ways, 2 different vcpkg instances
  • Option 1 runs the port installs in the order from the get_mumble_dependencies.ps1 powershell scripts
  • Option 2 run the port installs from one command line entry
  • Both use the --clean-after-build and --debug options (In case a port failed I would have more information)
  • Neither failed install on any port
  • Both vcpkg instances size are 14.2 GB
  • I did not run the x86-windows-static-md option, but the x64-windows-static-md run does include the boost-optional:x86-windows-static-md ports (I assume they are needed for some other portion of mumble)
• Attached is the script vcpkg_mumble.zip
• Attached list of installed ports and install logs from each instance
  • vcpkg_mumble_1
    vcpkg_list_1.zip
    vcpkg_mumble_1.zip
  • vcpkg_mumble_2
    vcpkg_list_2.zip
    vcpkg_mumble_2.zip

[ghost]

@StarGate-One

Of course putting all the ports on the same vcpkg install command line, vcpkg will compute the best order to install the ports with all required features.

I appreciate the help. I have submitted a PR for the revised script that installs all packages as an array param to the powershell function, as you suggested.

It is possible a prerequisite port is being installed with default/core only features then later on the same prerequisite port is required needing an optional feature in addition to the default/core features.
In this case, vcpkg will fail unless the vcpkg install command line includes the --recurse option. This basically tells vcpkg to remove the port then reinstall if with what ever features are currently installed plus the additional features needed now.

Or if you study each port, features and prerequisite ports features you could order them in the for each powershell loop and accomplish the same results but would require more maintenance than just letting vcpkg take care of it.

That seems like a lot of potential pitfalls when people already have a partial prerequisite installed and have to remove that and possibly other packages in order to satisfy the requirements of both needed dependents. A lot of the Qt packages take a while to build. Our scenario is mostly straightforward, and making the change you suggested is feasible. I hope this project continues to mature so this isn't a concern.

Thanks again. :)

[StarGate-One]

You are most welcome!

[NancyLi1013]

Hi @ZeroAbility

Sorry for the delay.

Please let me know if you have other concerns about this issue.

@StarGate-One

Thanks for your help. It's so nice of you.

[StarGate-One]

@NancyLi1013 You are most welcome!

[ghost]

Hi @ZeroAbility

Sorry for the delay.

Please let me know if you have other concerns about this issue.

We resolved this with the suggestion from @StarGate-One. I'm still concerned that in a different scenario this issue may come up again. This leads me to believe the files that get copied over should be verified and complete. Not every use case is going to be as simple as adding all of the needed packages, and reinstalling may not always be a good option. Especially when dealing with Qt components.