New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache #44644

Merged

kou merged 33 commits into apache:main from kou:ci-wheel-manylinux-vcpkg-cache

Nov 15, 2024

Member

kou commented Nov 5, 2024 •

edited

Loading

Rationale for this change

We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.

What changes are included in this PR?

Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.

See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages

NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [CI][Python] Improve vcpkg caching #43951

kou requested review from assignUser, jonkeane and raulcd as code owners

November 5, 2024 06:04

github-actions bot commented Nov 5, 2024

⚠️ GitHub issue #43951 has been automatically assigned in GitHub to PR creator.

github-actions bot added the awaiting committer review label

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

github-actions bot commented Nov 5, 2024

Failed to render template `python-wheels/github.linux.yml` with UndefinedError: 'secrets' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/11678507794

kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from c931193 to f401684 Compare

November 5, 2024 06:08

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 5, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

1 similar comment

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

kou changed the title ~~GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache~~ GH-43951: [CI][Python] Use GitHub Actions cache for vcpkg cache

kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from 78ee981 to 8b4f8ab Compare

November 6, 2024 08:00

kou mentioned this pull request

[CI][Dev] crossbow submit can't write a comment to the target PR #44657

Closed

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

2 similar comments

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

kou changed the title ~~GH-43951: [CI][Python] Use GitHub Actions cache for vcpkg cache~~ GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

1 similar comment

Member Author

kou commented Nov 6, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

kou added 6 commits

November 12, 2024 13:11


          Use mono-core on CentOS 7

d7114e7


          Don't use NuGet cache in manylinux2014_aarch64

e0bb0be


          Enable VCPKG_BINARY_SOURCES explicitly

66940c7

Fix

141cc07


          Fix arch

c239815


          Use secret mount

kou force-pushed the ci-wheel-manylinux-vcpkg-cache branch from 63d255c to 6730902 Compare

November 13, 2024 07:33

github-actions bot added awaiting change review and removed awaiting changes labels

kou added 3 commits

November 13, 2024 16:37


          Style

5d00978


          Remove java-jni

013c5bb


          Fix syntax

52b6ee9

Member Author

kou commented Nov 13, 2024

@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64

This comment was marked as outdated.

Sign in to view

Member Author

kou commented Nov 13, 2024

@github-actions crossbow submit -g wheel java-jars

github-actions bot commented Nov 13, 2024

Revision: 52b6ee9

Submitted crossbow builds: ursacomputing/crossbow @ actions-f4d0e5be13

Task	Status
java-jars
python-sdist
wheel-macos-monterey-cp310-cp310-amd64
wheel-macos-monterey-cp310-cp310-arm64
wheel-macos-monterey-cp311-cp311-amd64
wheel-macos-monterey-cp311-cp311-arm64
wheel-macos-monterey-cp312-cp312-amd64
wheel-macos-monterey-cp312-cp312-arm64
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-macos-monterey-cp39-cp39-amd64
wheel-macos-monterey-cp39-cp39-arm64
wheel-manylinux-2-28-cp310-cp310-amd64
wheel-manylinux-2-28-cp310-cp310-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2-28-cp39-cp39-amd64
wheel-manylinux-2-28-cp39-cp39-arm64
wheel-manylinux-2014-cp310-cp310-amd64
wheel-manylinux-2014-cp310-cp310-arm64
wheel-manylinux-2014-cp311-cp311-amd64
wheel-manylinux-2014-cp311-cp311-arm64
wheel-manylinux-2014-cp312-cp312-amd64
wheel-manylinux-2014-cp312-cp312-arm64
wheel-manylinux-2014-cp313-cp313-amd64
wheel-manylinux-2014-cp313-cp313-arm64
wheel-manylinux-2014-cp313-cp313t-amd64
wheel-manylinux-2014-cp313-cp313t-arm64
wheel-manylinux-2014-cp39-cp39-amd64
wheel-manylinux-2014-cp39-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp313-amd64
wheel-windows-cp39-amd64

github-actions bot added awaiting changes and removed awaiting change review labels

raulcd reviewed

View reviewed changes

Member

raulcd left a comment

LGTM, just a minor nit and a question

ci/docker/python-wheel-manylinux.dockerfile Outdated Show resolved Hide resolved

ci/scripts/install_vcpkg.sh

@@ @@ -53,4 +53,33 @@ if [ -f "${vcpkg_ports_patch}" ]; then @@
                 echo "Patch successfully applied to the VCPKG port files!"
               fi
+              if [ -n "${GITHUB_TOKEN:-}" ] && \
+                   [ -n "${GITHUB_REPOSITORY_OWNER:-}" ] && \
+                   [ "${VCPKG_BINARY_SOURCES:-}" = "clear;nuget,GitHub,readwrite" ] ; then

Member

raulcd Nov 13, 2024

in which case would VCPKG_BINARY_SOURCES be different to clear;nuget,GitHub,readwrite? on manylinux2014_aarch64 this wouldn't be present but not different, right?

Member Author

kou Nov 13, 2024

If VCPKG_BINARY_SOURCES is not clear;nuget,GitHub,readwrite (including not present case on manylinux2014_aarch64), we don't configure NuGet automatically.

Ignoring not clear;nuget,GitHub,readwrite VCPKG_BINARY_SOURCES is for using other cache. For example, we may want to use local file cache for local archery docker run. But it's out-of-scope of this PR and it's not tested. If we need it, we can work on it as a separated task.


          Fix English

b2238ef

Co-authored-by: Raúl Cumplido <[email protected]>

github-actions bot added awaiting change review awaiting changes and removed awaiting changes awaiting change review labels

assignUser approved these changes

View reviewed changes

Member

assignUser left a comment

Thanks for adding the proper secret handling, looks great!

Member Author

kou commented Nov 15, 2024

I'll merge this.

kou merged commit df40f7a into apache:main

46 checks passed

kou removed the awaiting changes label

kou deleted the ci-wheel-manylinux-vcpkg-cache branch

November 15, 2024 09:41

conbench-apache-arrow bot commented Nov 15, 2024

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit df40f7a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet