Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider pinning runner image instead of using ubuntu-latest in JIT CI #125560

Closed
savannahostrowski opened this issue Oct 16, 2024 · 7 comments
Closed
Assignees
Labels
build The build process and cross-build infra CI, GitHub Actions, buildbots, Dependabot, etc. topic-JIT

Comments

@savannahostrowski
Copy link
Member

savannahostrowski commented Oct 16, 2024

It looks like GitHub updated the default/latest Ubuntu runner to use Ubuntu 24.04 from 22.04, which is leading to failed CI runs. We should consider pinning the version here.

(GitHub's general guidance is to use ubuntu-latest, so if folks feel strongly against this, this could also be a temporary measure until the issue resolves.)

See also actions/runner-images#10788

Linked PRs

@savannahostrowski
Copy link
Member Author

cc: @brandtbucher in case you have additional thoughts here.

@ZeroIntensity
Copy link
Member

We should consider this for all our runners, as long as we remember to bump them every once in a while. I've had issues in the past with latest breaking things after an update.

@savannahostrowski
Copy link
Member Author

That said, I'm actually not sure if the image is missing dependencies...it seems like maybe something else has gone awry given that the error is actually a 503 on the workflow run.

@savannahostrowski savannahostrowski added the build The build process and cross-build label Oct 16, 2024
@hugovk
Copy link
Member

hugovk commented Oct 16, 2024

The 503s look like temporary network errors. You can click the "Re-run jobs" button and "Re-run failed jobs". If the errors persist on ubuntu-latest aka ubuntu-24.04, we can pin (in the short-to-medium term) to ubuntu-22.04 if that fixes it.

In general, pinning is a good idea to avoid surprises like in #125236, then we can do a controlled upgrade.

See also #122544.

@savannahostrowski
Copy link
Member Author

Yeah, that makes sense. Thanks for sharing those other issues!

Updating to pin to 22.04 did fix things last night. Judging by the noise on social media around this...it seems like a bunch of things broke (some of which were more ephemeral).

@mdboom
Copy link
Contributor

mdboom commented Oct 17, 2024

We should consider this for all our runners, as long as we remember to bump them every once in a while. I've had issues in the past with latest breaking things after an update.

It would be nice to have them pinned and make updating them part of something in the release cycle, maybe on tagging of alpha0.

@hugovk
Copy link
Member

hugovk commented Oct 17, 2024

Unfortunately, I don't think we can schedule them: we get Ubuntu releases every second year, and it will depend a lot on availability of the image on GitHub Actions, plus readiness of other dependencies.

For example, alpha0 was 8th May, and ubuntu-24.04 was available on (checking...) 14th May. That's not so bad, actually :) But it was a beta, the GA was last month, and when we checked upgrades in August, some things weren't ready, like Azure Pipelines (#122544).

And I wouldn't want it to be part of the release cycle (but it could be prompted by it) - there can be some non-trivial upgrades needed, and we wouldn't want to delay any part of the release because of it.

@picnixz picnixz added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Oct 21, 2024
savannahostrowski added a commit to savannahostrowski/cpython that referenced this issue Oct 22, 2024
savannahostrowski added a commit to savannahostrowski/cpython that referenced this issue Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build The build process and cross-build infra CI, GitHub Actions, buildbots, Dependabot, etc. topic-JIT
Projects
None yet
Development

No branches or pull requests

5 participants