Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trim down the upgrade testing workflow. #7966

Closed
wants to merge 9 commits into from

Conversation

desrosj
Copy link
Contributor

@desrosj desrosj commented Dec 6, 2024

As it stands today, the upgrade testing workflow is currently at ~978 jobs created. While it's great to test all possible combinations, GitHub's UI cannot keep up with tracking that number of jobs, and that is 2x the total number of allowed jobs for the organization (which slows everything down even more).

This PR trims down the number of combinations included in the testing matrices to be a bit more thoughtful following the following methodology:

  • The last two releases of WordPress are tested against all PHP and MySQL LTS version combinations and the most recent innovation release.
  • The next 6 oldest versions of WordPress are tested against both the oldest and newest releases of PHP currently supported for both PHP 7 & 8 along with the oldest and newest MySQL LTS versions currently supported (no innovation releases).
  • For the remaining versions of WordPress receiving security updates, they are only included if the database version was different that the previous major release.
  • The oldest version of WordPress receiving security updates should always be tested.

Some additional notes about the chosen MySQL versions:

  • Only the most recent innovation release should be included in testing.
  • Even though MySQL >= 5.5.5 is currently supported, there are no 5.5.x Docker containers available that work on modern architectures.
  • 5.6.x Docker containers are available and work, but 5.6 only accounts for ~2.3% of installs as of 12/6/2024.defaults:
  • 5.7.x accounts for ~20% of installs, so this is used below instead.

The contents of this PR description are included as inline documentation in the workflow to help guide contributors when updating this workflow in the future.

The changes in this workflow decrease the total number of jobs from 978 to 360 (a 63% reduction) and results in the entire job completing in less than 10 minutes (it currently takes anywhere from 20-40 minutes for the workflow to reflect the outcome of all spawned jobs).

Trac ticket: https://core.trac.wordpress.org/ticket/62221


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

Copy link

github-actions bot commented Dec 6, 2024

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • The Plugin and Theme Directories cannot be accessed within Playground.
  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

@desrosj desrosj self-assigned this Dec 6, 2024
@desrosj desrosj marked this pull request as ready for review December 6, 2024 16:32
Copy link

github-actions bot commented Dec 6, 2024

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props desrosj, mukesh27.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@desrosj
Copy link
Contributor Author

desrosj commented Dec 13, 2024

I've made one adjustment to the strategy as described above.

I've separated out the oldest version still receiving security updates and expanded the matrix to ensure it's tested against the same range of releases as the last 2 releases. I think that it makes sense to run the oldest version against the full set of combinations.

With this addition, it brings the reduction to ~57%, a total of 416 jobs.

@desrosj
Copy link
Contributor Author

desrosj commented Dec 13, 2024

@desrosj desrosj closed this Dec 13, 2024
@desrosj desrosj deleted the trim-upgrade-testing branch December 13, 2024 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants