Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace tools/download-cleaner #8279

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

antonlacon
Copy link
Contributor

This replaces tools/download-cleaner with tools/download-cleaner.py. Their intended purpose is the same, but the previous bash script had limitations. The main one is that it was cumbersome in considering multiple project/device/arches.

The replacement may consider all project/device/arch combinations (see builds_all), a limited portion of those with build directories (build.LibreELEC-RPi2.arm...), or the single build specified by PROJECT=X DEVICE=Y ARCH=Z ./download-cleaner.py. Each build combination takes around 30s to consider what packages are needed, so running --all by itself is around 30 minutes.

The way it works it by:

  1. create a build plan for each build combination chosen
  2. run pkginfo against each package specified in the build plan for the desired download filename
  3. compare tarballs found in the sources directory against what the buildplan wants (ex: dav1d-1.2.0.tar.xz is present, but tree is on dav1d-1.3.0.tar.xz, then add dav1d-1.2.0.tar.xz to list of unneeded files)
  4. delete file if asked (--delete)

By default, it only prints out what it would remove.

scripts/genbuildplan.py is extended with two additional CLI options to assist with this:
--hide-header prevents printing the statistics header in the output
--list-packages has it output only a list of packages - no depends or build steps

Work in progress because it doesn't consider packages removed from the tree yet. I'll add a commit to drop tools/download-cleaner when this replacement is ready. Probably still useful to someone else in the meantime, so sharing.

@antonlacon
Copy link
Contributor Author

This should be ready to replace tools/download-cleaner. The two features I didn't re-implement were downloading the current source tarball for an outdated package, and building a new sources directory.

Fetching outside of a build is a repeat of tools/download-tool (though it looks like it could use some attention), or this can be extended at a later date to replace that too.

Building a new sources dir was intended as a scriptable workaround for wanting source tarballs to multiple builds. Should no longer be needed.

The full list of builds (see L42) could use a second look. I thought there were more aarch64 builds.

@antonlacon
Copy link
Contributor Author

Added --export and --import to accommodate using between different git branches. --export will save the list of package names + package source tarballs to a file. --import will take one or more of the exported files and add them to its list of files to keep.

Switched more of builds_all over to aarch64.

@antonlacon antonlacon marked this pull request as draft February 4, 2024 06:21
@heitbaum heitbaum added LE 13.0 and removed LE 12.0 labels May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants