Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

offline archive #4353

Open
jwinarske opened this issue Aug 16, 2024 · 7 comments
Open

offline archive #4353

jwinarske opened this issue Aug 16, 2024 · 7 comments
Labels
type-enhancement A request for a change that isn't a bug

Comments

@jwinarske
Copy link

I have a need to support Long Term Support (LTS); automotive and IOT.

I'm proposing a new sub command called archive.

This would cache source types hosted and git into an archive folder.

Then without Internet access you can restore said archive, and build.

I have an implementation in Python that does this today. I am using this with the Yocto build system:
https://github.com/meta-flutter/meta-flutter/blob/scarthgap/tools/pubspec.py

@sigurdm
Copy link
Contributor

sigurdm commented Aug 19, 2024

We don't have a separate command for this today, but you can already do:

PUB_CACHE=.pub_cache dart pub get

Which would put the cache inside the current directory.
This way you can zip everything up.

Then when extracting somewhere run

PUB_CACHE=.pub_cache dart pub get --offline

To fix the absolute references in the .dart_tool/package_config.json

@sigurdm sigurdm added the type-enhancement A request for a change that isn't a bug label Aug 19, 2024
@jwinarske
Copy link
Author

jwinarske commented Aug 19, 2024

The use case is building 100+ apps simultaneously with LTS support. The proposed archive method dramatically decreases build storage. As there is only one archive of a package, which may show up 100+ times. The compress pub cache approach uses much more disk space at scale. The pub cache restore step takes ~1 second when multi-threaded.

@sigurdm
Copy link
Contributor

sigurdm commented Aug 19, 2024

Not sure I follow the use case exactly:

You have 100 apps, and now you want to make a single archive with all of them, or one archive for each?

Can you spell out what exactly the proposed command would do?

@jwinarske
Copy link
Author

@sigurdm
Archive would not include any redundant packages; only single references for git and hosted artifacts.

  1. pub archive
    Referencing the pubspec.lock file fetch hosted tar.gz, create tar.gz.256, and create .cache folder with -advisories.json, and -version.json files. This represents the archive; all the items fetched from the pub server.

  2. pub archive restore
    Referencing the pubspec.lock creates a pub cache using archive path. Restores pub cache to path specified by PUB_CACHE.

@sigurdm
Copy link
Contributor

sigurdm commented Aug 20, 2024

Archive would not include any redundant packages;

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

only single references for git and hosted artifacts.

What are "references" in this context?

@jwinarske
Copy link
Author

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

Yes this is true for a single application. If I look at all the artifacts fetched for 100 apps, there are many common packages shared between them. This means the archive folder would only contain one copy of each package version. Using a compressed pub archive duplicates the same package version many times; thus increasing disk space.

What are "references" in this context?

These would be considered a reference:

  • hosted/pub.dev/package-x.x.x.tar.gz
    hosted/pub.dev/.cache/package-versions.json
    hosted/pub.dev/.cache/package-advisories.json
  • git mirror
    git/cache/repo-sha1

@jwinarske
Copy link
Author

jwinarske commented Aug 21, 2024

Archive listing for flutter-gallery
archive_example.txt

File sizes

Flutter Gallery

  • Archive folder - 291M
  • tar.gz of PUB_CACHE - 304M
  • uncompressed PUB_CACHE - 700M

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-enhancement A request for a change that isn't a bug
Projects
None yet
Development

No branches or pull requests

2 participants