Tracking package dependencies

[olexandr-konovalov]

What happens if one installs a package which requires/suggest another package that is missing or has too old version? How to install dependencies automatically? If there is a newer version of the package, published by the authors, how to know whether it is already "stable" i.e. included in the latest GAP distribution, or "testing" (released and picked up by GAP for testing) or released and not picked up yet, or even not redistributed with GAP (also, whether the package archive has been updated without the change of the version number?)

Thinking of that could lead to the necessity of some local storing of the details of what has been installed, with checksums, and some registry which is run on GAP side, and could be used to get e.g. selection of packages from GAP X.Y.Z release (and maybe to autoupdate GAP as well?).

[mtorpey]

A lot of good ideas in here! A really good package manager would address all of these points, but in the interest of incremental development, I thought I'd sort the idea into different levels of dependency tracking, which can allow us to build up one step at a time. Currently we get all packages from your gap-distribution repository here, which I'm happy to say seems to pick up new releases almost immediately.

Here's a possible incremental list, as I see it:

1. No dependency tracking or version checking at all (just use InstallPackage for every package you need)
2. Check dependencies and install latest release (ignore required version numbers completely)
3. Check dependencies and install latest release, but return false if that release isn't new enough
4. All of the above, and check version against installed version of GAP, returning false if incompatible
5. Deliberately choose package versions that are compatible with the installed version of GAP, possibly giving preference to versions that were included as part of the release

Does that look like a reasonable roadmap? The first 2 or 3 points on the list will be very easy to implement (I might well manage them this week) but the later ones are a bit more complicated. Perhaps getting as far as 3 would be a good place for a v0.1 release.

[olexandr-konovalov]

Thanks. Clearly, this was a long-term goal, but we need to anticipate that to see where it goes and make good design choices.

The roadmap sounds good. I wonder what happens if installation fails (e.g. new version can not be compiled). Is there a way to roll back to the previously installed version?

Currently we get all packages from your gap-distribution repository here, which I'm happy to say seems to pick up new releases almost immediately.

Thanks, although I am not sure what do you mean. I only need to update URLs there when packages are moved to a new location or when packages are added/removed to/from the redistribution.

We need to think what would be the alternative or addition to https://raw.githubusercontent.com/gap-system/gap-distribution/master/DistributionUpdate/PackageUpdate/currentPackageInfoURLList. What do you think the GAP website should publish together with each release to be used by gpm?

[mtorpey]

I'm glad the roadmap seems reasonable! I'm willing to take advice from anyone based on what they would expect from a package manager.

Speaking of currentPackageInfoURLList, I meant to point out that the URLs it contains are not version-specific, i.e. they point to a location that is always up-to-date with the latest release. At least for early versions, this seems like a reasonable choice of directory, but I'm currently making this configurable so that we could maybe point it to a GAP version-specific list later.

[olexandr-konovalov]

yes - I envisage something similar to specifying source in apt-get. I think it is important that by default, users should be getting "approved" versions of GAP packages from the GAP website. Although, we publish them together with a GAP release, but not in the meantime. Will gpm be able to upgrade the whole GAP installation?

[mtorpey]

I guess this is an open question, which is worth discussing among anyone who wants to use this package:

Who curates the code that is available from this package manager? Package authors, or the GAP system?

On the one hand, we have systems like pip, which (I believe) points at a whole bunch of different servers so that any package author can update their code and have it available immediately.

On the other hand, we have systems like apt, which usually points at a single repository that gathers software itself and only makes it available after centralised checks.

Each style has its own advantages and disadvantages. My feeling tends to lean towards pip, because of the simple use-case that is most often what I have: "I want the latest version of this package right now". However, if people are still using older versions of GAP, they'll obviously want packages that are compatible. At the moment what I've written is pretty flexible and simple, and could easily be changed or made configurable later on. But it'd be good to hear what others want from this package. :)

[james-d-mitchell]

+1 for a pip style package manager

[mtorpey]

With regard to the roadmap above, I'm keen to get a basic release soon, so I'm planning to release v0.1 at level 1. Getting to level 2 or 3 should be the first priority in v0.2.

[mtorpey]

For this release, I'm going for level 3 on the above roadmap:

Check dependencies and install latest release, but return false if that release isn't new enough

We should install dependencies only if they're not available anywhere, so no need to install a load of packages that are already in the gap/pkg directory. We should only install dependencies that are completely unavailable, or whose current version is too old.

[olexandr-konovalov]

After changing my mind between pip and apt styles a couple of times, I am also for the pip style. I think the defining question is who is the potential user of this tool. A GAP user who has installed the latest GAP release already would already have the latest "approved" versions of all packages, so after that if they will run gpm, they may not get any updates (Remark: in principle, they may get new all-in-one archive of approved packages via make bootstrap-pkg-full, and rebuild all packages. It will be extra work also to prepare individual archives of approved packages, but I am not eliminating it at this stage).

To the contrary, developers who would like to work on the bleeding edge of GAP package development may find useful to be able to get all development versions of package via gpm easily. I think that this category will be the main use case of gpm.

Going for the pip-style, we need to continue to emphasise that:

• it's not a good practice to publish different versions of the code under the same version number
• one should develop more responsible attitude to specifying versions in package dependencies.

[mtorpey]

As of cc6fd45 we have achieved level 3 on the roadmap. Documentation and tests are required, and then I will make a 0.2 release.

[mtorpey]

As of d91dbe9, the documentation is done and dependency installation (including dependencies of dependencies) is tested.

I'll make a release shortly.

[olexandr-konovalov]

Related question: if package A needs package B, and package B is already installed, but not the latest version - will installing package A lead to updating package B?

[olexandr-konovalov]

Also, what happens if a package requires GAP version newer than the currently used?

[mtorpey]

Issue #84 represents the current state of this discussion, so I'll close this.