Metadata mismatch between Release and Packages.bz
#46
Comments
|
This is a bit of a tricky issue. As you've discovered, there is certainly the possibility for this to happen. The problem is that a Release file generally references a lot of package files, many of which won't be used and are rather large. So the options are: 1.) Whenever we fetch a Release file, force a download of all the corresponding Package files as well, which can take a long time 2.) Whenever we fetch a Release file, invalidate the metadata cache, and hope that the client is going to fetch the metadata files it cares about in the next couple of seconds Right now we do option 2 (https://github.com/sonatype-nexus-community/nexus-repository-apt/blob/master/src/main/java/net/staticsnow/nexus/repository/apt/internal/proxy/AptProxyFacet.java#L80) since for the use cases I cared about when I wrote this code the assumption in option 2 was fine. If you take a snapshot, that will actually do option 1: download a snapshot of the Release and Packages files, it just takes longer than perhaps we would want to block the Release request for. If someone wanted to create a PR that added an option to force download of package indexes whenever the Release file was fetched that would be a great addition I think. |
It does just proxy and store whatever files are fetched. If you want to freeze the manifests at a point in time you can create a snapshot. |
|
Oh I see. Now that I understand the problem, I think the snapshot feature makes more sense to me (I didn't really understand how it worked). Since the snapshot feature does #1 I think that is probably what I want to be using for my use case then. |
|
I've followed the instructions on the README of how to create a snapshot, but how does one use a snapshot. After creating Then I try adding |
|
I think you need to add the snapshot to the end of the url in sources.list, not the end of the line. The url part of the line should look exactly like the MKCOL url. |
|
Ah, yes sorry, that is what I mean (end of the URL). Trying to dig in a little, for example I am able to fetch When I put I get the following from apt-get update: |
|
Try adding a |
|
Ah, whoops on apt-get update output with For reference, if I remove |
|
Oh! Even though curl exits with 0, I think that MKCOL is failing: The response is and the response authentication realm is "Sonatype Nexus Repository Manager" so I think the nginx setup isn't getting in the way. Also, I tried the following directly on the machine itself: and got Notabily, the request.log entry does not show a username: for other entries in this log, the second dash is a username like "admin" or "josh" |
|
Hm... it seems to be some kind of interaction with the "Remote User Token" configuration. If I add Is that somehow intended? Posting files to the repository does not require adding the RUT header (normal |
|
Except that for some repositories it yields a null pointer exception: |
|
@DarthHater - Any idea on what could cause the "Remote User Token" problem? I don't use that, and the plugin doesn't do anything specific with it, so pretty clueless on that. As far as the exception: I looked at the code, and it seems like this happens if there's no |
|
Here's the metadata tree for "packages.cloud.google.com" as viewed on our nexus deployment. So it has
My goal today is to try to disable RUT and see if snapshot creation works using |
|
Ok yeah, I disabled RUT and now however So when I add the following to my and run I appreciate all the help you've provided on this thread so far. To summarize I think we've identified (potentially) three issues so far:
I would be happy to break these out into separate issue reports on github if you'd like. At this point they appear to be unrelated to each other. |
|
Yeah, definitely think they're unrelated. For point 2, I filed an issue to improve the logging, but the root cause is almost certainly what you mentioned: the distribution in your config does not match the folder name in For point 3: it's expected. The i18n and translations are huuuuuge, and I wanted to keep snapshots as small as possible. For my use cases these files are unneeded, and the install works fine without them. Seems like a reasonable thing to put behind a config (i.e. |
|
Ok great. Edit: disable apt translations: https://askubuntu.com/a/74663 |
|
I guess appstream (DEP-11) files are also not snapshotted? I needed to do this as well in order to use a snapshot of a the proxy for |
|
Alright, I've successfully implemented apt configurations using snapshots for both workstation configurations and container/image builds... so I think I'm pretty much covered for my use case here. Thanks so much for writing the plugin and for taking the time to help me through these issues. I'm comfortable closing this issue unless you wish to keep it open for some reason. I know that @DarthHater mentioned updating documentation for this plugin. I think there were a few "gotchas" covered in this thread that might be worth adding to docs:
|
|
APT is now part of Nexus Repository Manager. Version 3.17.0 includes the APT plugin by default. |
Thanks for writing and sharing the apt repository plugin.
Provide a persistent proxy of upstream apt repositories for ubuntu which doesn't delete old packages. Today we ran into an issue with a proxy repository for
http://dl.google.com/linux/chrome/deb/. The filemetadata/dists/stable/main/Releasewas stale. It was "last verified" on2018-03-21. It contained an invalid hash sum formetadata/dists/stable/main/binary-amd64/Packages.bz2. That file was "last verified" on2018-03-23.Workstations configured with the nexus server in
/etc/apt/sources.listwere failing toapt-get updatewith:I manually deleted the
Releasefile through the nexus UI and after doing anapt-get updatefrom a workstation this file was pulled into nexus at the correct (newer) version.What feature or behavior is this required for?
Reproducible system image building
How could we solve this issue? (Not knowing is okay!)
I'm not sure exactly but it seems that possibly metadata files are cached with a policy that ignores the relationship between them. Without knowing the full mechanisms of the debian repository format, my suggestion would be to consider the file
Releaseand all of thePackages.bz-like files it points to as a single "unit". Update them all together, or none at all.Anything else?
I don't know exactly how the plugin manages the metadata cache and I don't understand the meaning of the "maximum metadata age" parameter. Does the plugin just exactly proxy the file contents from the upstream server? Or does it parse the upstream file and build a local database?
If it just proxies the upstream server files, is it not possible to install old packages that are currently cached in nexus but have been removed from the upstream?
The text was updated successfully, but these errors were encountered: