Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downloads /latest directory out of date #3034

Open
simonh-mmw opened this issue Jul 5, 2022 · 27 comments
Open

Downloads /latest directory out of date #3034

simonh-mmw opened this issue Jul 5, 2022 · 27 comments
Assignees
@dduportal
Labels
blobxfer bug Something isn't working get.jenkins.io update-center updateCenter updates.jenkins.io
Milestone
infra-team-sync-next

Comments

@simonh-mmw
Copy link

Service(s)

get.jenkins.io

Summary

Is there a reason why the /latest download does not match the currently latest LTS of 2.346.1?

https://get.jenkins.io/war-stable/latest/

Fixed URLs are really useful for a number of reasons, and if /latest is no longer to be used, should it not to removed?

Reproduction steps

  1. Latest published version https://get.jenkins.io/war-stable/2.346.1/jenkins.war
  2. Compare to https://get.jenkins.io/war-stable/latest/jenkins.war
@simonh-mmw simonh-mmw added the triage Incoming issues that need review label Jul 5, 2022
@timja
Copy link
Member

timja commented Jul 5, 2022

For the latest links to work you need to use the update center which has logic for that.

Weekly: https://updates.jenkins.io/latest/jenkins.war

Not sure exactly what the LTS url is, the script is here though:
https://github.com/jenkins-infra/update-center2/blob/master/site/generate.sh

@dduportal dduportal added this to the infra-team-sync-2022-07-12 milestone Jul 5, 2022
@dduportal dduportal added bug Something isn't working and removed triage Incoming issues that need review labels Jul 5, 2022
@MarkEWaite
Copy link

Thanks for the pointer to https://updates.jenkins.io @timja . The link to latest LTS is https://updates.jenkins.io/stable/latest/ with the jenkins.war at https://updates.jenkins.io/stable/latest/jenkins.war

@timja
Copy link
Member

timja commented Jul 5, 2022

This does raise though that ATH is pointing to get.Jenkins.io in run.sh and if that that’s not working we should swap to the update center, let’s keep it open to check that

@dduportal
Copy link
Contributor

Quick check before nighty night:

The only "culprit" is get.jenkins.io itself which has its own "fallback" file server.

@dduportal
Copy link
Contributor

On get.jenkins.io's file system (took one of the 3 "fileserver" pod randomly:

root@htdocs/war-stable# ls -l latest/
total 64896
-rwxrwxrwx 1 1000 1000 66452488 Aug 17  2020 jenkins.war
-rwxrwxrwx 1 1000 1000       77 Aug 17  2020 jenkins.war.sha256

while it is asymlink on archives.jenkins.io, or on updates.jenkins.io

$ ls -l latest
lrwxrwxrwx 1 www-data www-data 7 Jul  5 18:00 latest -> 2.346.1

=> either we should "delete" the directory latest/ and see if the "sync script" takes care of copying the correct symlink, or we might need to create this symlink and make sure that whatever script used for sync updates it

@dduportal dduportal self-assigned this Jul 6, 2022
@dduportal
Copy link
Contributor

As a general fact, there is none of the latest links on get.jenkins.io which are valid.

@dduportal
Copy link
Contributor

Temporarly removed the war-stable/latest directory to check wether or not the update_center script synchronizes the symlink.

=> https://get.jenkins.io/war-stable/latest/ is HTTP/404 for now!

@timja timja mentioned this issue Jul 6, 2022
Merged
@timja
Copy link
Member

timja commented Jul 6, 2022

I raised jenkinsci/acceptance-test-harness#812 on ATH

@dduportal
Copy link
Contributor

I raised jenkinsci/acceptance-test-harness#812 on ATH

Thinking aloud there but this ATh change should only be temporarly until we fix the issue. get.jenkins.io should still be considered as the default download source.

@dduportal
Copy link
Contributor

Update: https://get.jenkins.io/war-stable/latest/ is now showing the same content as the reference servers and mirrors 🥳

  • Step 1: deletion of the latest directory in the mirrorbits azure file bucket (mounted in R/W in the mirrorbits pods)
  • Step 2: wait for the update_center job (on trusted.ci) which runs freqeuntly (Every 5 min roughly) to finish with success and synchronize files.

Next step: I'll perform the same one-time operation to one of the plugins to see if it also fixes the issue

@timja
Copy link
Member

timja commented Jul 6, 2022

Will it work next time though 👅

@dduportal
Copy link
Contributor

Will it work next time though 👅

Good question: the filesystem does NOT have a symlink now, but a directory. I assume that whatever rsync / blobxfer command which does the synchronization is dereferencing the symlinks.

Let's check on plugins: easier to trigger a new release and see the result.

@dduportal
Copy link
Contributor

Plugin test:

repo/plugins/jenkins-infra-test$ ls -l
total 0
drwxrwxrwx 2 mirrorbits mirrorbits 0 Nov 16  2020 1.0
drwxrwxrwx 2 mirrorbits mirrorbits 0 Dec  1  2020 1.1
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 12.vb659278afc6b
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 13.v50f44d6f3875
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 25.vc6471c550e70
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 37.v560be292b3e4
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 42.v19b576f53ae3
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 25  2021 44.vb770b1e25577
drwxrwxrwx 2 mirrorbits mirrorbits 0 Aug 17  2021 48.v44dd7301178e
drwxrwxrwx 2 mirrorbits mirrorbits 0 Aug 27  2021 50.v756d280f8072
drwxrwxrwx 2 mirrorbits mirrorbits 0 Sep 10  2021 54.vd2265a5fa9a2
drwxrwxrwx 2 mirrorbits mirrorbits 0 Sep 10  2021 56.v40a08369fa8c
drwxrwxrwx 2 mirrorbits mirrorbits 0 Mar 27 15:56 58.vdf0feab90495
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun  2 21:38 62.v0d4f02c3969f
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 11 04:34 64.v21d9d793c858
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 20:45 69.v49b_5815e8158
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 22:32 71.vdb_a_65780b_f0a_
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 13 23:42 73.v10b_5ff00a_a_d6
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 14 00:32 75.v3d8c8a_c0177e
drwxrwxrwx 2 mirrorbits mirrorbits 0 Jun 14 18:32 78.veb_b_67886f8b_9
drwxrwxrwx 2 mirrorbits mirrorbits 0 Dec  9  2020 latest
repo/plugins/jenkins-infra-test$ ls -l latest/
total 6
-rwxrwxrwx 1 mirrorbits mirrorbits 5127 Dec  9  2020 jenkins-infra-test.hpi

@dduportal
Copy link
Contributor

=> then: we'll trigger a new release of the plugin and see what happens.

@timja
Copy link
Member

timja commented Jul 6, 2022

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

@dduportal
Copy link
Contributor

=> then: we'll trigger a new release of the plugin and see what happens.

@dduportal
Copy link
Contributor

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

That is ... unexpected 🤔

@timja
Copy link
Member

timja commented Jul 6, 2022

Next step: let's trigger a new plugin release (@timja what is process: should I open a new dummy PR on it and once merged it will release? Or is there another way?) for jenkinsci/jenkins-infra-test-plugin

Yes do that, make sure you label it with enhancement

@dduportal
Copy link
Contributor

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

It's another issue: sounds like the weekl packaging process is stuck. Just received pagerdutty alerts about this.

@dduportal
Copy link
Contributor

FTR https://updates.jenkins.io/latest/jenkins.war is currently returning 404 as it's pointing to https://get.jenkins.io/war/2.358/jenkins.war which doesn't exist yet

#3037

This was referenced Jul 6, 2022
Merged
Closed
@dduportal
Copy link
Contributor

@dduportal
Copy link
Contributor

OK, so confirmation that, once the latest/ directory exists, then the "sync script" does not try to update it on the azurefile bucket:

[latest/](https://get.jenkins.io/plugins/jenkins-infra-test/latest/)                 2022-07-06 10:16    -   
[80.v441ea_08a_25b_e/](https://get.jenkins.io/plugins/jenkins-infra-test/80.v441ea_08a_25b_e/)    2022-07-06 13:05    -   
[78.veb_b_67886f8b_9/](https://get.jenkins.io/plugins/jenkins-infra-test/78.veb_b_67886f8b_9/)    2022-06-14 18:32    -

=> It means that we should not repeat the manual operation that I did earlier on the production data volume, but instead update the script of the update_center repository so that it overrides the latest directories when uploading to get.jenkins.io

@dduportal
Copy link
Contributor

dduportal commented Jul 7, 2022 •
edited
Loading

Had to deep on the machine pkg.origin.jenkins.io history since it was manually managed for months before we put it back under puppet management.
The goal is to understand "what process" is in charge of syncing the recent changes to get.jenkins.io's files.

# Puppet Name: mirrorbrain-sync-releases
0 * * * * cd /srv/releases && ./sync.sh --full-sync > last_sync.log
$ grep 'latest' last_sync.log | head -n20
>> Updating the latest symlink for weekly
>> Updating the latest symlink for weekly RC
>> Updating the latest symlink for LTS
>> Updating the latest symlink for LTS RC
2022-07-07 16:02:17,154 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war (local: /srv/releases/jenkins/war-stable/latest/jenkins.war)
2022-07-07 16:02:17,173 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable/latest/jenkins.war.sha256)
2022-07-07 16:02:57,462 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war (local: /srv/releases/jenkins/war/latest/jenkins.war)
2022-07-07 16:02:57,480 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war/latest/jenkins.war.sha256)
2022-07-07 16:03:26,270 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war)
2022-07-07 16:03:26,288 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war.sha256)
2022-07-07 16:03:28,979 INFO - not overwriting remote file: mirrorbits/war-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-rc/latest/jenkins.war)
2022-07-07 16:03:36,088 INFO - not overwriting remote file: mirrorbits/plugins/gitlab-branch-source/latest/gitlab-branch-source.hpi (local: /srv/releases/jenkins/plugins/gitlab-branch-source/latest/gitlab-branch-source.hpi)
2022-07-07 16:03:36,593 INFO - not overwriting remote file: mirrorbits/plugins/appray/latest/appray.hpi (local: /srv/releases/jenkins/plugins/appray/latest/appray.hpi)
2022-07-07 16:03:36,651 INFO - not overwriting remote file: mirrorbits/plugins/disable-github-multibranch-status/latest/disable-github-multibranch-status.hpi (local: /srv/releases/jenkins/plugins/disable-github-multibranch-status/latest/disable-github-multibranch-status.hpi)
2022-07-07 16:03:36,738 INFO - not overwriting remote file: mirrorbits/plugins/r/latest/r.hpi (local: /srv/releases/jenkins/plugins/r/latest/r.hpi)
2022-07-07 16:03:36,876 INFO - not overwriting remote file: mirrorbits/plugins/nant/latest/nant.hpi (local: /srv/releases/jenkins/plugins/nant/latest/nant.hpi)
2022-07-07 16:03:37,031 INFO - not overwriting remote file: mirrorbits/plugins/policycenter-gate-validator/latest/policycenter-gate-validator.hpi (local: /srv/releases/jenkins/plugins/policycenter-gate-validator/latest/policycenter-gate-validator.hpi)
2022-07-07 16:03:37,150 INFO - not overwriting remote file: mirrorbits/plugins/deployed-on-column/latest/deployed-on-column.hpi (local: /srv/releases/jenkins/plugins/deployed-on-column/latest/deployed-on-column.hpi)
2022-07-07 16:03:37,674 INFO - not overwriting remote file: mirrorbits/plugins/tasks/latest/tasks.hpi (local: /srv/releases/jenkins/plugins/tasks/latest/tasks.hpi)
2022-07-07 16:03:39,599 INFO - not overwriting remote file: mirrorbits/plugins/github/latest/github.hpi (local: /srv/releases/jenkins/plugins/github/latest/github.hpi)


$ grep 'latest' last_sync.log | wc -l
2198

$ grep 'latest' last_sync.log | grep -v '/plugins/'
>> Updating the latest symlink for weekly
>> Updating the latest symlink for weekly RC
>> Updating the latest symlink for LTS
>> Updating the latest symlink for LTS RC
2022-07-07 16:02:17,154 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war (local: /srv/releases/jenkins/war-stable/latest/jenkins.war)
2022-07-07 16:02:17,173 INFO - not overwriting remote file: mirrorbits/war-stable/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable/latest/jenkins.war.sha256)
2022-07-07 16:02:57,462 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war (local: /srv/releases/jenkins/war/latest/jenkins.war)
2022-07-07 16:02:57,480 INFO - not overwriting remote file: mirrorbits/war/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war/latest/jenkins.war.sha256)
2022-07-07 16:03:26,270 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war)
2022-07-07 16:03:26,288 INFO - not overwriting remote file: mirrorbits/war-stable-rc/latest/jenkins.war.sha256 (local: /srv/releases/jenkins/war-stable-rc/latest/jenkins.war.sha256)
2022-07-07 16:03:28,979 INFO - not overwriting remote file: mirrorbits/war-rc/latest/jenkins.war (local: /srv/releases/jenkins/war-rc/latest/jenkins.war)
2022-07-07 16:17:05,317 INFO - not overwriting remote file: mirrorbits/windows/latest (local: /srv/releases/jenkins/windows/latest)
2022-07-07 16:17:20,527 INFO - not overwriting remote file: mirrorbits/windows-stable/latest (local: /srv/releases/jenkins/windows-stable/latest)

@dduportal
Copy link
Contributor 

$ pip list | grep blobxfer 
blobxfer (1.6.0)
$ blobxfer  --version
blobxfer, version 1.6.0
$ dpkg -l | grep -c blobxfer # No package
0

@dduportal dduportal mentioned this issue Jul 7, 2022
Closed
@dduportal
Copy link
Contributor

Ping @olblak @daniel-beck @timja do you have any memory if blobxfer supports symlinks?
Wondering because if it does not, then will have to add a specific blobxfer command to check the latest md5 with a force-override

@smerle33 smerle33 modified the milestones: infra-team-sync-2022-07-26, infra-team-sync-2022-08-02, infra-team-sync-next Jul 27, 2022
@dduportal dduportal mentioned this issue Aug 19, 2022
Closed
@dduportal
Copy link
Contributor

Also, as per #3100 (closed as duplicate), @jnord raised the following issue:

Summary

According to the timestamps https://get.jenkins.io/war/latest/jenkins.war was last updated in 2020
This is not the case and if you download the file you can see it is 2.364 (which is the latest at the time of writing).
However this causes confusion as it appears it has not been updated.

Reproduction steps:

go to https://get.jenkins.io/war/
Notice the modification data on latest (possibly ok if this is a directory and the children are symlink)s.
go to https://get.jenkins.io/war/latest/
Notice the modification date on the files is still in 2020
Expected result, the timestamps shown in the web interface correspond to (approximatly) the time of publishing of the war.

@lemeurherve
Copy link
Member

if blobxfer supports symlinks?

azcopy supports them: https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy

--follow-symlinks Follow symbolic links when uploading from local file system.

@lemeurherve lemeurherve mentioned this issue Sep 11, 2022
Open
@lemeurherve lemeurherve mentioned this issue Feb 27, 2023
Closed
@lemeurherve lemeurherve mentioned this issue Dec 12, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dduportal dduportal

Labels
blobxfer bug Something isn't working get.jenkins.io update-center updateCenter updates.jenkins.io
Projects
None yet
Milestone
infra-team-sync-next
Development

No branches or pull requests
6 participants
@MarkEWaite @dduportal @timja @simonh-mmw @lemeurherve @smerle33