[INFRA-3100] Migrate updates.jenkins.io to another Cloud

[jenkins-infra-bot]

Why

Read the EPIC (aws cost decrease: #2646)

What

• updates.jenkins.io serves the JSON of the update center, but it is not behind a CDN (need to be updated regularly) causing a lot of outbound data transfer (2.4 Mb for a JSON download, esitmation around 300 Gb of data sent daily, estimation around 10 Tb per month which costs some $$$.

• The VM pkg.origin.jenkins.io is managing multiple services and separting them could be a great help.

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

• Migrating the service to Oracle Cloud:
  • the first 10 Tb of outbound data is free, and if the worst case, 50 Tb of outbound would cost less than 3k $ per month - https://www.oracle.com/be/cloud/networking/networking-pricing.html
  • ARM machines: cheaper and better performance for a simple webserver + SSH + data
  • Block storage: close to archives.jenkins.io
  • Risk: Oracle sponsoring program is only 1 year, and is not easily secured for paiment as other account

• Migrating to Azure
  • Easily in AKS: easier management
  • Safer paiment process
  • Risk: cost of outbound to be evaluated

---

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[note]

• Check that Claim that updates.jenkins.io certificate is sometimes expiring #3017 is not happening once migration is finished.
• The error curl: (16) Error in the HTTP2 framing layer should not happen anymore with the latest Apache versions

[dduportal]

Requires setting up the Oracle Terraform project, like #2682

Todo list:

• Specify a new infra (VM + data storage of the same size as the actual pkg.origin.jenkins.io machine + any other Oracle infra requirements). Implies checking the pricing to get an overall idea
• Once VM is created, add it to puppet management: new node, with the role / profiles associated to "update.jenkins.io"
• Rsync the data one time
• Update the jenkins-infra/update-center2 's associated job in trusted.ci to update both "pkg.origin.jenkins.io" VM and the new one (⚠️ inline pipeline, don't search for a Jenkinsfile as code)
• Validate the new instance (including checking with Daniel, Tim and other contributors)
• Communicate about the upcoming change
• As proposed by Stephane, use a round-robin DNS record value to split traffic between old/new during some time and check for error

[dduportal]

Blocked by #2973

[smerle33]

actual machine size is :
32Gb RAM
8cpu
1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

[smerle33]

Infra to specify :

• 1 VM (similar to archive.jenkins.io)
• 1 network ("mirrors") + 1 subnet ("20210630-1531")
• 1 volume 1,2Tb
• 1 set of security groups to restrict network access
• 1 ssh key pair

[smerle33]

VM specifications :

• 4vCPU/16Gb RAM : half of actual machine
• proposal to use ARM like archive.jenkins.io --> 4ocpu (VM Type Standard A1 Flex)
• Image Ubuntu 20.04 (FULL [non minimal]) --> upgrade from ubuntu 18.04 for actual machine
  • option for 22.04 but need testing for puppet agent (+ openssl v3)

[dduportal]

Post mortem of the fourth brownout (26 -> 27 September):

• ✅ Same results as previous brownouts regarding resource usage and resiliency of the new system: we're good on this scope

• ✅ The new HTTP component worked as expected: https://issues.jenkins.io/browse/JENKINS-73760?focusedId=449550&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-449550

• As per https://groups.google.com/g/jenkinsci-dev/c/XD-kPbDqaRA/m/oIQBLXdDAAAJ (user report in the mailing list), there were caught by 2 issues:

  • ⚠️ There received HTTP/403 (Forbidden) errors on requests such as https://updates.jenkins.io/updates/hudson.tasks.Maven.MavenInstaller.json.html. Problem was fixed during the brownout
    • ✅ Redirection were happening as expected: the URL receiving HTTP/403 was Cloudflare: http://westeurope.cloudflare.jenkins.io/updates/hudson.tasks.Maven.MavenInstaller.json.html
    • ⚠️ Looking at the Cloudflare settings, we discovered that there is a WAF with an default security setting "Browser Integrity Check" which was triggered by the HTTP requests headers of this user (and many more). We disabled the BIC as it is not expected to help us, which fixed the HTTP/403
  • 💥 They also reported that the HTML pages listing plugins versions had their links resulting in HTTP/404. Example: https://westeurope.cloudflare.jenkins.io/download/plugins/gradle/index.html. This is an unexpected behavior which we did not catch earlier as it's client side. Discussion on this in [Update Center] Fix HTTP/404 errors due to broken links in HTML listing pages and missing ?uctest endpoint #4311

• As reported by @daniel-beck, the new system is missing the "?uctest feature": the URL https://updates.jenkins.io/update-center.json?uctest is expected to answer (HTTP/200) with an empty JSON, which does not work on the new system

  • 💥 We have to fix this behavior. If we set it up at the httpd levels, it could be useful to provide readiness healthcheck.
  • ⚠️ We failed to check the new UC system with the update_center2 test.sh. We must do it to detect other endpoints we could have missed

• 💥 Four times during the brownout, our Pagerduty system did alert us about Apache answering HTTP/404 errors instead of usual redirections. The root cause is the htdocs/ directory, filled every 10 min by update_center2, which is a SMB/CIFS share (default for Azure file share PVCs in AKS. All files in this directory suddenly were shown as empty. See [Update Center - Azure] HTTP/404 responses instead of redirections with error AH01630: client denied by server configuration #4312 for details and follow-up.

  • We must find a solution for this one as it is considered a major blocker.

[dduportal]

Update:

• [Update Center] Fix HTTP/404 errors due to broken links in HTML listing pages and missing ?uctest endpoint #4311 has been fixed with the help of @daniel-beck (❤️ )
• [Update Center - Azure] HTTP/404 responses instead of redirections with error AH01630: client denied by server configuration #4312 has been worked on: no NFS possible (requires huge architectural changes), SMB+Apache tuning applied. If it fails, then we'll have to change Apache (and better to use local htdocs dir instead of PVC on the network).
• Announcing the 5th brownout:
  • Blog post: feat(blog) add a new post about the Update Center (5th brownout) jenkins.io#7628
  • Mailing lists (users, dev and infra): https://groups.google.com/g/jenkinsci-dev/c/AqCdi2Tyfvw/m/uBybpi80BAAJ
  • Matrix message on jenkins-infra and jenkinsci/jenkins
  • status.jenkins.io: open 5th UC brownout status#547
• DNS record change: feat(dns records) start 5th brownout for UC azure-net#306

[dduportal]

Fifth brownout finished! (ref. jenkins-infra/azure-net#309)