Decomission Koji setup

[ekohl]

With the move to COPR (#1795) the Koji setup is no longer needed. Today we still build Foreman 3.7 & 3.8, but after those are EOL the infrastructure can be decommissioned.

[ehelms]

If I do the math correctly, that means with the 3.11 release we can take on this work.

[evgeni]

I think that's correct.

Technically we also have to consider Katello 4.10 and whatever Pulpcore was before 3.39, but luckily all those align together.

[ekohl]

I think these get rid of everything that requires Koji:

• Remove Pulpcore 3.28 pipelines jenkins-jobs#435
• Drop Foreman 3.8/Katello 4.10 pipelines & Koji related code jenkins-jobs#436
• Drop pulp-installer pipelines and switch Pulp 3.39 to puppet-pulpcore forklift#1799
• Drop koji support forklift#1800

[ekohl]

When we've removed the VMs, we should make sure koji.katello.org is also removed.

[ekohl]

Expanding on my previous comment, I think it's a good time to consider what the full plan is. First of all, can we start turning off koji itself?

http://koji.katello.org/koji shows the last build was a month and a half ago, so that's good. Then the question is whether we need anything from http://koji.katello.org/releases/.

Currently we have:

/dev/xvda1      8.0G  3.5G  4.6G  44% /
/dev/nvme0n1p2  505G  274G  206G  58% /mnt/tmp
/dev/xvdx1     1008G  939G   19G  99% /mnt/koji

So that's not easy to just dump somewhere. Another thing to note is that /mnt/tmp is empheral and will be deleted. Quoting /mnt/tmp/README:

This volume is ephemeral, stopping the VM immediately deletes it.

It only contains temporary and work files, but we keep backups at
/mnt/koji/backups/ephemeral (which is AWS EBS volume). The backup
script is at:

/etc/cron.weekly/koji-backup

To restore do this:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force --no-encryption

The backup skips filenames with RPM extension to keep the backup
clean and small - the most important are directories and permissions.

Perhaps we can copy over the backups:

# du -sh /mnt/koji/backups/
209G	/mnt/koji/backups/

But it should be noted that those have file timestamps back to 2017 so it's probably sufficient to take the latest ones.

As for storage to place it: we have 650 GB of unallocated storage at OSUOSL.

[evgeni]

Expanding on my previous comment, I think it's a good time to consider what the full plan is. First of all, can we start turning off koji itself?

I think so, yes.

http://koji.katello.org/koji shows the last build was a month and a half ago, so that's good.

That correlates with theforeman/jenkins-jobs#436 that dropped the support of being able to build there ;)

Then the question is whether we need anything from http://koji.katello.org/releases/.

I'd argue we have everything we need on yum.theforeman.org?

Currently we have:

/dev/xvda1      8.0G  3.5G  4.6G  44% /
/dev/nvme0n1p2  505G  274G  206G  58% /mnt/tmp
/dev/xvdx1     1008G  939G   19G  99% /mnt/koji

So that's not easy to just dump somewhere. Another thing to note is that /mnt/tmp is empheral and will be deleted. Quoting /mnt/tmp/README:

This volume is ephemeral, stopping the VM immediately deletes it.

It only contains temporary and work files, but we keep backups at
/mnt/koji/backups/ephemeral (which is AWS EBS volume). The backup
script is at:

/etc/cron.weekly/koji-backup

To restore do this:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force --no-encryption

The backup skips filenames with RPM extension to keep the backup
clean and small - the most important are directories and permissions.

Perhaps we can copy over the backups:

# du -sh /mnt/koji/backups/
209G	/mnt/koji/backups/

But it should be noted that those have file timestamps back to 2017 so it's probably sufficient to take the latest ones.

As for storage to place it: we have 650 GB of unallocated storage at OSUOSL.

If we only take things from 2024, it's 13G.
But given it does not include the RPMs, I don't see much value in the backups outside the existing install?

[ekohl]

But given it does not include the RPMs, I don't see much value in the backups outside the existing install?

I was leaning the same way. So a concrete plan:

• 2024-05-06 Announce plan on Discourse
• 2024-05-13 turn machine(s) off
• 2024-06-01 delete machine(s)
  • Remove AWS machine(s) (IIRC @pcreech has access)
  • Remove DNS records like koji.katello.org (IIRC managed by Red Hat IT)
  • Remove from Foreman (multiple people have access)

[evgeni]

@pcreech has shut down the machine last Friday.

[evgeni]

@pcreech you can go ahead and delete it

TODO:

• DNS
• any readmes/docs referring to koji need to be cleaned up

[pcreech]

@evgeni Deleted. Koji is gone.

[evgeni]

Awesome.

Do you know who I need to pester about katello dot org DNS?

[evgeni]

@ekohl please file a ticket with RH IT

[ekohl]

Ticket has been filed and resolved. The DNS record is gone and with that I consider this resolved.