frontend: automatically EOL lifeless rolling chroots

When rolling policy finishes, we want to remove the data, but we also
want to mark the CoprChroot.deleted (disable it from the project)
because the corresponding mock chroot is still active in Copr.

When user does a new build, we reset the CoprChroot
delete_after/notification state, always.  This is ok because during the
normal MockChroot EOL policy, we do not allow new builds.

Implements: https://github.com/fedora-copr/debate/blob/main/2024-04-23-rawhide-chroots-eol.md
Fixes: #2933

doc/conf.py

@@ -281,6 +281,31 @@
 .. |se| raw:: html
 
     </strike>
+
+.. |br| raw:: html
+
+    <br/>
+
+.. |p_red| raw:: html
+
+    <p style="background-color: #f1948a;">
+
+.. |p_end| raw:: html
+
+    </p>
+
+.. |p_yel| raw:: html
+
+    <p style="background-color: #ffbf00;">
+
+.. |p_gre| raw:: html
+
+    <p style="background-color: #80ff00;">
+
+.. |p_gra| raw:: html
+
+    <p style="background-color: #d1e0e0;">
+
 """
 
 # Documents to append as an appendix to all manuals.

doc/developer_documentation.rst

@@ -40,6 +40,7 @@ Misc
 
    Brainstorming <brainstorming>
    Populate DB <seeddb>
+   Chroot EOL implementation <developer_documentation/eol-logic>
 
 
 The design and diagrams

doc/developer_documentation/eol-logic.rst

@@ -0,0 +1,48 @@
+.. _eol_logic:
+
+Handling EOL CoprChroot entries
+-------------------------------
+
+There are currently three cases when we schedule a CoprChroot for removal but
+preserve the data for some time to allow users to recover:
+
+    1. User disables the chroot in a project ("unclicks" the checkbox).  We give
+       them 14 days "preservation period" to reverse the decision without
+       forcing them to rebuild everything.
+    2. Copr Admin makes a chroot EOL, e.g., `fedora-38-x86_64` because the
+       Fedora 38 version goes EOL.  We keep the builds for several months, and
+       users can extend the preservation period.
+    3. We disable rolling chroots (e.g., Fedora Rawhide or Fedora ELN) after a
+       reasonable period of inactivity.
+
+There are three database fields used for handling the EOL/preservation policies:
+``CoprChroot.deleted`` (bool), ``CoprChroot.delete_after`` (timestamp),
+and ``MockChroot.is_active`` (bool, 1:N mapped to ``CoprChroot``).  The
+following table describes certain implications behind the logic:
+
+
+.. table:: Logical implications per in-DB chroot state
+
+
+    =========   ============    ======= ====================    =========       ===========
+    is_active   delete_after    deleted e-mail |br|             can build       State && |br| Description
+                                        notifications
+    =========   ============    ======= ====================    =========       ===========
+    yes         yes             yes     no                      no              |p_yel| ``preserved`` PM - preserved - manual removal |p_end|
+    yes         no              yes     --                      no              |p_red| ``deleted`` manual removal or rolling removal (or EOL removal, and reactivated) |p_end|
+    yes         yes             no      yes                     yes             |p_yel| ``preserved`` rolling |p_end|
+    yes         no              no      --                      yes             |p_gre| ``active`` normal chroot state |p_end|
+    no          yes             yes     no                      no              |p_yel| ``preserved`` (deleted manualy, then mock chroot EOL or deactivation) |p_end|
+    no          no              yes     --                      no              |p_red| ``deleted`` manually OR rolling deleted, and THEN EOLed/deactivated by Copr admin |p_end|
+    no          yes             no      yes                     no              |p_yel| ``preserved`` mock chroot EOLed by Copr admin |p_end|
+    no          no              no      --                      no              |p_gra| ``deactivated`` deactivated by Copr admin, data preserved |p_end|
+    =========   ============    ======= ====================    =========       ===========
+
+There's also a chroot state ``expired``, which is a special state of
+the ``preserved`` state.  It is "still preserved", but the time for removal is
+already there, namely ``now() >= delete_after``.
+
+Note that when ``e-mail notifications`` are ``yes``, the time for removal has
+come (``now() >= delete_after``) and we **were not** able to send the
+notification e-mail, we **don't** remove the chroot data.  **No unexpected
+removals.**

doc/how_to_manage_chroots.rst

@@ -20,6 +20,7 @@ Chroots can be easily managed with these few commands.
     copr-frontend branch-fedora <new-branched-version>
     copr-frontend rawhide-to-release <rawhide-chroot> <newly-created-chroot>
     copr-frontend chroots-template [--template PATH]
+    copr-frontend eol-lifeless-rolling-chroots
 
 However, `enablement process upon Fedora branching <#branching-process>`_ and also
 `chroot deactivation when Fedora reaches it's EOL phase <#eol-deactivation-process>`_, are not that simple.
@@ -110,6 +111,22 @@ When it is done, `send an information email to a mailing list <#mailing-lists>`_
 See the :ref:`the disable chroots template <disable_chroots_template>`.
 
 
+Rawhide (and other rolling) chroots EOL
+---------------------------------------
+
+Run ``copr-frontend eol-lifeless-rolling-chroots`` to mark existing rolling Copr
+chroots for the future removal/deactivation — if they appear lifeless.  You
+might want to run this daily in the ``copr-frontend-optional`` cron-job file.
+The logic in this command checks that no build happened in particular rolling
+chroot for a long time, so likely no work is being done there, and the old built
+packages are _likely_ non-installable anyway (as the rolling distro moves
+forward with dependencies, but no dependency resolution is being done with
+RPMs).  If such a chroot is marked EOL, this command applies the same
+notification policy/process as with the :ref:`eol_deactivation_process` so users
+can keep the chroot alive (either by prolonging the chroot, or triggering a new
+build).
+
+
 .. _managing_chroot_comments:
 
 Managing chroot comments

frontend/conf/cron.daily/copr-frontend-optional

@@ -2,6 +2,7 @@
 
 # Optional Copr frontend tasks to be executed daily.
 
+#runuser -c 'copr-frontend eol-lifeless-rolling-chroots' - copr-fe
 #runuser -c 'copr-frontend notify-outdated-chroots' - copr-fe
 #runuser -c 'copr-frontend delete-outdated-chroots' - copr-fe
 #/usr/libexec/copr_dump_db.sh /var/lib/copr/data/db_dumps/

frontend/coprs_frontend/alembic/versions/d23f84f87130_eol_rolling_builds.py

@@ -0,0 +1,30 @@
+"""
+record last copr_chroot build, allow marking Rawhide as rolling
+Create Date: 2024-05-13 08:56:31.557843
+"""
+
+import time
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.sql import text
+
+revision = 'd23f84f87130'
+down_revision = '9fec2c962fcd'
+
+def upgrade():
+    op.add_column('mock_chroot', sa.Column('rolling', sa.Boolean(), nullable=True))
+    op.add_column('copr_chroot', sa.Column('last_build_timestamp', sa.Integer(), nullable=True))
+    conn = op.get_bind()
+    conn.execute(
+        text("update copr_chroot set last_build_timestamp = :start_stamp;"),
+        {"start_stamp": int(time.time())},
+    )
+    op.create_index('copr_chroot_rolling_last_build_idx', 'copr_chroot',
+                    ['mock_chroot_id', 'last_build_timestamp', 'delete_after'],
+                    unique=False)
+
+
+def downgrade():
+    op.drop_index('copr_chroot_rolling_last_build_idx', table_name='copr_chroot')
+    op.drop_column('copr_chroot', 'last_build_timestamp')
+    op.drop_column('mock_chroot', 'rolling')

frontend/coprs_frontend/commands/eol_lifeless_rolling_chroots.py

@@ -0,0 +1,14 @@
+"""
+copr-frontend eol-lifeless-rolling-chroots command
+"""
+
+import click
+from coprs.logic.outdated_chroots_logic import OutdatedChrootsLogic
+
+@click.command()
+def eol_lifeless_rolling_chroots():
+    """
+    Go through all rolling CoprChroots and check whether they shouldn't be
+    scheduled for future removal.
+    """
+    OutdatedChrootsLogic.trigger_rolling_eol_policy()

frontend/coprs_frontend/config/copr.conf

@@ -207,6 +207,16 @@ HIDE_IMPORT_LOG_AFTER_DAYS = 14
 # failed builds.  Currently only integrated with the Fedora Copr instance.
 #LOG_DETECTIVE_BUTTON = False
 
+# After a given _INACTIVITY_WARNING period (DAYS), when no new build appeared
+# appeared in a rolling chroot, we start the "rolling" EOL policy.  It means
+# that - after the next _INACTIVITY_REMOVAL DAYS - the chroot gets disabled, and
+# the corresponding builds are removed to save storage.  This is only applicable
+# to MockChroots.rolling = True, and the command
+# `copr-frontend eol-lifeless-rolling-chroots` must be executed periodically via
+# cron or similar.
+#ROLLING_CHROOTS_INACTIVITY_WARNING = 180
+#ROLLING_CHROOTS_INACTIVITY_REMOVAL = 180
+
 #############################
 ##### DEBUGGING Section #####
 

frontend/coprs_frontend/coprs/config.py

@@ -198,6 +198,10 @@ class Config(object):
 
     LOG_DETECTIVE_BUTTON = False
 
+    ROLLING_CHROOTS_INACTIVITY_WARNING = 180
+    ROLLING_CHROOTS_INACTIVITY_REMOVAL = 180
+
+
 class ProductionConfig(Config):
     DEBUG = False
     # SECRET_KEY = "put_some_secret_here"

frontend/coprs_frontend/coprs/helpers.py

@@ -202,6 +202,8 @@ class ChrootDeletionStatus(metaclass=EnumType):
     When a chroot is marked as EOL or when it is unclicked from a project,
     it goes through several stages before its data is finally deleted.
     Each `models.CoprChroot` is in one of the following states.
+
+    See https://docs.pagure.org/copr.copr/developer_documentation/eol-logic.html
     """
     # pylint: disable=too-few-public-methods
     vals = {
@@ -213,14 +215,10 @@ class ChrootDeletionStatus(metaclass=EnumType):
         # marked as EOL and its data is never going to be deleted.
         "deactivated": 1,
 
-        # There are multiple possible scenarios for chroots in this state:
-        # 1) The standard preservation period is not over yet. Its length
-        #    differs on whether the chroot is EOL or was unclicked from
-        #    a project but the meaning is same for both cases
-        #
-        # 2) If the chroot is EOL and we wasn't able to send a notification
-        #    about it.
-        #
+        # Preserved state.  There are multiple possible scenarios for chroots in
+        # this state:
+        # 1) The standard preservation period is not over yet.
+        # 2) If the chroot is EOL, but we weren't able to notify the user.
         # 3) Any other constraint that disallows the chroot to be deleted yet.
         #    At this moment there shouldn't be any.
         "preserved": 2,

frontend/coprs_frontend/coprs/logic/actions_logic.py

@@ -398,6 +398,7 @@ def send_delete_chroot(cls, copr_chroot):
             created_on=int(time.time()),
             copr_id=copr_chroot.copr.id,
         )
+        copr_chroot.deleted = True
         db.session.add(action)
         return action
 

frontend/coprs_frontend/coprs/logic/builds_logic.py

@@ -1473,6 +1473,7 @@ def new(cls, build, mock_chroot, **kwargs):
             copr_chroot = coprs_logic.CoprChrootsLogic.get_by_mock_chroot_id(
                 build.copr, mock_chroot.id
             ).one()
+        copr_chroot.build_done()
         return models.BuildChroot(
             mock_chroot=mock_chroot,
             copr_chroot=copr_chroot,

frontend/coprs_frontend/coprs/logic/complex_logic.py

@@ -785,6 +785,7 @@ def _delete_reason(cls, copr_chroots):
         """
         In case some of the `copr_chroots` is going to be deleted in the future,
         describe the reason why
+        TODO: merge with the `CoprChroot.delete_status` property
         """
         # Do we even want to show a trash icon? I.e. are any of the project
         # chroots set to be deleted in the future?
@@ -798,13 +799,21 @@ def _delete_reason(cls, copr_chroots):
         # project owner. Or all of them for the same reason. Also all
         # architectures may be deleted by the project owner but on a different
         # day and therefore the remaining time may be different)
+        # See https://docs.pagure.org/copr.copr/developer_documentation/eol-logic.html
         reasons = {}
         for chroot in delete_chroots:
             reason_format = "{0} and will remain available for another {1} days"
-            reason = reason_format.format(
-                "disabled by a project owner" if chroot.is_active else "EOL",
-                chroot.delete_after_days,
-            )
+            reason = "EOL"
+            if chroot.is_active:
+                if chroot.deleted:
+                    # This is in `preserved` state, not `deleted`, otherwise we
+                    # wouldn't even ask for the _delete_reason().
+                    reason = "disabled by a project owner"
+                elif chroot.mock_chroot.rolling:
+                    reason = "a rolling chroot inactive for a long time"
+                else:
+                    raise exceptions.BadRequest(f"Unknown EOL reason {chroot.name}")
+            reason = reason_format.format(reason, chroot.delete_after_days)
             reasons.setdefault(reason, [])
             reasons[reason].append(chroot.mock_chroot.arch)
 

frontend/coprs_frontend/coprs/logic/coprs_logic.py

@@ -8,7 +8,7 @@
 
 import flask
 
-from sqlalchemy import not_
+from sqlalchemy import not_, or_
 from sqlalchemy import desc
 from sqlalchemy import func
 from sqlalchemy.event import listens_for
@@ -1251,18 +1251,22 @@ def remove_copr_chroot(cls, user, copr_chroot):
     @classmethod
     def filter_outdated(cls, query):
         """
-        Filter query to fetch only `CoprChroot` instances that are EOL but still
-        in the data preservation period
+        Filter query to fetch only `CoprChroot` instances that are in the data
+        preservation period, but not yet expired (not yet to be removed).  Used
+        for sending e-mail notifications.
+
+        See https://docs.pagure.org/copr.copr/developer_documentation/eol-logic.html
         """
         return (query.filter(models.CoprChroot.delete_after
                              >= datetime.datetime.now())
-                     # Filter only such chroots that are not unclicked (deleted)
-                     # from a project. We don't want the EOL machinery for them,
-                     # they are deleted.
+                     # Filter out chroots that are manually disabled by user
+                     # (deleted, aka "unclicked", we never send e-mails there)
                      .filter(models.CoprChroot.deleted.isnot(True))
-
-                     # Filter only inactive (i.e. EOL) chroots
-                     .filter(not_(models.MockChroot.is_active)))
+                     .filter(or_(
+                         # inactive (i.e. EOL) chroots
+                         not_(models.MockChroot.is_active),
+                         # rolling EOL candidate (still active, though)
+                         models.MockChroot.rolling.is_(True))))
 
 
     @classmethod

frontend/coprs_frontend/coprs/logic/outdated_chroots_logic.py

@@ -83,10 +83,43 @@ def expire(cls, copr_chroot):
         cls._update_copr_chroot(copr_chroot, delete_after_days)
 
     @classmethod
-    def _update_copr_chroot(cls, copr_chroot, delete_after_days):
+    def _update_copr_chroot(cls, copr_chroot, delete_after_days, actor=None):
         delete_after_timestamp = (
             datetime.now()
             + timedelta(days=delete_after_days)
         )
-        CoprChrootsLogic.update_chroot(flask.g.user, copr_chroot,
+
+        if actor is None:
+            actor = flask.g.user
+
+        CoprChrootsLogic.update_chroot(actor, copr_chroot,
                                        delete_after=delete_after_timestamp)
+
+
+    @classmethod
+    def trigger_rolling_eol_policy(cls):
+        """
+        Go through all the MockChroot.rolling -> CoprChroots, and check when the
+        last build has been done.  If it is more than
+        config.ROLLING_CHROOTS_INACTIVITY_WARNING days, mark the CoprChroot for
+        removal after ROLLING_CHROOTS_INACTIVITY_REMOVAL days.
+        """
+
+        period = app.config["ROLLING_CHROOTS_INACTIVITY_WARNING"] * 24 * 3600
+        warn_timestamp = int(datetime.now().timestamp()) - period
+
+        query = (
+            db.session.query(models.CoprChroot).join(models.MockChroot)
+            .filter(models.MockChroot.rolling.is_(True))
+            .filter(models.MockChroot.is_active.is_(True))
+            .filter(models.CoprChroot.delete_after.is_(None))
+            .filter(models.CoprChroot.last_build_timestamp.isnot(None))
+            .filter(models.CoprChroot.last_build_timestamp < warn_timestamp)
+        )
+
+        when = app.config["ROLLING_CHROOTS_INACTIVITY_REMOVAL"]
+        for chroot in query:
+            cls._update_copr_chroot(chroot, when,
+                                    models.AutomationUser("Rolling-Builds-Cleaner"))
+
+        db.session.commit()