Development: Document Jenkins and LocalVC setup (#9266)

README.md

@@ -139,8 +139,9 @@ You can find a guide on [how to write documentation](docs/README.md).
 Setting up Artemis in your development environment or a demo production environment is really easy following the instructions on https://docs.artemis.cit.tum.de/dev/setup. When you want to support programming exercises, we recommend using the [Integrated Code Lifecycle](https://docs.artemis.cit.tum.de/dev/setup/#integrated-code-lifecycle-setup).
 
 Artemis can also be set up in conjunction with external tools for version control and continuous integration:
-1. [GitLab and Jenkins](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-gitlab-setup)
-2. [GitLab and GitLab CI (experimental)](https://docs.artemis.cit.tum.de/dev/setup/#gitlab-ci-and-gitlab-setup)
+1. [LocalVC and Jenkins](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-localvc-setup)
+2. [GitLab and Jenkins (deprecated)](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-gitlab-setup)
+3. [GitLab and GitLab CI (experimental)](https://docs.artemis.cit.tum.de/dev/setup/#gitlab-ci-and-gitlab-setup)
 
 Artemis uses these external tools for user management and the configuration of programming exercises.
 

docker/gitlab/gitlab-local-setup.sh

@@ -47,7 +47,7 @@ artemis:
 "
 
 echo
-echo 'The access Jenkins token has been created and can be copied into your jenkins-casc-config.yml file:'
+echo 'The access Jenkins token has been created and can be copied into your jenkins-casc-config-gitlab.yml file:'
 echo "
 credentials:
   system:

docker/jenkins.yml

@@ -13,7 +13,7 @@ services:
             - artemis-jenkins-data:/var/jenkins_home
             - ./jenkins/jenkins-disable-csrf.groovy:/var/jenkins_home/init.groovy # Disable CSRF token
             - ./jenkins/master-key-only-for-testing.key:/var/jenkins_home/master.key # Preset master key to use pre-generated secrets
-            - ./jenkins/jenkins-casc-config.yml:/usr/share/jenkins/ref/jenkins-casc-config.yml:ro
+            - ./jenkins/jenkins-casc-config-gitlab.yml:/usr/share/jenkins/ref/jenkins-casc-config.yml:ro
             - /var/run/docker.sock:/var/run/docker.sock
         ports:
             - "8082:8080"

docker/jenkins/jenkins-casc-config-localvc.yml

@@ -0,0 +1,33 @@
+jenkins:
+  systemMessage: "Jenkins has been configured to run with Artemis\n\n"
+  securityRealm:
+    local:
+      allowsSignup: false
+      users:
+        - id: "artemis_admin" # Artemis admin username
+          password: "artemis_admin" # Artemis admin password
+  labelString: "docker"
+  authorizationStrategy:
+    projectMatrix:
+        permissions:
+            - "Overall/Administer:artemis_admin"  # Make sure that the username matches the Artemis admin username
+            - "Overall/Read:authenticated"
+credentials:
+  system:
+    domainCredentials:
+      - credentials:
+          - string:
+              id: artemis_notification_plugin_token # Value of the "artemis-authentication-token-key" defined key in application-artemis.yml
+              description: 'Access token for Artemis Test Notification Plugin'
+              scope: GLOBAL
+              secret: artemis_admin  # Value of the "artemis-authentication-token-value" key defined in application-artemis.yml
+          - usernamePassword:
+              id: artemis_localvc_credentials
+              scope: GLOBAL
+              username: "jenkins" # artemis.version-control.build-agent-git-username
+              password: "artemis_admin" # artemis.version-control.build-agent-git-password
+              description: "LocalVC credentials for Artemis"
+unclassified:
+  timestamper:
+    elapsedTimeFormat: "'<b>'yyyy-MM-dd'T'HH:mm:ssX'</b> '"
+    systemTimeFormat: "'<b>'yyyy-MM-dd'T'HH:mm:ssX'</b> '"

docker/jenkins/plugins.yml

@@ -2,7 +2,6 @@
 plugins:
   - artifactId: matrix-auth
   - artifactId: docker-workflow
-  - artifactId: gitlab-plugin
   - artifactId: pipeline-maven
   - artifactId: timestamper
   - artifactId: configuration-as-code

docs/dev/setup.rst

@@ -46,7 +46,8 @@ following dependencies/tools on your machine:
    There are multiple stacks available for the integration with Artemis:
 
    * :ref:`Integrated Code Lifecycle Setup <Integrated Code Lifecycle Setup>`
-   * :ref:`GitLab and Jenkins <Jenkins and GitLab Setup>`
+   * :ref:`LocalVC and Jenkins <Jenkins and LocalVC Setup>`
+   * :ref:`GitLab and Jenkins <Jenkins and GitLab Setup>` (deprecated)
    * :ref:`GitLab and GitLab CI <GitLab CI and GitLab Setup>` (experimental, not yet production ready)
 
 ------------------------------------------------------------------------------------------------------------------------
@@ -74,6 +75,7 @@ following dependencies/tools on your machine:
    setup/server
    setup/client
    setup/integrated-code-lifecycle
+   setup/jenkins-localvc
    setup/jenkins-gitlab
    setup/gitlabci-gitlab
    setup/aeolus

docs/dev/setup/jenkins-gitlab.rst

@@ -3,6 +3,12 @@
 Jenkins and GitLab Setup
 ------------------------
 
+.. warning::
+
+    GitLab support will be removed with Artemis 8.0.0.
+    Please use :ref:`LocalVC and Jenkins <Jenkins and LocalVC Setup>` instead for new installations with Jenkins as the CI system.
+    For existing Jenkins and GitLab Setups, you can migrate to LocalVC with this `not merged Pull Request <https://github.com/ls1intum/Artemis/pull/8972>`__.
+
 This section describes how to set up a programming exercise environment
 based on Jenkins and GitLab. Optional commands are in curly brackets ``{}``.
 
@@ -63,7 +69,6 @@ the `Gitlab Server Quickstart <#gitlab-server-quickstart>`__ guide.
         user: artemis_admin
         password: artemis_admin
         url: http://localhost:8082
-        secret-push-token: AQAAABAAAAAg/aKNFWpF9m2Ust7VHDKJJJvLkntkaap2Ka3ZBhy5XjRd8s16vZhBz4fxzd4TH8Su # pre-generated or replaced in Automated Jenkins Server step 3
         vcs-credentials: artemis_gitlab_admin_credentials
         artemis-authentication-token-key: artemis_notification_plugin_token
         artemis-authentication-token-value: artemis_admin
@@ -466,8 +471,8 @@ Automated Jenkins Server Setup
 The following steps describe how to deploy a pre-configured version of the Jenkins server.
 This is ideal as a quickstart for developers. For a more detailed setup, see
 `Manual Jenkins Server Setup <#manual-jenkins-server-setup>`__.
-In a production setup, you have to at least change the user credentials (in the file ``jenkins-casc-config.yml``) and
-generate random access tokens and push tokens.
+In a production setup, you have to at least change the user credentials (in the file ``jenkins-casc-config-gitlab.yml``) and
+generate random access tokens.
 
 1. Create a new access token in GitLab named ``Jenkins`` and give it **api** and **read_repository** rights. You can
 do either do it manually or using the following command:
@@ -486,20 +491,9 @@ do either do it manually or using the following command:
        docker compose -f docker/<Jenkins setup to be launched>.yml up -d
 
    Jenkins is then reachable under ``http://localhost:8082/`` and you can login using the credentials specified
-   in ``jenkins-casc-config.yml`` (defaults to ``artemis_admin`` as both username and password).
-
-3. You need to generate the `secret-push-token`.
+   in ``jenkins-casc-config-gitlab.yml`` (defaults to ``artemis_admin`` as both username and password).
 
-   ..
-       Workaround as long as Github Issue 5973 (Default Push Notifications GitLab → Jenkins not working)
-       for now just generate the secret-push-token manually
-
-   As there is currently an `open issue with the presets for Jenkins in Development environments <https://github.com/ls1intum/Artemis/issues/5973>`__,
-   follow the steps described in
-   `Gitlab to Jenkins push notification token <#gitlab-to-jenkins-push-notification-token>`__ to generate the token.
-   In a production setup, you should use a random ``master.key`` in the file ``gitlab-jenkins-mysql.yml``.
-
-4. The `application-local.yml` must be adapted with the values configured in ``jenkins-casc-config.yml``:
+3. The `application-local.yml` must be adapted with the values configured in ``jenkins-casc-config-gitlab.yml``:
 
 .. code:: yaml
 
@@ -517,7 +511,6 @@ do either do it manually or using the following command:
             user: artemis_admin
             password: artemis_admin
             url: http://localhost:8082
-            secret-push-token: # pre-generated or replaced in Automated Jenkins Server step 3
             vcs-credentials: artemis_gitlab_admin_credentials
             artemis-authentication-token-key: artemis_notification_plugin_token
             artemis-authentication-token-value: artemis_admin
@@ -711,22 +704,19 @@ The list of plugins is maintained in ``docker/jenkins/plugins.yml``.
 You will need to install the following plugins (apart from the
 recommended ones that got installed during the setup process):
 
-1.  `GitLab <https://plugins.jenkins.io/gitlab-plugin/>`__ for enabling
-    webhooks to and from GitLab
-
-2.  `Timestamper <https://plugins.jenkins.io/timestamper/>`__ for adding the
+1.  `Timestamper <https://plugins.jenkins.io/timestamper/>`__ for adding the
     time to every line of the build output (Timestamper might already be installed)
 
-3.  `Pipeline <https://plugins.jenkins.io/workflow-aggregator/>`__ for defining the
+2.  `Pipeline <https://plugins.jenkins.io/workflow-aggregator/>`__ for defining the
     build description using declarative files (Pipeline might already be installed)
 
     **Note:** This is a suite of plugins that will install multiple plugins
 
-4. `Pipeline Maven <https://plugins.jenkins.io/pipeline-maven/>`__ to use maven within the pipelines. If you want to
+3. `Pipeline Maven <https://plugins.jenkins.io/pipeline-maven/>`__ to use maven within the pipelines. If you want to
    use Docker for your build agents you may also need to install
    `Docker Pipeline <https://plugins.jenkins.io/docker-workflow/>`__ .
 
-5. `Matrix Authorization Strategy Plugin <https://plugins.jenkins.io/matrix-auth/>`__ for configuring permissions
+4. `Matrix Authorization Strategy Plugin <https://plugins.jenkins.io/matrix-auth/>`__ for configuring permissions
    for users on a project and build plan level (Matrix Authorization Strategy might already be installed).
 
 
@@ -868,87 +858,6 @@ GitLab Repository Access
            continuous-integration:
                vcs-credentials: the.id.of.the.username.and.password.credentials.from.jenkins
 
-GitLab to Jenkins push notification token
-"""""""""""""""""""""""""""""""""""""""""
-
-GitLab has to notify Jenkins build plans if there are any new commits to
-the repository. The push notification that gets sent here is secured by
-a token generated by Jenkins. In order to get this token, you have to do
-the following steps:
-
-1.  Create a new item in Jenkins (use the Freestyle project type) and
-    name it **TestProject**
-
-2.  In the project configuration, go to *Build Triggers → Build when a
-    change is pushed to GitLab* and activate this option
-
-3.  Click on *Advanced*.
-
-4.  You will now have a couple of new options here, one of them being a
-    “**Secret token**”.
-
-5.  Click on the “*Generate*” button right below the text box for that
-    token.
-
-6.  Copy the generated value, let’s call it **$gitlab-push-token**
-
-7.  Apply these change to the plan (i.e. click on *Apply*)
-
-   .. figure:: jenkins-gitlab/jenkins_test_project.png
-      :align: center
-
-8.  Perform a *GET* request to the following URL (e.g. with Postman)
-    using Basic Authentication and the username and password you chose
-    for the Jenkins admin account:
-
-    ::
-
-        GET https://your.jenkins.domain/job/TestProject/config.xml
-
-    If you have xmllint installed, you can use this command, which will output the ``secret-push-token`` from
-    steps 9 and 10 (you may have to adjust the username and password):
-
-    .. code:: bash
-
-        curl -u artemis_admin:artemis_admin http://localhost:8082/job/TestProject/config.xml | xmllint --nowarning --xpath "//project/triggers/com.dabsquared.gitlabjenkins.GitLabPushTrigger/secretToken/text()" - | sed 's/^.\(.*\).$/\1/'
-
-9.  You will get the whole configuration XML of the just created build
-    plan, there you will find the following tag:
-
-    ::
-
-        <secretToken>{$some-long-encrypted-value}</secretToken>
-
-   .. figure:: jenkins-gitlab/jenkins_project_config_xml.png
-      :align: center
-
-      Job configuration XML
-
-10. Copy the ``secret-push-token value`` in the line
-    ``<secretToken>{secret-push-token}</secretToken>``. This is the encrypted value of the ``gitlab-push-token``
-    you generated in step 5.
-
-11. Now, you can delete this test project and input the following values
-    into your Artemis configuration *application-artemis.yml* (replace
-    the placeholders with the actual values you wrote down)
-
-    .. code:: yaml
-
-       artemis:
-           continuous-integration:
-               secret-push-token: $some-long-encrypted-value
-
-12. In a local setup, you have to disable CSRF otherwise some API endpoints will return HTTP Status 403 Forbidden.
-    This is done be executing the following command:
-    ``docker compose -f docker/<Jenkins setup to be launched>.yml exec -T jenkins dd of=/var/jenkins_home/init.groovy < docker/jenkins/jenkins-disable-csrf.groovy``
-
-    The last step is to disable the ``use-crumb`` option in ``application-local.yml``:
-
-    .. code:: yaml
-
-       jenkins:
-           use-crumb: false
-
 Upgrading Jenkins
 """""""""""""""""
 
@@ -1010,7 +919,7 @@ You should also update the Jenkins plugins regularly due to security
 reasons. You can update them directly in the Web User Interface in the
 Plugin Manager.
 
-.. _jenkins_build_agents:
+.. _jenkins_build_agents_gitlab:
 
 Build agents
 ^^^^^^^^^^^^