diff --git a/README.md b/README.md index 040487b2ae20..044b49dec4f5 100644 --- a/README.md +++ b/README.md @@ -139,8 +139,9 @@ You can find a guide on [how to write documentation](docs/README.md). Setting up Artemis in your development environment or a demo production environment is really easy following the instructions on https://docs.artemis.cit.tum.de/dev/setup. When you want to support programming exercises, we recommend using the [Integrated Code Lifecycle](https://docs.artemis.cit.tum.de/dev/setup/#integrated-code-lifecycle-setup). Artemis can also be set up in conjunction with external tools for version control and continuous integration: -1. [GitLab and Jenkins](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-gitlab-setup) -2. [GitLab and GitLab CI (experimental)](https://docs.artemis.cit.tum.de/dev/setup/#gitlab-ci-and-gitlab-setup) +1. [LocalVC and Jenkins](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-localvc-setup) +2. [GitLab and Jenkins (deprecated)](https://docs.artemis.cit.tum.de/dev/setup/#jenkins-and-gitlab-setup) +3. [GitLab and GitLab CI (experimental)](https://docs.artemis.cit.tum.de/dev/setup/#gitlab-ci-and-gitlab-setup) Artemis uses these external tools for user management and the configuration of programming exercises. diff --git a/docker/gitlab/gitlab-local-setup.sh b/docker/gitlab/gitlab-local-setup.sh index 8a29ab818082..4e979c3ab2ea 100644 --- a/docker/gitlab/gitlab-local-setup.sh +++ b/docker/gitlab/gitlab-local-setup.sh @@ -47,7 +47,7 @@ artemis: " echo -echo 'The access Jenkins token has been created and can be copied into your jenkins-casc-config.yml file:' +echo 'The access Jenkins token has been created and can be copied into your jenkins-casc-config-gitlab.yml file:' echo " credentials: system: diff --git a/docker/jenkins.yml b/docker/jenkins.yml index 86b532593197..ca4f35492475 100644 --- a/docker/jenkins.yml +++ b/docker/jenkins.yml @@ -13,7 +13,7 @@ services: - artemis-jenkins-data:/var/jenkins_home - ./jenkins/jenkins-disable-csrf.groovy:/var/jenkins_home/init.groovy # Disable CSRF token - ./jenkins/master-key-only-for-testing.key:/var/jenkins_home/master.key # Preset master key to use pre-generated secrets - - ./jenkins/jenkins-casc-config.yml:/usr/share/jenkins/ref/jenkins-casc-config.yml:ro + - ./jenkins/jenkins-casc-config-gitlab.yml:/usr/share/jenkins/ref/jenkins-casc-config.yml:ro - /var/run/docker.sock:/var/run/docker.sock ports: - "8082:8080" diff --git a/docker/jenkins/jenkins-casc-config.yml b/docker/jenkins/jenkins-casc-config-gitlab.yml similarity index 100% rename from docker/jenkins/jenkins-casc-config.yml rename to docker/jenkins/jenkins-casc-config-gitlab.yml diff --git a/docker/jenkins/jenkins-casc-config-localvc.yml b/docker/jenkins/jenkins-casc-config-localvc.yml new file mode 100644 index 000000000000..36766ce7dd01 --- /dev/null +++ b/docker/jenkins/jenkins-casc-config-localvc.yml @@ -0,0 +1,33 @@ +jenkins: + systemMessage: "Jenkins has been configured to run with Artemis\n\n" + securityRealm: + local: + allowsSignup: false + users: + - id: "artemis_admin" # Artemis admin username + password: "artemis_admin" # Artemis admin password + labelString: "docker" + authorizationStrategy: + projectMatrix: + permissions: + - "Overall/Administer:artemis_admin" # Make sure that the username matches the Artemis admin username + - "Overall/Read:authenticated" +credentials: + system: + domainCredentials: + - credentials: + - string: + id: artemis_notification_plugin_token # Value of the "artemis-authentication-token-key" defined key in application-artemis.yml + description: 'Access token for Artemis Test Notification Plugin' + scope: GLOBAL + secret: artemis_admin # Value of the "artemis-authentication-token-value" key defined in application-artemis.yml + - usernamePassword: + id: artemis_localvc_credentials + scope: GLOBAL + username: "jenkins" # artemis.version-control.build-agent-git-username + password: "artemis_admin" # artemis.version-control.build-agent-git-password + description: "LocalVC credentials for Artemis" +unclassified: + timestamper: + elapsedTimeFormat: "''yyyy-MM-dd'T'HH:mm:ssX' '" + systemTimeFormat: "''yyyy-MM-dd'T'HH:mm:ssX' '" diff --git a/docker/jenkins/plugins.yml b/docker/jenkins/plugins.yml index 028d098e34fb..859200f9f59a 100644 --- a/docker/jenkins/plugins.yml +++ b/docker/jenkins/plugins.yml @@ -2,7 +2,6 @@ plugins: - artifactId: matrix-auth - artifactId: docker-workflow - - artifactId: gitlab-plugin - artifactId: pipeline-maven - artifactId: timestamper - artifactId: configuration-as-code diff --git a/docs/dev/setup.rst b/docs/dev/setup.rst index 3ed91396dc39..01c6a0acec05 100644 --- a/docs/dev/setup.rst +++ b/docs/dev/setup.rst @@ -46,7 +46,8 @@ following dependencies/tools on your machine: There are multiple stacks available for the integration with Artemis: * :ref:`Integrated Code Lifecycle Setup ` - * :ref:`GitLab and Jenkins ` + * :ref:`LocalVC and Jenkins ` + * :ref:`GitLab and Jenkins ` (deprecated) * :ref:`GitLab and GitLab CI ` (experimental, not yet production ready) ------------------------------------------------------------------------------------------------------------------------ @@ -74,6 +75,7 @@ following dependencies/tools on your machine: setup/server setup/client setup/integrated-code-lifecycle + setup/jenkins-localvc setup/jenkins-gitlab setup/gitlabci-gitlab setup/aeolus diff --git a/docs/dev/setup/jenkins-gitlab.rst b/docs/dev/setup/jenkins-gitlab.rst index 60a226292c68..59fe3b0bde26 100644 --- a/docs/dev/setup/jenkins-gitlab.rst +++ b/docs/dev/setup/jenkins-gitlab.rst @@ -3,6 +3,12 @@ Jenkins and GitLab Setup ------------------------ +.. warning:: + + GitLab support will be removed with Artemis 8.0.0. + Please use :ref:`LocalVC and Jenkins ` instead for new installations with Jenkins as the CI system. + For existing Jenkins and GitLab Setups, you can migrate to LocalVC with this `not merged Pull Request `__. + This section describes how to set up a programming exercise environment based on Jenkins and GitLab. Optional commands are in curly brackets ``{}``. @@ -63,7 +69,6 @@ the `Gitlab Server Quickstart <#gitlab-server-quickstart>`__ guide. user: artemis_admin password: artemis_admin url: http://localhost:8082 - secret-push-token: AQAAABAAAAAg/aKNFWpF9m2Ust7VHDKJJJvLkntkaap2Ka3ZBhy5XjRd8s16vZhBz4fxzd4TH8Su # pre-generated or replaced in Automated Jenkins Server step 3 vcs-credentials: artemis_gitlab_admin_credentials artemis-authentication-token-key: artemis_notification_plugin_token artemis-authentication-token-value: artemis_admin @@ -466,8 +471,8 @@ Automated Jenkins Server Setup The following steps describe how to deploy a pre-configured version of the Jenkins server. This is ideal as a quickstart for developers. For a more detailed setup, see `Manual Jenkins Server Setup <#manual-jenkins-server-setup>`__. -In a production setup, you have to at least change the user credentials (in the file ``jenkins-casc-config.yml``) and -generate random access tokens and push tokens. +In a production setup, you have to at least change the user credentials (in the file ``jenkins-casc-config-gitlab.yml``) and +generate random access tokens. 1. Create a new access token in GitLab named ``Jenkins`` and give it **api** and **read_repository** rights. You can do either do it manually or using the following command: @@ -486,20 +491,9 @@ do either do it manually or using the following command: docker compose -f docker/.yml up -d Jenkins is then reachable under ``http://localhost:8082/`` and you can login using the credentials specified - in ``jenkins-casc-config.yml`` (defaults to ``artemis_admin`` as both username and password). - -3. You need to generate the `secret-push-token`. + in ``jenkins-casc-config-gitlab.yml`` (defaults to ``artemis_admin`` as both username and password). - .. - Workaround as long as Github Issue 5973 (Default Push Notifications GitLab → Jenkins not working) - for now just generate the secret-push-token manually - - As there is currently an `open issue with the presets for Jenkins in Development environments `__, - follow the steps described in - `Gitlab to Jenkins push notification token <#gitlab-to-jenkins-push-notification-token>`__ to generate the token. - In a production setup, you should use a random ``master.key`` in the file ``gitlab-jenkins-mysql.yml``. - -4. The `application-local.yml` must be adapted with the values configured in ``jenkins-casc-config.yml``: +3. The `application-local.yml` must be adapted with the values configured in ``jenkins-casc-config-gitlab.yml``: .. code:: yaml @@ -517,7 +511,6 @@ do either do it manually or using the following command: user: artemis_admin password: artemis_admin url: http://localhost:8082 - secret-push-token: # pre-generated or replaced in Automated Jenkins Server step 3 vcs-credentials: artemis_gitlab_admin_credentials artemis-authentication-token-key: artemis_notification_plugin_token artemis-authentication-token-value: artemis_admin @@ -711,22 +704,19 @@ The list of plugins is maintained in ``docker/jenkins/plugins.yml``. You will need to install the following plugins (apart from the recommended ones that got installed during the setup process): -1. `GitLab `__ for enabling - webhooks to and from GitLab - -2. `Timestamper `__ for adding the +1. `Timestamper `__ for adding the time to every line of the build output (Timestamper might already be installed) -3. `Pipeline `__ for defining the +2. `Pipeline `__ for defining the build description using declarative files (Pipeline might already be installed) **Note:** This is a suite of plugins that will install multiple plugins -4. `Pipeline Maven `__ to use maven within the pipelines. If you want to +3. `Pipeline Maven `__ to use maven within the pipelines. If you want to use Docker for your build agents you may also need to install `Docker Pipeline `__ . -5. `Matrix Authorization Strategy Plugin `__ for configuring permissions +4. `Matrix Authorization Strategy Plugin `__ for configuring permissions for users on a project and build plan level (Matrix Authorization Strategy might already be installed). @@ -868,87 +858,6 @@ GitLab Repository Access continuous-integration: vcs-credentials: the.id.of.the.username.and.password.credentials.from.jenkins -GitLab to Jenkins push notification token -""""""""""""""""""""""""""""""""""""""""" - -GitLab has to notify Jenkins build plans if there are any new commits to -the repository. The push notification that gets sent here is secured by -a token generated by Jenkins. In order to get this token, you have to do -the following steps: - -1. Create a new item in Jenkins (use the Freestyle project type) and - name it **TestProject** - -2. In the project configuration, go to *Build Triggers → Build when a - change is pushed to GitLab* and activate this option - -3. Click on *Advanced*. - -4. You will now have a couple of new options here, one of them being a - “**Secret token**”. - -5. Click on the “*Generate*” button right below the text box for that - token. - -6. Copy the generated value, let’s call it **$gitlab-push-token** - -7. Apply these change to the plan (i.e. click on *Apply*) - - .. figure:: jenkins-gitlab/jenkins_test_project.png - :align: center - -8. Perform a *GET* request to the following URL (e.g. with Postman) - using Basic Authentication and the username and password you chose - for the Jenkins admin account: - - :: - - GET https://your.jenkins.domain/job/TestProject/config.xml - - If you have xmllint installed, you can use this command, which will output the ``secret-push-token`` from - steps 9 and 10 (you may have to adjust the username and password): - - .. code:: bash - - curl -u artemis_admin:artemis_admin http://localhost:8082/job/TestProject/config.xml | xmllint --nowarning --xpath "//project/triggers/com.dabsquared.gitlabjenkins.GitLabPushTrigger/secretToken/text()" - | sed 's/^.\(.*\).$/\1/' - -9. You will get the whole configuration XML of the just created build - plan, there you will find the following tag: - - :: - - {$some-long-encrypted-value} - - .. figure:: jenkins-gitlab/jenkins_project_config_xml.png - :align: center - - Job configuration XML - -10. Copy the ``secret-push-token value`` in the line - ``{secret-push-token}``. This is the encrypted value of the ``gitlab-push-token`` - you generated in step 5. - -11. Now, you can delete this test project and input the following values - into your Artemis configuration *application-artemis.yml* (replace - the placeholders with the actual values you wrote down) - - .. code:: yaml - - artemis: - continuous-integration: - secret-push-token: $some-long-encrypted-value - -12. In a local setup, you have to disable CSRF otherwise some API endpoints will return HTTP Status 403 Forbidden. - This is done be executing the following command: - ``docker compose -f docker/.yml exec -T jenkins dd of=/var/jenkins_home/init.groovy < docker/jenkins/jenkins-disable-csrf.groovy`` - - The last step is to disable the ``use-crumb`` option in ``application-local.yml``: - - .. code:: yaml - - jenkins: - use-crumb: false - Upgrading Jenkins """"""""""""""""" @@ -1010,7 +919,7 @@ You should also update the Jenkins plugins regularly due to security reasons. You can update them directly in the Web User Interface in the Plugin Manager. -.. _jenkins_build_agents: +.. _jenkins_build_agents_gitlab: Build agents ^^^^^^^^^^^^ diff --git a/docs/dev/setup/jenkins-gitlab/jenkins_project_config_xml.png b/docs/dev/setup/jenkins-gitlab/jenkins_project_config_xml.png deleted file mode 100644 index 4d5cc9bda468..000000000000 Binary files a/docs/dev/setup/jenkins-gitlab/jenkins_project_config_xml.png and /dev/null differ diff --git a/docs/dev/setup/jenkins-gitlab/jenkins_test_project.png b/docs/dev/setup/jenkins-gitlab/jenkins_test_project.png deleted file mode 100644 index f37c0dd6ff6f..000000000000 Binary files a/docs/dev/setup/jenkins-gitlab/jenkins_test_project.png and /dev/null differ diff --git a/docs/dev/setup/jenkins-localvc.rst b/docs/dev/setup/jenkins-localvc.rst new file mode 100644 index 000000000000..384affe8dc05 --- /dev/null +++ b/docs/dev/setup/jenkins-localvc.rst @@ -0,0 +1,779 @@ +.. _Jenkins and LocalVC Setup: + +Jenkins and LocalVC Setup +------------------------- + +This section describes how to set up a programming exercise environment +based on Jenkins and LocalVC, which is integrated in Artemis. Optional commands are in curly brackets ``{}``. + +The following assumes that all instances run on separate servers. If you +have one single server, or your own NGINX instance, just skip all NGINX +related steps and use the configurations provided under *Separate NGINX +Configurations* + +**If you want to setup everything on your local development computer, +ignore all NGINX related steps.** **Just make sure that you use +unique port mappings for your Docker containers (e.g.** ``8081`` **for +Jenkins,** ``8080`` **for Artemis)** + +**Prerequisites:** + +* `Docker `__ + + Make sure that docker has enough memory (~ 6GB). To adapt it, go to ``Preferences -> Resources`` and restart Docker. + +.. contents:: Content of this section + :local: + :depth: 3 + +Artemis +^^^^^^^ + +In order to use Artemis with Jenkins as **Continuous Integration** +Server and LocalVC as integrated **Version Control** Server, you have to configure +the file ``application-prod.yml`` (Production Server) or +``application-artemis.yml`` (Local Development) accordingly. Please note +that all values in ``<..>`` have to be configured properly. These values +will be explained below in the corresponding sections. If you want to set up a local environment, copy the values +below into your ``application-artemis.yml`` or ``application-local.yml`` file (the latter is recommended). + +.. code:: yaml + + artemis: + course-archives-path: ./exports/courses + repo-clone-path: ./repos + repo-download-clone-path: ./repos-download + bcrypt-salt-rounds: 11 # The number of salt rounds for the bcrypt password hashing. Lower numbers make it faster but more unsecure and vice versa. + # Please use the bcrypt benchmark tool to determine the best number of rounds for your system. https://github.com/ls1intum/bcrypt-Benchmark + user-management: + use-external: false + internal-admin: + username: artemis_admin + password: artemis_admin + accept-terms: false + login: + account-name: TUM + version-control: + url: http://localhost:8080 + user: root + password: dummy # has to be set, but does not matter for LocalVC + build-agent-git-username: jenkins + build-agent-git-password: artemis_admin # choose some strong password and username (gives read access to all repositories) + continuous-integration: + user: artemis_admin + password: artemis_admin + url: http://localhost:8082 + vcs-credentials: artemis_localvc_credentials + artemis-authentication-token-key: artemis_notification_plugin_token + artemis-authentication-token-value: artemis_admin + build-timeout: 30 + git: + name: Artemis + email: artemis@xcit.tum.de + jenkins: + # only required if Artemis and Jenkins cannot communicate on their public URLs + # e.g., Jenkins is only available in a local container network + internal-urls: + ci-url: http://jenkins:8080 + vcs-url: http://172.17.0.1:8080 # `http://host.docker.internal:8080` for Windows + use-crumb: false + server: + port: 8080 + url: http://172.17.0.1:8080 # `http://host.docker.internal:8080` for Windows + +In addition, you have to start Artemis with the profiles ``localvc`` and +``jenkins`` so that the correct adapters will be used, e.g.: + +:: + + --spring.profiles.active=dev,jenkins,localvc,artemis,scheduling + +Please read :ref:`Server Setup` for more details. + +For a local setup on Windows you can use `http://host.docker.internal` appended +by the chosen ports as the continuous-integration url and the internal vcs url. + +Make sure to change the ``server.url`` and ``artemis.version-control.url`` value in ``application-dev.yml`` +or ``application-prod.yml`` accordingly. The ``server.url`` value will be used for the +communication hooks from Jenkins to Artemis. +In case you use a different port than 80 (http) or 443 (https) for the +communication, you have to append it to the both urls value, +e.g. \ ``127.0.0.1:8080``. + +When you start Artemis for the first time, it will automatically create +an admin user. + +**Note:** Sometimes Artemis does not generate the admin user which may lead to a startup +error. You will have to create the user manually in the MySQL database. Make sure +both are set up correctly and follow these steps: + +1. Use the tool mentioned above to generate a password hash. +2. Connect to the database via a client like `MySQL Workbench `__ + and execute the following query to create the user. Replace `artemis_admin` and `HASHED_PASSWORD` with your + chosen username and password: + + .. code:: sql + + INSERT INTO `artemis`.`jhi_user` (`id`,`login`,`password_hash`,`first_name`,`last_name`,`email`, + `activated`,`lang_key`,`activation_key`,`reset_key`,`created_by`,`created_date`,`reset_date`, + `last_modified_by`,`last_modified_date`,`image_url`,`last_notification_read`,`registration_number`) + VALUES (1,"artemis_admin","HASHED_PASSWORD","artemis","administrator","artemis_admin@localhost", + 1,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); +3. Give the user admin and user roles: + + .. code:: sql + + INSERT INTO `artemis`.`jhi_user_authority` (`user_id`, `authority_name`) VALUES (1,"ROLE_ADMIN"); + INSERT INTO `artemis`.`jhi_user_authority` (`user_id`, `authority_name`) VALUES (1,"ROLE_USER"); + +Starting the Artemis server should now succeed. + +Jenkins +^^^^^^^ + +Automated Jenkins Server Setup +"""""""""""""""""""""""""""""" + +The following steps describe how to deploy a pre-configured version of the Jenkins server. +This is ideal as a quickstart for developers. For a more detailed setup, see +`Manual Jenkins Server Setup <#manual-jenkins-server-setup>`__. +In a production setup, you have to at least change the user credentials (in the file ``jenkins-casc-config-localvc.yml``). + +#. You can now first build and deploy Jenkins, then you can also start the other services which weren't started yet: + + .. code:: bash + + JAVA_OPTS=-Djenkins.install.runSetupWizard=false docker compose -f docker/.yml up --build -d jenkins + docker compose -f docker/.yml up -d + + Jenkins is then reachable under ``http://localhost:8082/`` and you can login using the credentials specified + in ``jenkins-casc-config-localvc.yml`` (defaults to ``artemis_admin`` as both username and password). + +#. The `application-local.yml` must be adapted with the values configured in ``jenkins-casc-config-localvc.yml``: + + .. code:: yaml + + artemis: + user-management: + use-external: false + internal-admin: + username: artemis_admin + password: artemis_admin + version-control: + url: http://localhost:8080 + user: root + password: dummy # have to be set, but does not matter for LocalVC + build-agent-git-username: jenkins + build-agent-git-password: artemis_admin # choose some strong password and username (gives read access to all repositories) + continuous-integration: + user: artemis_admin + password: artemis_admin + url: http://localhost:8082 + vcs-credentials: artemis_localvc_credentials + artemis-authentication-token-key: artemis_notification_plugin_token + artemis-authentication-token-value: artemis_admin + +#. Open the ``src/main/resources/config/application-jenkins.yml`` and change the following: + Again, if you are using a development setup, the template in the beginning of this page already contains the + correct values. + + .. code:: yaml + + jenkins: + internal-urls: + ci-url: http://jenkins:8080 + vcs-url: http://172.17.0.1:8080 # `http://host.docker.internal:8080` for Windows + +#. You're done. You can now run Artemis with the LocalVC/Jenkins environment. + +Manual Jenkins Server Setup +""""""""""""""""""""""""""" + +1. Pull the latest Jenkins LTS Docker image + + Run the following command to get the latest jenkins LTS docker image. + + .. code:: bash + + docker pull jenkins/jenkins:lts + +Nginx proxy setup +################# + +If you run your own NGINX or if you install Jenkins on a local development computer, you can skip this section. + +2. Create a file increasing the maximum file size for the Nginx proxy. + The nginx-proxy uses a default file limit that is too small for the + plugin that will be uploaded later. **Skip this step if you have your + own NGINX instance.** + + .. code:: bash + + echo "client_max_body_size 16m;" > client_max_body_size.conf + +3. The NGINX default timeout is pretty low. For plagiarism check and unlocking student repos for the exam a higher + timeout is advisable. Therefore we write our own nginx.conf and load it in the container. + + + .. code:: nginx + + user nginx; + worker_processes auto; + + error_log /var/log/nginx/error.log warn; + pid /var/run/nginx.pid; + + + events { + worker_connections 1024; + } + + + http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + fastcgi_read_timeout 300; + proxy_read_timeout 300; + + sendfile on; + #tcp_nopush on; + + keepalive_timeout 65; + + #gzip on; + + include /etc/nginx/conf.d/*.conf; + } + daemon off + +4. Run the NGINX proxy docker container, this will automatically setup + all reverse proxies and force https on all connections. (This image + would also setup proxies for all other running containers that have + the VIRTUAL_HOST and VIRTUAL_PORT environment variables). **Skip this + step if you have your own NGINX instance.** + + .. code:: bash + + docker run -itd --name nginx_proxy \ + -p 80:80 -p 443:443 \ + --restart always \ + -v /var/run/docker.sock:/tmp/docker.sock:ro \ + -v /etc/nginx/certs \ + -v /etc/nginx/vhost.d \ + -v /usr/share/nginx/html \ + -v $(pwd)/client_max_body_size.conf:/etc/nginx/conf.d/client_max_body_size.conf:ro \ + -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf:ro \ + jwilder/nginx-proxy + +5. The nginx proxy needs another docker-container to generate + letsencrypt certificates. Run the following command to start it (make + sure to change the email-address). **Skip this step if you have your + own NGINX instance.** + + .. code:: bash + + docker run --detach \ + --name nginx_proxy-letsencrypt \ + --volumes-from nginx_proxy \ + --volume /var/run/docker.sock:/var/run/docker.sock:ro \ + --env "DEFAULT_EMAIL=mail@yourdomain.tld" \ + jrcs/letsencrypt-nginx-proxy-companion + +Start Jenkins +############# + +6. Run Jenkins by executing the following command (change the hostname + and choose which port alternative you need) + + .. code:: bash + + docker run -itd --name jenkins \ + --restart always \ + -v jenkins_data:/var/jenkins_home \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /usr/bin/docker:/usr/bin/docker:ro \ + -e VIRTUAL_HOST=your.jenkins.domain -e VIRTUAL_PORT=8080 \ # Alternative 1: If you are NOT using a separate NGINX instance + -e LETSENCRYPT_HOST=your.jenkins.domain \ # Only needed if Alternative 1 is used + -p 8082:8080 \ # Alternative 2: If you ARE using a separate NGINX instance OR you ARE installing Jenkins on a local development computer + -u root \ + jenkins/jenkins:lts + + Note that you can omit the ``-u root``, ``-v /var/run/docker.sock:/var/run/docker.sock`` and + ``-v /usr/bin/docker:/usr/bin/docker:ro`` parameters, if you do not want to run Docker builds on the Jenkins controller + (but e.g. use remote agents). + +7. Open Jenkins in your browser (e.g. ``localhost:8082``) and setup the + admin user account (install all suggested plugins). You can get the + initial admin password using the following command. + + .. code:: bash + + # Jenkins highlights the password in the logs, you can't miss it + docker logs -f jenkins + or alternatively + docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword + +8. Set the chosen credentials in the Artemis configuration + *application-artemis.yml* + + .. code:: yaml + + artemis: + continuous-integration: + user: your.chosen.username + password: your.chosen.password + +Required Jenkins Plugins +"""""""""""""""""""""""" + +**Note:** The custom Jenkins Dockerfile takes advantage of the +`Plugin Installation Manager Tool for Jenkins `__ +to automatically install the plugins listed below. If you used the Dockerfile, you can skip these steps and +`Server Notification Plugin <#server-notification-plugin>`__. +The list of plugins is maintained in ``docker/jenkins/plugins.yml``. + + +You will need to install the following plugins (apart from the +recommended ones that got installed during the setup process): + +1. `Timestamper `__ for adding the + time to every line of the build output (Timestamper might already be installed) + +2. `Pipeline `__ for defining the + build description using declarative files (Pipeline might already be installed) + + **Note:** This is a suite of plugins that will install multiple plugins + +3. `Pipeline Maven `__ to use maven within the pipelines. If you want to + use Docker for your build agents you may also need to install + `Docker Pipeline `__ . + +4. `Matrix Authorization Strategy Plugin `__ for configuring permissions + for users on a project and build plan level (Matrix Authorization Strategy might already be installed). + + +The plugins above (and the pipeline-setup associated with it) got introduced in Artemis 4.7.3. +If you are using exercises that were created before 4.7.3, you also have to install these plugins: + +Please note that this setup is **deprecated** and will be removed in the future. +Please migrate to the new pipeline-setup if possible. + +1. `Multiple SCMs `__ for combining the + exercise test and assignment repositories in one build + +2. `Post Build Task `__ for preparing build + results to be exported to Artemis + +3. `Xvfb `__ for exercises based on GUI + libraries, for which tests have to have some virtual display + +Choose “Download now and install after restart” and checking the +“Restart Jenkins when installation is complete and no jobs are running” box + +Timestamper Configuration +""""""""""""""""""""""""" + +Go to *Manage Jenkins → System Configuration → Configure*. There you will find the +Timestamper configuration, use the following value for both formats: + +:: + + ''yyyy-MM-dd'T'HH:mm:ssX' ' + +.. figure:: jenkins-gitlab/timestamper_config.png + :align: center + +Server Notification Plugin +"""""""""""""""""""""""""" + +Artemis needs to receive a notification after every build, which +contains the test results and additional commit information. For that +purpose, we developed a Jenkins plugin, that can aggregate and *POST* +JUnit formatted results to any URL. + +You can download the current release of the plugin +`here `__ +(Download the **.hpi** file). Go to the Jenkins plugin page (*Manage +Jenkins → System Configuration → Plugins*) and install the downloaded file under the +*Advanced settings* tab under *Deploy Plugin* + +.. figure:: jenkins-gitlab/jenkins_custom_plugin.png + :align: center + +Jenkins Credentials +""""""""""""""""""" + +Go to *Manage Jenkins → Security → Credentials → Jenkins → Global credentials* and create the +following credentials + +Server Notification Token +######################### + +1. Create a new Jenkins credential containing the token, which gets send + by the server notification plugin to Artemis with every build result: + + 1. **Kind**: Secret text + 2. **Scope**: Global + 3. **Secret**: *your.secret_token_value* (choose any value you want, + copy it for the nex step) + 4. Leave the ID field blank + 5. The description is up to you + +2. Copy the generated ID of the new credentials and put it into the + Artemis configuration *application-artemis.yml* + + .. code:: yaml + + artemis: + continuous-integration: + artemis-authentication-token-key: the.id.of.the.notification.token.credential + +3. Copy the actual value you chose for the token and put it into the + Artemis configuration *application-artemis.yml* + + .. code:: yaml + + artemis: + continuous-integration: + artemis-authentication-token-value: the.actual.value.of.the.notification.token + +LocalVC Repository Access +######################### + +1. Create a new Jenkins credentials containing the username and password + of the build-agent-git-user: + + 1. **Kind**: Username with password + 2. **Scope**: Global + 3. **Username**: *the_username_you_chose_at_build-agent-git-username* + 4. **Password**: *the_password_you_chose_at_build-agent-git-password* + 5. Leave the ID field blank + 6. The description is up to you + +2. Copy the generated ID (e.g. ``ea0e3c08-4110-4g2f-9c83-fb2cdf6345fa``) + of the new credentials and put it into the Artemis configuration file + *application-artemis.yml* + + .. code:: yaml + + artemis: + continuous-integration: + vcs-credentials: the.id.of.the.username.and.password.credentials.from.jenkins + +Upgrading Jenkins +""""""""""""""""" + +In order to upgrade Jenkins to a newer version, you need to rebuild the Docker image targeting the new version. +The stable LTS versions can be viewed through the `changelog `__ +and the corresponding Docker image can be found on +`dockerhub `__. + +1. Open the Jenkins Dockerfile and replace the value of ``FROM`` with ``jenkins/jenkins:lts``. + After running the command ``docker pull jenkins/jenkins:lts``, this will use the latest LTS version + in the following steps. + You can also use a specific LTS version. + For example, if you want to upgrade Jenkins to version ``2.289.2``, you will need to use the + ``jenkins/jenkins:2.289.2-lts`` image. + +2. If you're using ``docker compose``, you can simply use the following command and skip the next steps. + + .. code:: bash + + docker compose -f docker/.yml up --build -d + +3. Build the new Docker image: + + .. code:: bash + + docker build --no-cache -t jenkins-artemis . + + The name of the image is called ``jenkins-artemis``. + +4. Stop the current Jenkins container (change jenkins to the name of your container): + + .. code:: bash + + docker stop jenkins + +5. Rename the container to ``jenkins_old`` so that it can be used as a backup: + + .. code:: bash + + docker rename jenkins jenkins_old + +6. Run the new Jenkins instance: + + .. code:: bash + + docker run -itd --name jenkins --restart always \ + -v jenkins_data:/var/jenkins_home \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -p 9080:8080 jenkins-artemis \ + +7. You can remove the backup container if it's no longer needed: + + .. code:: bash + + docker rm jenkins_old + + +You should also update the Jenkins plugins regularly due to security +reasons. You can update them directly in the Web User Interface in the +Plugin Manager. + +.. _jenkins_build_agents: + +Build agents +^^^^^^^^^^^^ + +You can either run the builds locally (that means on the machine that hosts Jenkins) or on remote build agents. + +Configuring local build agents +"""""""""""""""""""""""""""""" + +Go to `Manage Jenkins` → `Nodes` → `Built-In Node` → `Configure` + +Configure your master node like this (adjust the number of executors, if needed). Make sure to add the docker label. + + .. figure:: jenkins-gitlab/jenkins_local_node.png + :align: center + + Jenkins local node + +Alternative local build agents setup using docker +""""""""""""""""""""""""""""""""""""""""""""""""" + +An alternative way of adding a build agent that will use docker (similar to the remote agents below) but running +locally, can be done using the jenkins/ssh-agent docker image `docker image `__. + +Prerequisites: + +1. Make sure to have Docker `installed `__ + +Agent setup: + +1. Create a new SSH key using ``ssh-keygen`` (if a passphrase is added, store it for later) + +2. Copy the public key content (e.g. in ~/.ssh/id_rsa.pub) + +3. Run: + + .. code:: bash + + docker run -d --name jenkins_agent -v /var/run/docker.sock:/var/run/docker.sock \ + jenkins/ssh-agent:latest "" + +4. Get the GID of the 'docker' group with ``cat /etc/groups`` and remember it for later + +5. Enter the agent's container with ``docker exec -it jenkins_agent bash`` + +6. Install Docker with ``apt update && apt install docker.io`` + +7. Check if group 'docker' already exists with ``cat /etc/groups``. If yes, remove it with ``groupdel docker`` + +8. Add a new 'docker' group with the same GID as seen in point 2 with ``groupadd -g docker`` + +9. Add 'jenkins' user to the group with ``usermod -aG docker jenkins`` + +10. Activate changes with ``newgrp docker`` + +11. Now check if 'jenkins' has the needed permissions to run docker commands + + 1. Log in as 'jenkins' with ``su jenkins`` + + 2. Try if ``docker inspect `` works or if a permission error occurs + + 3. If an permission error occurs, try to restart the docker container + +12. Now you can exit the container executing ``exit`` twice (the first will exit the jenkins user and + the second the container) + +Add agent in Jenkins: + +1. Open Jenkins in your browser (e.g. localhost:8082) + +2. Go to Manage Jenkins → Credentials → System → Global credentials (unrestricted) → Add Credentials + + - Kind: SSH Username with private key + + - Scope: Global (Jenkins, nodes, items, all child items, etc) + + - ID: leave blank + + - Description: Up to you + + - Username: jenkins + + - Private Key: (e.g /root/.ssh/id_rsa) + + - Passphrase: (you can leave it blank if none has been specified) + + .. figure:: jenkins-gitlab/alternative_jenkins_node_credentials.png + :align: center + +3. Go to Manage Jenkins → Nodes → New Node + + - Node name: Up to you (e.g. Docker agent node) + + - Check 'Permanent Agent' + + .. figure:: jenkins-gitlab/alternative_jenkins_node_setup.png + :align: center + +4. Node settings: + + - # of executors: Up to you (e.g. 4) + + - Remote root directory: /home/jenkins/agent + + - Labels: docker + + - Usage: Only build jobs with label expressions matching this node + + - Launch method: Launch agents via SSH + + - Host: output of command ``docker inspect --format '{{ .Config.Hostname }}' jenkins_agent`` + + - Credentials: + + - Host Key Verification Strategy: Non verifying Verification Strategy + + - Availability: Keep this agent online as much as possible + + .. figure:: jenkins-gitlab/alternative_jenkins_node.png + :align: center + +5. Save the new node + +6. Node should now be up and running + +Installing remote build agents +"""""""""""""""""""""""""""""" +You might want to run the builds on additional Jenkins agents, especially if a large amount of students should use +the system at the same time. +Jenkins supports remote build agents: The actual compilation of the students submissions happens on these +other machines but the whole process is transparent to Artemis. + +This guide explains setting up a remote agent on an Ubuntu virtual machine that supports docker builds. + +Prerequisites: +1. Install Docker on the remote machine: https://docs.docker.com/engine/install/ubuntu/ + +2. Add a new user to the remote machine that Jenkins will use: ``sudo adduser --disabled-password --gecos "" jenkins`` + +3. Add the jenkins user to the docker group (This allows the jenkins user to interact with docker): + ``sudo usermod -a -G docker jenkins`` + +4. Generate a new SSH key locally (e.g. using ``ssh-keygen``) and add the public key to the ``.ssh/authorized_keys`` + file of the jenkins user on the agent VM. + +5. Validate that you can connect to the build agent machine using SSH and the generated private key and validate that + you can use docker (`docker ps` should not show an error) + +6. Log in with your normal account on the build agent machine and install Java: ``sudo apt install default-jre`` + +7. Add a new secret in Jenkins, enter private key you just generated and add the passphrase, if set: + + .. figure:: jenkins-gitlab/jenkins_ssh_credentials.png + :align: center + + Jenkins SSH Credentials + +8. Add a new node (select a name and select `Permanent Agent`): + Set the number of executors so that it matches your machine's specs: This is the number of concurrent builds + this agent can handle. It is recommended to match the number of cores of the machine, + but you might want to adjust this later if needed. + + Set the remote root directory to ``/home/jenkins/remote_agent``. + + Set the usage to `Only build jobs with label expressions matching this node`. + This ensures that only docker-jobs will be built on this agent, and not other jobs. + + Add a label ``docker`` to the agent. + + Set the launch method to `Launch via SSH` and add the host of the machine. + Select the credentials you just created and select `Manually trusted key Verification Strategy` + as Host key verification Strategy. + Save it. + + + .. figure:: jenkins-gitlab/jenkins_node.png + :align: center + + Add a Jenkins node + +9. Wait for some moments while jenkins installs it's remote agent on the agent's machine. + You can track the progress using the `Log` page when selecting the agent. System information should also be available. + +10. Change the settings of the master node to be used only for specific jobs. + This ensures that the docker tasks are not executed on the master agent but on the remote agent. + + + .. figure:: jenkins-gitlab/jenkins_master_node.png + :align: center + + Adjust Jenkins master node settings + +11. You are finished, the new agent should now also process builds. + + +Jenkins User Management +^^^^^^^^^^^^^^^^^^^^^^^ + +Artemis supports user management in Jenkins as of version 4.11.0. Creating an account in Artemis will also create an +account on Jenkins using the same password. This enables users to login and access Jenkins. Updating and/or deleting +users from Artemis will also lead to updating and/or deleting from Jenkins. + +Unfortunately, Jenkins does not provide a Rest API for user management which present the following **caveats**: + + - The username of a user is treated as a unique identifier in Jenkins. + - It's not possible to update an existing user with a single request. + We update by deleting the user from Jenkins and recreating it with the updated data. + - In Jenkins, users are created in an on-demand basis. + For example, when a build is performed, its change log is computed and as a result commits from users + who Jenkins has never seen may be discovered and created. + - Since Jenkins users may be re-created automatically, issues may occur such as 1) creating a user, deleting it, + and then re-creating it and 2) changing the username of the user and reverting back to the previous one. + - Updating a user will re-create it in Jenkins and therefore remove any additionally saved Jenkins-specific + user data such as API access tokens. + + +Jenkins Build Plan Access Control Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Artemis takes advantage of the Project-based Matrix Authorization Strategy plugin to support build plan +access control in Jenkins. +This enables specific Artemis users to access build plans and execute actions such as triggering a build. +This section explains the changes required in Jenkins in order to set up build plan access control: + +1. Navigate to Manage Jenkins → Plugins → Installed plugins and make sure that you have the + `Matrix Authorization Strategy `__ plugin installed + +2. Navigate to Manage Jenkins → Security and navigate to the "Authorization" section + +3. Select the "Project-based Matrix Authorization Strategy" option + +4. In the table make sure that the "Read" permission under the "Overall" section is assigned to + the "Authenticated Users" user group. + +5. In the table make sure that all "Administer" permission is assigned to all administrators. + +6. You are finished. If you want to fine-tune permissions assigned to teaching assistants and/or instructors, + you can change them within the ``JenkinsJobPermission.java`` file. + +.. figure:: jenkins-gitlab/jenkins_authorization_permissions.png + :align: center + + +Caching +^^^^^^^ + +You can configure caching for e.g. Maven repositories. +See :ref:`this section in the administration documentation ` for more details.