Merge pull request #929 from openzim/upgrade_receiver_workers

Upgrade receiver and workers

dev/README.md

@@ -29,9 +29,17 @@ This container is a PostgreSQL DB. DB data is kept in a volume, persistent acros
 
 This container hosts the frontend UI for end-users.
 
-### ssh-host
+### receiver
 
-This container hosts a minimal SSH server, usefull for uploading Zimfarm artifacts locally during tests.
+This container hosts a customized SSH server, used to receive Zimfarm ZIMs coming from workers.
+
+### worker_mgr
+
+This container is the main worker container, responsible to start tasks. It is commented by default.
+
+### task_worker
+
+This container is a sample task executor. It is commented by default.
 
 ## Instructions
 
@@ -42,7 +50,7 @@ cd dev
 docker compose -p zimfarm up -d
 ```
 
-## Setup Postgresql DB
+### Setup Postgresql DB
 
 If this is your first run or if you made any schema change, you need to set/update the DB schema before having all containers OK.
 
@@ -67,7 +75,7 @@ alembic check
 Note that to run integration tests, we use a separate DB, you hence have to set/update the DB schema as well.
 Just do the same as above with the backend-tests container (instead of the backend-tools)
 
-## Restart the backend
+### Restart the backend
 
 The backend might typically fail if the DB schema is not up-to-date, or if you create some nasty bug while modifying the code.
 
@@ -78,15 +86,15 @@ docker restart zf_backend
 
 Other containers might be restarted the same way.
 
-## Browse the web UI
+### Browse the web UI
 
 To develop: open [the development web UI](http://localhost:8002). This version has hot reload of UI code changes.
 
 To test build version: open [the web UI](http://localhost:8001) in your favorite browser.
 
 You can login with username `admin` and password `admin`.
 
-## Run tests
+### Run backend tests
 
 Do not forget to set/update the test DB schema
 
@@ -125,4 +133,40 @@ docker-compose setup is ready (and supposing that your local DB is up-to-date):
 ```sh
 cd dispatcher/backend
 docker run --network zimfarm_default -v "$(pwd)/docs/schemaspy.properties:/schemaspy.properties" -v "$(pwd)/docs/schemaspy:/output" schemaspy/schemaspy:latest
-```
+```
+
+### create a test worker
+
+In order to test worker manager and task worker, but also to test some other stuff, you will need to have a test worker.
+
+It is not mandatory to have the worker manager running in most situation, but you will need to have both a worker user in the Zimfarm, and associated private/public key pairs.
+
+A usefull script to perform all the test worker creation is at `contrib/create_worker.sh`: call it once to create a `test_worker` user, the associated worker object, and upload a test public key. You will then be able to assign tasks to this worker in the UI, and use this test worker for running the worker manager and the task worker.
+
+Once this is is done, you can start the worker manager simply by uncommenting the `worker_mgr` container in `docker-compose.yml`. 
+
+**Important:** Beware that once you start the worker manager, any pending task will be automatically started by the worker manager. You might want to clear the pending tasks list before starting the worker manager.
+
+### mark a task as started
+
+Through the UI, it is easy to create a requested task for your test worker. However, if you do not want to run the worker manager because you do not want the task to really proceed, it gets complicated to fake the start this requested task, i.e mark the fact that the test worker manager has reserved this requested task.
+
+A usefull script is at `contrib/start_first_req_task.sh`: this will mark the first task in the pipe (oldest one) as reserved for the test worker, and hence transform the requested task into a task. You can obviously call it many times to reserve many tasks. The script displays the whole task, including its id.
+
+### tweak receiver configuration
+
+Receiver is responsible to receive ZIMs, logs and artifacts created by the task worker. It is a modified SSH server which performs authentication against the Zimfarm DB.
+
+In order to use it with a task manager, you have to create one directory per warehouse path (or at least create the ones for the tasks you will run).
+
+A usefull script has been added to the dev stack to create these directories: 
+
+```
+docker exec -it zf_receiver /contrib/create-warehouse-paths.sh
+```
+
+### test a task manager
+
+You can start a task manager manually simply by requesting a task in the UI and starting it manually (see above).
+
+Once the task is reserved for the `test_worker`, you can modify the `task_worker` container `command` in `docker-compose.yml` with this ID, uncomment the `task_worker` section and start it.

dev/contrib/create_worker.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+
+# Call it once to create a `test_worker`:
+# - retrieve an admin token
+# - create the `test_worker`` user
+# - create the associated worker object
+# - upload a test public key. 
+#
+# To be used to have a "real" test worker for local development, typically to start
+# a worker manager or a task manager or simply assign tasks to a worker in the UI/API
+
+set -e
+
+echo "Retrieving admin access token"
+
+ZF_ADMIN_TOKEN="$(curl -s -X 'POST' \
+    'http://localhost:8000/v1/auth/authorize' \
+    -H 'accept: application/json' \
+    -H 'Content-Type: application/x-www-form-urlencoded' \
+    -d 'username=admin&password=admin' \
+    | jq -r '.access_token')"
+
+echo "Create test_worker user"
+
+curl -s -X 'POST' \
+  'http://localhost:8000/v1/users/' \
+  -H 'accept: */*' \
+  -H 'Content-Type: application/json' \
+  -H "Authorization: Bearer $ZF_ADMIN_TOKEN" \
+  -d '{
+  "role":"worker",
+  "username": "test_worker",
+  "email":"[email protected]",
+  "password":"test_worker"
+}'
+
+echo "Retrieving test_worker access token"
+
+ZF_USER_TOKEN="$(curl -s -X 'POST' \
+    'http://localhost:8000/v1/auth/authorize' \
+    -H 'accept: application/json' \
+    -H 'Content-Type: application/x-www-form-urlencoded' \
+    -d 'username=test_worker&password=test_worker' \
+    | jq -r '.access_token')"
+
+echo "Worker check-in (will create it since missing)"
+
+curl -s -X 'PUT' \
+  'http://localhost:8000/v1/workers/test_worker/check-in' \
+  -H 'accept: */*' \
+  -H 'Content-Type: application/json' \
+  -H "Authorization: Bearer $ZF_USER_TOKEN" \
+  -d '{
+  "username": "test_worker",
+  "cpu": 3,
+  "memory": 1024,
+  "disk": 0,
+  "offliners": [
+    "zimit"
+  ]
+}'
+
+echo "Add private key to test_worker"
+
+curl -X POST http://localhost:8000/v1/users/test_worker/keys \
+  -H 'accept: */*' \
+  -H "Authorization: Bearer $ZF_USER_TOKEN" \
+  -H 'Content-Type: application/json; charset=utf-8' \
+  -d '{"name": "test_key", "key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCn2r5IZSJp02FBAYSZBQRdOBKBK2VOErdrBCZm5Ig3hDKQuxq38+W5CJ2JUJU+LQm//uenm58scGlEtk5+w5SjObjzK8Qx6JeRhAiZ8xpyydSoUIvd0ARD9OKwdiQFqVlLPlOyrdIpQ2vRESdwzhe0f7EYUwgKzBw5k0foxQsGxTiztY/ugWJ8Jso5WOxXwzEw4cSnGhdrehqLphlZanr54wj5oTcrj/vJHlpbxkYzFMc2Zgj81GdIV4yP3H1yX4ySK8VkDPOCczHacdRnHw4u8Vgf6wS6Zy3iMpvuGu7BJkwNoTXvmVV5BXUm6GAMSQTAPcw5T8M+eXjSAnriGDAL"}'
+
+echo "DONE"

dev/contrib/start_first_req_task.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Call as many times as necessary to transition the first (oldest) requested task in the
+# database into a reserved task, assigned to the `test_worker` worker.
+#
+# Displays the whole task JSON.
+
+set -e
+
+echo "Retrieving access token"
+
+ZF_ADMIN_TOKEN="$(curl -s -X 'POST' \
+    'http://localhost:8000/v1/auth/authorize' \
+    -H 'accept: application/json' \
+    -H 'Content-Type: application/x-www-form-urlencoded' \
+    -d 'username=admin&password=admin' \
+    | jq -r '.access_token')"
+
+echo "Get last requested task"
+
+FIRST_TASK_ID="$(curl -s -X 'GET' \
+  'http://localhost:8000/v1/requested-tasks/' \
+  -H 'accept: application/json' \
+  -H "Authorization: Bearer $ZF_ADMIN_TOKEN" \
+  | jq -r '.items[0]._id')"
+
+if [ "$FIRST_TASK_ID" = "null" ]; then
+    echo "No pending requested task. Exiting script."
+    exit 1
+fi
+
+echo "Start task (i.e. mark it as started)"
+
+curl -s -X 'POST' \
+  "http://localhost:8000/v1/tasks/$FIRST_TASK_ID?worker_name=worker" \
+  -H 'accept: application/json' \
+  -H "Authorization: Bearer $ZF_ADMIN_TOKEN" \
+  -d ''
+
+echo "DONE"

dev/docker-compose.yml

@@ -28,9 +28,11 @@ services:
       JWT_SECRET: DH8kSxcflUVfNRdkEiJJCn2dOOKI3qfw
       POSTGRES_URI: postgresql+psycopg://zimfarm:zimpass@postgresdb:5432/zimfarm
       ALEMBIC_UPGRADE_HEAD_ON_START: "1"
-      ARTIFACTS_UPLOAD_URI: scp://root@ssh-host:22/artifacts/
-      LOGS_UPLOAD_URI: scp://root@ssh-host:22/logs/
-      ZIM_UPLOAD_URI: scp://root@ssh-host:22/zims/
+      # upload artifacts, logs and zim to receiver for simplicity
+      ARTIFACTS_UPLOAD_URI: sftp://uploader@receiver:22/logs/ # reusing logs dir, kind of a hack
+      LOGS_UPLOAD_URI: sftp://uploader@receiver:22/logs/
+      ZIM_UPLOAD_URI: sftp://uploader@receiver:22/zim/
+      ZIMCHECK_OPTION: --all
     depends_on:
       - postgresdb
   frontend-ui:
@@ -81,13 +83,47 @@ services:
       POSTGRES_URI: postgresql+psycopg://zimfarm:zimpass@postgresdb:5432/zimtest
     depends_on:
       - postgresdb
-  ssh-host:
-    build:
-      context: ssh-host
-    container_name: zf_ssh_host
+  receiver:
+    build: ../receiver
+    container_name: zf_receiver
     ports:
-      - 127.0.0.1:8022:22
+      - 127.0.0.1:8222:22
+    volumes:
+      - ./receiver/create-warehouse-paths.sh:/contrib/create-warehouse-paths.sh
+    environment:
+      - ZIMFARM_WEBAPI=http://backend:8000/v1
+    depends_on:
+      - backend
+
+  # # uncomment this only if you want to run a worker manager
+  # worker_mgr:
+  #   build:
+  #     context: ../workers
+  #     dockerfile: manager-Dockerfile
+  #   container_name: zf_worker_mgr
+  #   depends_on:
+  #     - backend
+  #   command: worker-manager --webapi-uri 'http://backend:8000/v1' --username test_worker --name test_worker
+  #   volumes:
+  #     - /var/run/docker.sock:/var/run/docker.sock
+  #     - ./test_worker-identity/id_rsa:/etc/ssh/keys/zimfarm
 
+  # # uncomment this only if you want to run a 'standalone' task worker
+  # # you have to modify the <your_task_id> in the command with a real requested task
+  # task_worker:
+  #   build:
+  #     context: ../workers
+  #     dockerfile: task-Dockerfile
+  #   container_name: zf_task_worker
+  #   depends_on:
+  #     - backend
+  #   command: task-worker --webapi-uri 'http://backend:8000/v1' --username test_worker --task-id <your_task_id>
+  #   volumes:
+  #     - /var/run/docker.sock:/var/run/docker.sock
+  #     - ./test_worker-identity/id_rsa:/etc/ssh/keys/zimfarm
+  #   environment:
+  #     - DEBUG=1
+  #     - DOCKER_NETWORK=zimfarm_default
 
 volumes:
   pg_data_zimfarm:

dev/receiver/create-warehouse-paths.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+set -e
+
+mkdir -p \
+    /jail/zim/freecodecamp \
+    /jail/zim/gutenberg \
+    /jail/zim/ifixit \
+    /jail/zim/mooc \
+    /jail/zim/other \
+    /jail/zim/phet \
+    /jail/zim/stack_exchange \
+    /jail/zim/ted \
+    /jail/zim/videos \
+    /jail/zim/vikidia \
+    /jail/zim/wikibooks \
+    /jail/zim/wikihow \
+    /jail/zim/wikinews \
+    /jail/zim/wikipedia \
+    /jail/zim/wikiquote \
+    /jail/zim/wikisource \
+    /jail/zim/wikiversity \
+    /jail/zim/wikivoyage \
+    /jail/zim/wiktionary \
+    /jail/zim/zimit
+
+chmod 777 \
+    /jail/zim/freecodecamp \
+    /jail/zim/gutenberg \
+    /jail/zim/ifixit \
+    /jail/zim/mooc \
+    /jail/zim/other \
+    /jail/zim/phet \
+    /jail/zim/stack_exchange \
+    /jail/zim/ted \
+    /jail/zim/videos \
+    /jail/zim/vikidia \
+    /jail/zim/wikibooks \
+    /jail/zim/wikihow \
+    /jail/zim/wikinews \
+    /jail/zim/wikipedia \
+    /jail/zim/wikiquote \
+    /jail/zim/wikisource \
+    /jail/zim/wikiversity \
+    /jail/zim/wikivoyage \
+    /jail/zim/wiktionary \
+    /jail/zim/zimit