Rework local dev/test container setup #1196

docs/_locale/nl/LC_MESSAGES/how-to/development.po

@@ -68,8 +68,8 @@ msgid "Other DB engines can be tested as well, but the CI will take care of it a
 msgstr "Andere DB-engines kunnen ook getest worden, maar dit doet de CI ook al. De SQLite-engine ondersteunt overigens 99%% van de features die DMSR-reader nodig heeft. Daarnaast draait deze ook in-memory, waardoor de tests weer sneller draaien."
 
 #: ../../how-to/development.rst:65
-msgid "When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container."
-msgstr "Wanneer je PyCharm gebruikt kun je een nieuwe Interpreter toevoegen die Docker Compose gebruikt. Selecteer ``dsmr-app`` en stel ``/opt/venv/bin/python`` in als interpreter path. Nu zouden alle gebruikte dependencies goed gemapt moeten worden vanuit de container."
+msgid "When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dev-dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container."
+msgstr "Wanneer je PyCharm gebruikt kun je een nieuwe Interpreter toevoegen die Docker Compose gebruikt. Selecteer ``dev-dsmr-app`` en stel ``/opt/venv/bin/python`` in als interpreter path. Nu zouden alle gebruikte dependencies goed gemapt moeten worden vanuit de container."
 
 #: ../../how-to/development.rst:69
 msgid "Initial data to develop with"
@@ -116,8 +116,8 @@ msgid "Running DSMR-reader locally"
 msgstr "DSMR-reader lokaal draaien"
 
 #: ../../how-to/development.rst:115
-msgid "When running it with the default Docker compose config, the ``dsmr-app`` `Django Development Server application <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ will be accessible at: ``http://localhost:8000/``."
-msgstr "Wanneer je de standaard Docker compose configuratie gebruikt, is de ``dsmr-app`` `Django Development Server applicatie <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ toegankelijk op: ``http://localhost:8000/``."
+msgid "When running it with the default Docker compose config, the ``dev-dsmr-app`` `Django Development Server application <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ will be accessible at: ``http://localhost:8000/``."
+msgstr "Wanneer je de standaard Docker compose configuratie gebruikt, is de ``dev-dsmr-app`` `Django Development Server applicatie <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ toegankelijk op: ``http://localhost:8000/``."
 
 #: ../../how-to/development.rst:117
 msgid "Any Python code changes you make will cause the Django Development Server to reload itself automatically."
@@ -152,8 +152,8 @@ msgid "Translations"
 msgstr "Vertalingen"
 
 #: ../../how-to/development.rst:146
-msgid "You can find the translations (.PO files) for the main application in ``dsmrreader/locales/``. To regenerate them, just execute the ``docker exec -it dsmr-app poetry run ./tools/check-translations.sh`` script."
-msgstr "Je kunt de vertalingen (.PO-bestanden) voor de hoofdapplicatie vinden in ``dsmrreader/locales/``. Om ze te hergenereren voer het ``docker exec -it dsmr-app poetry run ./tools/check-translations.sh`` script uit, zodat een van de tests de vertalingen opnieuw checkt."
+msgid "You can find the translations (.PO files) for the main application in ``dsmrreader/locales/``. To regenerate them, just execute the ``docker exec -it dev-dsmr-app poetry run ./tools/check-translations.sh`` script."
+msgstr "Je kunt de vertalingen (.PO-bestanden) voor de hoofdapplicatie vinden in ``dsmrreader/locales/``. Om ze te hergenereren voer het ``docker exec -it dev-dsmr-app poetry run ./tools/check-translations.sh`` script uit, zodat een van de tests de vertalingen opnieuw checkt."
 
 #: ../../how-to/development.rst:151
 msgid "Editing documentation"

docs/how-to/development.rst

@@ -48,7 +48,7 @@ Setting up a development environment using Docker
 
 - Containers built? See if this command works::
 
-    docker exec -it dsmr-app poetry run /app/manage.py check
+    docker exec -it dev-dsmr-app poetry run /app/manage.py check
 
     # Expected output: "System check identified no issues (0 silenced)"
 
@@ -60,7 +60,7 @@ Setting up a development environment using Docker
 
     Other DB engines can be tested as well, but the CI will take care of it anyway. The SQLite engine matches 99%% of the features DSMR-reader requires and it also runs in-memory, speeding up tests.
 
-- When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container.
+- When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dev-dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container.
 
 
 Initial data to develop with
@@ -74,7 +74,7 @@ To be honest, the best initial/fixture data is simply a backup of your own syste
 
     After importing the backup of your production system, simply run::
 
-        docker exec -it dsmr-app poetry run /app/manage.py development_reset
+        docker exec -it dev-dsmr-app poetry run /app/manage.py development_reset
 
     This will remove all API keys and other links to externals systems, as well as reset the admin user credentials to ``admin / admin`` (user / password).
 
@@ -98,19 +98,19 @@ Fake datalogger
 
     There is a builtin command that can somewhat fake a datalogger::
 
-        docker exec -it dsmr-app poetry run /app/manage.py dsmr_fake_datasource --with-gas --with-electricity-returned
+        docker exec -it dev-dsmr-app poetry run /app/manage.py dsmr_fake_datasource --with-gas --with-electricity-returned
 
 It will generate random data every second in a certain pattern and should be fine for basic testing. 
 
 Please note that it only inserts unprocessed readings, so you'll still have to run the following command to have the readings processed::
 
-    docker exec -it dsmr-app poetry run /app/manage.py dsmr_backend --run-once
+    docker exec -it dev-dsmr-app poetry run /app/manage.py dsmr_backend --run-once
 
 
 Running DSMR-reader locally
 ---------------------------
 
-When running it with the default Docker compose config, the ``dsmr-app`` `Django Development Server application <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ will be accessible at: ``http://localhost:8000/``.
+When running it with the default Docker compose config, the ``dev-dsmr-app`` `Django Development Server application <https://docs.djangoproject.com/en/3.2/intro/tutorial01/#the-development-server>`_ will be accessible at: ``http://localhost:8000/``.
 
 Any Python code changes you make will cause the Django Development Server to reload itself automatically.
 
@@ -122,11 +122,11 @@ DSMR-reader's test coverage should remain as high as possible, however this does
 
 The easiest way to run tests is to use the SQLite (in-memory) tests::
 
-    docker exec -it dsmr-app poetry run ./tools/quick-test.sh
+    docker exec -it dev-dsmr-app poetry run ./tools/quick-test.sh
     
 To test a single app within DSMR-reader, just append it::
 
-    docker exec -it dsmr-app poetry run ./tools/quick-test.sh dsmr_frontend
+    docker exec -it dev-dsmr-app poetry run ./tools/quick-test.sh dsmr_frontend
 
 The test coverage should be visible in the terminal after running tests.
 There are detailed HTML pages available as well, after each test run, in ``coverage_report/html/index.html``. 
@@ -142,7 +142,7 @@ Translations
 ------------
 
 You can find the translations (.PO files) for the main application in ``dsmrreader/locales/``.
-To regenerate them, just execute the ``docker exec -it dsmr-app poetry run ./tools/check-translations.sh`` script.
+To regenerate them, just execute the ``docker exec -it dev-dsmr-app poetry run ./tools/check-translations.sh`` script.
 
 
 Editing documentation

provisioning/container/Containerfile-dev

@@ -20,13 +20,19 @@ RUN apk add --update \
     mariadb-dev \
     postgresql-client
 ENV POETRY_VIRTUALENVS_IN_PROJECT=false
+COPY ./src/pyproject.toml ./src/poetry.lock /app/
 RUN pip install poetry
 RUN poetry self add poetry-plugin-export
-RUN poetry install --no-root --with dev --without docs
+RUN poetry install --no-root --with dev
 EXPOSE 8000
 ENTRYPOINT poetry run python manage.py runserver 0.0.0.0:8000
 
 
+FROM local-dsmrreader-dev AS local-dsmrreader-test
+EXPOSE ""
+ENTRYPOINT poetry run pytest
+
+
 # -----------------------------------------------------------
 
 

provisioning/container/compose.dev.yml

@@ -5,9 +5,9 @@
 version: "3.9"
 services:
   # http://localhost:8000
-  dsmr-app:
-    container_name: dsmr-app
-    restart: unless-stopped
+  dev-dsmr-app:
+    container_name: dev-dsmr-app
+    restart: 'unless-stopped'
     build:
       context: .
       dockerfile: provisioning/container/Containerfile-dev
@@ -17,9 +17,7 @@ services:
     ports:
       - '8000:8000'
     depends_on:
-      dsmr-dev-db:
-        condition: service_healthy
-      dsmr-tests-db:
+      dev-dsmr-db:
         condition: service_healthy
       mosquitto:
         condition: service_started
@@ -28,10 +26,10 @@ services:
     env_file:
       - provisioning/container/dev.env
 
-  dsmr-dev-db:
-    container_name: dsmr-dev-db
+  dev-dsmr-db:
+    container_name: dev-dsmr-db
     image: postgres:15-alpine
-    restart: unless-stopped
+    restart: 'unless-stopped'
     ports:
       - '5432:5432'
     healthcheck:
@@ -43,13 +41,13 @@ services:
       - provisioning/container/dev.env
     volumes:
       # Persist database storage between container recreation
-      - ./.container-data/dsmr-dev-db/:/var/lib/postgresql/
+      - ./.container-data/dev-dsmr-db/:/var/lib/postgresql/
       # Only use for restoring DB-backups. E.g.: psql -U dsmrreader -f /run/host/Desktop/dsmrreader-postgresql-backup-Tuesday.sql dsmrreader
       - ~/Desktop:/run/host/Desktop/
 
   dsmr-docs-en:  # http://localhost:10000
     container_name: dsmr-docs-en
-    restart: unless-stopped
+    restart: 'unless-stopped'
     build:
       context: .
       dockerfile: provisioning/container/Containerfile-dev
@@ -61,7 +59,7 @@ services:
 
   dsmr-docs-nl:  # http://localhost:10001
     container_name: dsmr-docs-nl
-    restart: unless-stopped
+    restart: 'unless-stopped'
     build:
       context: .
       dockerfile: provisioning/container/Containerfile-dev
@@ -74,7 +72,7 @@ services:
   mosquitto:
     container_name: mosquitto-dev
     image: eclipse-mosquitto:2
-    restart: unless-stopped
+    restart: 'unless-stopped'
     volumes:
       - ./provisioning/container/mosquitto-dev/:/mosquitto/config/
     ports:
@@ -83,19 +81,33 @@ services:
   influxdb:
     container_name: influxdb-dev
     image: influxdb:2.0-alpine
-    restart: unless-stopped
+    restart: 'unless-stopped'
     ports:
       - '8086:8086'
     env_file:
       - provisioning/container/dev.env
 
 
   ##### Automated testing #####
+  tests-dsmr-app:
+    container_name: tests-dsmr-app
+    restart: 'no'
+    build:
+      context: .
+      dockerfile: provisioning/container/Containerfile-dev
+      target: local-dsmrreader-test
+    volumes:
+      - ./src:/app
+    depends_on:
+      tests-dsmr-db:
+        condition: service_healthy
+    env_file:
+      - provisioning/container/test.env
 
-  dsmr-tests-db:
-    container_name: dsmr-tests-db
+  tests-dsmr-db:
+    container_name: tests-dsmr-db
     image: postgres:15-alpine
-    restart: unless-stopped
+    restart: 'unless-stopped'
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U dsmrreader"]
       interval: 1s

provisioning/container/compose.test.yml

@@ -0,0 +1,32 @@
+######################################################################
+# Compose for DSMR-reader development only, NEVER use in production! #
+######################################################################
+
+version: "3.9"
+services:
+  tests-dsmr-app:
+    container_name: tests-dsmr-app
+    restart: 'no'
+    build:
+      context: .
+      dockerfile: provisioning/container/Containerfile-dev
+      target: local-dsmrreader-test
+    volumes:
+      - ./src:/app
+    depends_on:
+      tests-dsmr-db:
+        condition: service_healthy
+    env_file:
+      - provisioning/container/test.env
+
+  tests-dsmr-db:
+    container_name: tests-dsmr-db
+    image: postgres:15-alpine
+    restart: 'no'
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U dsmrreader"]
+      interval: 1s
+      timeout: 5s
+      retries: 20
+    env_file:
+      - provisioning/container/test.env

provisioning/container/dev.env

@@ -2,19 +2,19 @@
 # Env file for DSMR-reader development only, NEVER use in production! #
 #######################################################################
 
-### dsmr-app ###
+### dev-dsmr-app ###
 DJANGO_SETTINGS_MODULE=dsmrreader.config.development
 DJANGO_DATABASE_ENGINE=django.db.backends.postgresql
-DJANGO_DATABASE_HOST=dsmr-dev-db
+DJANGO_DATABASE_HOST=dev-dsmr-db
 DJANGO_DATABASE_PORT=5432
 DJANGO_DATABASE_NAME=dsmrreader
 DJANGO_DATABASE_USER=dsmrreader
 DJANGO_DATABASE_PASSWORD=postgres-password
-DJANGO_SECRET_KEY=development
+DJANGO_SECRET_KEY=localhost-development
 DSMRREADER_LOGLEVEL=DEBUG
 
 
-### dsmr-dev-db ###
+### dev-dsmr-db ###
 POSTGRES_DB=${DJANGO_DATABASE_NAME}
 POSTGRES_USER=${DJANGO_DATABASE_USER}
 POSTGRES_PASSWORD=${DJANGO_DATABASE_PASSWORD}

provisioning/container/test.env

@@ -2,18 +2,19 @@
 # Env file for DSMR-reader AUTOMATED TESTING only, NEVER use in production! #
 #############################################################################
 
-### dsmr-app ###
+### tests-dsmr-app ###
 DJANGO_SETTINGS_MODULE=dsmrreader.config.test
 DJANGO_DATABASE_ENGINE=django.db.backends.postgresql
-DJANGO_DATABASE_HOST=dsmr-tests-db
+DJANGO_DATABASE_HOST=tests-dsmr-db
 DJANGO_DATABASE_PORT=5432
 DJANGO_DATABASE_NAME=test_dsmrreader
 DJANGO_DATABASE_USER=testuser
 DJANGO_DATABASE_PASSWORD=testpasswd
+DJANGO_SECRET_KEY=automated-tests
 DSMRREADER_LOGLEVEL=ERROR
 
 
-### test-db-postgres ###
+### tests-dsmr-db ###
 POSTGRES_DB=${DJANGO_DATABASE_NAME}
 POSTGRES_USER=${DJANGO_DATABASE_USER}
 POSTGRES_PASSWORD=${DJANGO_DATABASE_PASSWORD}