diff --git a/documentation/docs/explained/about.md b/documentation/docs/explained/about.md index 818d050ff..4078e3c3a 100644 --- a/documentation/docs/explained/about.md +++ b/documentation/docs/explained/about.md @@ -1,3 +1,7 @@ +# About DSMR-reader + +[TOC] + DSMR-reader project information. ## Project goals diff --git a/documentation/docs/explained/hall-of-fame.md b/documentation/docs/explained/hall-of-fame.md index a6af162ca..282251e75 100644 --- a/documentation/docs/explained/hall-of-fame.md +++ b/documentation/docs/explained/hall-of-fame.md @@ -1,3 +1,5 @@ +# Contributions + DSMR-reader is originally created and authored by: - Dennis Siemensma diff --git a/documentation/docs/how-to/development.md b/documentation/docs/how-to/development.md new file mode 100644 index 000000000..06c5c952e --- /dev/null +++ b/documentation/docs/how-to/development.md @@ -0,0 +1,57 @@ +# Developing DSMR-reader + +## Setting up a development environment using Docker + +> I'm using JetBrain's PyCharm IDE for local development, which has builtin support for Git and Docker. +> Therefor some steps or information below may or may not match your own development stack. + +- Install Docker on your system. E.g. [how-to for Ubuntu](https://docs.docker.com/engine/install/ubuntu/) and consider [Docker rootless](https://docs.docker.com/engine/security/rootless/). + +- Clone DSMR-reader repository from GitHub: + +```shell + git clone > + cd dsmr-reader/ +``` +- Symlink Docker files required (or just copy them): + +```shell + # Either symlink + ln -s provisioning/container/compose.dev.yml compose.yml + + # Or copy + cp provisioning/container/compose.dev.yml compose.yml +``` + +- Try running Docker (compose): +```shell + # This should build all the containers for local development + docker-compose up -d +``` +- Containers built? See if this command works: +```shell + docker exec -it dev-dsmr-app poetry run /app/manage.py check + + # Expected output: "System check identified no issues (0 silenced)" +``` +- Now check whether tests run well in SQLite: +```shell + ./tools/test-quick.sh +``` + + +> Other DB engines can be tested as well, but the CI will take care of it anyway. The SQLite engine matches 99% of the features DSMR-reader requires and it also runs in-memory, speeding up tests. + +- When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dev-dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container. + + +## Running DSMR-reader locally + +When running it with the default Docker compose config, the Django Development Server application will be accessible at: + +- [http://localhost:8000](http://localhost:8000/) + +Any Python code changes you make will cause the Django Development Server to reload itself automatically. + +## Other stuff +There is some more to it, such as tests, translations and documentation. If you ever need to work on those, just see how similar stuff works in the project. Or ask for more information. \ No newline at end of file diff --git a/documentation/docs/how-to/donating.md b/documentation/docs/how-to/donating.md new file mode 100644 index 000000000..203c6ce3d --- /dev/null +++ b/documentation/docs/how-to/donating.md @@ -0,0 +1,8 @@ +# Donating + +If you really like DSMR-reader, feel free to use it without anything in return, you're welcome! ❤ + + +Still want to return a small token of appreciation? You can: + +- [Donate with PayPal](https://www.paypal.com/donate/?hosted_button_id=GCJZF72C28QM4) diff --git a/documentation/docs/index.md b/documentation/docs/index.md index cacb77965..c06c3f05f 100644 --- a/documentation/docs/index.md +++ b/documentation/docs/index.md @@ -7,3 +7,10 @@ # DSMR-reader documentation See sidebar for a more specific navigation or below for a (global) overview. +## [About DSMR-reader](explained/about.md#about-dsmr-reader) +## [Contributions](explained/hall-of-fame.md#contributions) +## [Developing DSMR-reader](how-to/development.md#developing-dsmr-reader) +## [Donating](how-to/donating.md#donating) +## [API](reference/API.md#api) +## [Environment variables](reference/environment-variables.md#environment-variables) +## [Plugins](reference/plugins.md#plugins) diff --git a/documentation/docs/reference/API.md b/documentation/docs/reference/API.md index 53f59e5b3..4f117ddc8 100644 --- a/documentation/docs/reference/API.md +++ b/documentation/docs/reference/API.md @@ -1,8 +1,10 @@ -The application has an API, allowing you to insert/create readings and retrieve statistics. +# API + +The application has a REST API, allowing you to insert/create readings and retrieve statistics. ## Configuration -You can access the API-documentation by selecting the **Support** menu item in DSMR-reader. +You can access the API documentation by selecting the **Support** menu item in DSMR-reader. ![Screenshot](../static/screenshots/api/support.png) diff --git a/documentation/docs/reference/environment-variables.md b/documentation/docs/reference/environment-variables.md new file mode 100644 index 000000000..b545aa04a --- /dev/null +++ b/documentation/docs/reference/environment-variables.md @@ -0,0 +1,294 @@ +# Environment variables + +[TOC] + +**Django settings/overrides** + +DSMR-reader utilizes the Python Django framework. All settings below directly affect or override Django. + + +## ``DJANGO_SECRET_KEY`` + +!!! failure inline end "" + + This setting is **required**. + +The secret key Django should use for some security internals. Should be unique and kept a secret. Generate a random value + +See [``SECRET_KEY`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#secret-key) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_ENGINE`` + +!!! failure inline end "" + + This setting is **required**. + +The database engine to use. Officially DSMR-reader only supports PostgreSQL, but others supported by Django may work as well. + +See [``DATABASES.ENGINE`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#engine) for more information about the setting and what it does. + +!!! danger + + Use engines other than ``django.db.backends.postgresql`` at your own risk! + +---- + +## ``DJANGO_DATABASE_HOST`` + +!!! failure inline end "" + + This setting is **required**. But also depends on the engine used (and may be optional in some situations). + +Database connection setting. For the default engine it's the host name to connect to. + +See [``DATABASES.HOST`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#host) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_PORT`` + +!!! failure inline end "" + + This setting is **required**. But also depends on the engine used (and may be optional in some situations). + +Database connection setting. For the default engine it's the host port to connect to. + +See [``DATABASES.PORT`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#port) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_NAME`` + +!!! failure inline end "" + + This setting is **required**. But also depends on the engine used (and may be optional in some situations). + +Database connection setting. For the default engine it's the database name to connect to. + +See [``DATABASES.NAME`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#name) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_USER`` + +!!! failure inline end "" + + This setting is **required**. But also depends on the engine used (and may be optional in some situations). + +Database connection setting. For the default engine it's the database username to connect with. + +See [``DATABASES.USER`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#user) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_PASSWORD`` + +!!! failure inline end "" + + This setting is **required**. But also depends on the engine used (and may be optional in some situations). + +Database connection setting. For the default engine it's the database password to connect with. + +See [``DATABASES.PASSWORD`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#password) for more information about the setting and what it does. + +---- + +## ``DJANGO_DATABASE_CONN_MAX_AGE`` + +!!! info inline end "" + + This setting is **optional**. + +Database connection setting. For the default engine is not needed. + +See [``DATABASES.CONN_MAX_AGE`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#conn-max-age) for more information about the setting and what it does. + +---- + +## ``DJANGO_TIME_ZONE`` + +!!! info inline end "" + + This setting is **optional**. + +The timezone Django should use. Alter at your own risk. Omit to use the default, using the CET/CEST timezone (applicable to the Netherlands). + +See [``TIME_ZONE`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-TIME_ZONE) for more information about the setting and what it does. + + +---- + + +## ``DJANGO_STATIC_URL`` + +!!! example inline end "" + + This setting is **situational**. + +See [``STATIC_URL`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#static-url) for more information about the setting and what it does. + +---- + + +## ``DJANGO_FORCE_SCRIPT_NAME`` + +!!! example inline end "" + + This setting is **situational**. + +See [``FORCE_SCRIPT_NAME`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#force-script-name) for more information about the setting and what it does. + +---- + +## ``DJANGO_USE_X_FORWARDED_HOST`` + +!!! example inline end "" + + This setting is **situational**. + +See [``USE_X_FORWARDED_HOST`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#use-x-forwarded-host) for more information about the setting and what it does. + +---- + +## ``DJANGO_USE_X_FORWARDED_PORT`` + +!!! example inline end "" + + This setting is **situational**. + +See [``USE_X_FORWARDED_PORT`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#use-x-forwarded-port) for more information about the setting and what it does. + +---- + +## ``DJANGO_X_FRAME_OPTIONS`` + +!!! example inline end "" + + This setting is **situational**. + +See [``X_FRAME_OPTIONS`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#x-frame-options) for more information about the setting and what it does. + +---- + +## ``DJANGO_STATIC_ROOT`` + +!!! example inline end "" + + This setting is **situational**. + +See [``STATIC_ROOT`` in Django docs](https://docs.djangoproject.com/en/4.2/ref/settings/#static-root) for more information about the setting and what it does. + +---- + + +**DSMR-reader settings** + +These settings are for this project only. + +## ``DSMRREADER_ADMIN_USER`` + +!!! example inline end "" + + This setting is **situational**. + +The username of the **webinterface** (super)user to create when running ``./manage.py dsmr_superuser``. + +---- + +## ``DSMRREADER_ADMIN_PASSWORD`` + +!!! example inline end "" + + This setting is **situational**. + +The password of the ``DSMRREADER_ADMIN_USER`` user to create (or update if the user exists) when running ``./manage.py dsmr_superuser``. + +---- + + +## ``DSMRREADER_LOGLEVEL`` + +!!! example inline end "" + + This setting is **situational**. + +The log level DSMR-reader should use. Choose either: + +- ``ERROR`` (omit for this default) +- ``WARNING`` +- ``DEBUG`` + +The latter should only be used for debugging DSMR-reader and pinpointing weird issues. + +---- + +## ``DSMRREADER_PLUGINS`` + +!!! example inline end "" + + This setting is **situational**. + +The [plugins DSMR-reader should use](./plugins.md). Omit to use the default of no plugins. +Note that this should be a comma separated list when specifying multiple plugins. E.g.: + +```ini +DSMRREADER_PLUGINS=dsmr_plugins.modules.plugin_name1 +DSMRREADER_PLUGINS=dsmr_plugins.modules.plugin_name1,dsmr_plugins.modules.plugin_name2 +``` + +---- + +## ``DSMRREADER_SUPPRESS_STORAGE_SIZE_WARNINGS`` + +!!! example inline end "" + + This setting is **situational**. + +Whether to suppress any warnings regarding too many readings stored or the database size. +Set it to ``True`` to **disable the warnings** or omit it to use the default (= ``False``). + +!!! danger + + Suppress warnings at your own risk. + +---- + +## ``DSMRREADER_MQTT_MAX_MESSAGES_IN_QUEUE`` + +!!! example inline end "" + + This setting is **situational**. + +The maximum amount of MQTT messages queued in DSMR-reader until new ones will be **rejected**. No need to tweak this for healthy installations. + +This prevents creating an infinite backlog of messages queued. +For example when the pile of unsent messages keeps increasing and DSMR-reader is unable to send them faster than new ones are created. + +However, you may increase the maximum for whatever reason along your local setup. +Omit to use the default (a few thousand). + +!!! danger + + Increase queue size at your own risk. + +---- + +## ``DSMRREADER_MQTT_MAX_CACHE_TIMEOUT`` + +!!! example inline end "" + + This setting is **situational**. + +Updating MQTT topics consecutively **with the same value has no effect**, depending on its usage in your setup. + +DSMR-reader sends MQTT messages to your broker with a ``retain`` flag, resulting the broker keeping the last value received for every topic. +Which means that any MQTT subscribers should always receive the retained value, even when DSMR-reader has no updates. + +DSMR-reader can be set to *cache the last value sent for each topic*. It will cause DSMR-reader to **not send the same value to the same topic consecutively** (within the caching duration). +This may greatly reduce the number of MQTT messages sent when there is nothing to update. If the value of a topic changes, DSMR-reader will still send the updated value. + +!!! danger + + Enable caching only if you understand what it does. diff --git a/documentation/docs/reference/plugins.md b/documentation/docs/reference/plugins.md new file mode 100644 index 000000000..6a40497c9 --- /dev/null +++ b/documentation/docs/reference/plugins.md @@ -0,0 +1,307 @@ +# Plugins + +The application allows you to create and add plugins, hooking on certain events triggered. + +!!! danger + + These are very technical scripts, and you should **not use any of them** unless you understand what they do. + Most of them originate from single-user feature requests in the form of a workaround instead. + + Refrain from using them if you're not sure. + + +## Configuration + +You can create plugins in their own file in ``dsmr_plugins/modules/plugin_name.py``, +where ``plugin_name`` is the name of your plugin. + +Please make sure the ``plugin_name``, + +* is lowercase (``plugin_name`` and **not** ``PLUGIN_NAME``), +* does not contains spaces or dashes, only use underscores and do not start the name with a digit. + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + +Your plugin file is imported once, so you should make sure to hook any events you want. + + +## Events / Signals +These are either dispatched by the Django framework or the application at some point. + +### ``dsmr_backend.signals.backend_called`` + +Called by each iteration of the backend. Please use with caution, as it may block all backend operations when used improperly. + + +### ``dsmr_pvoutput.signals.pvoutput_upload`` +Called by dsmr_pvoutput just before uploading data to PVOutput. The ``data`` kwarg contains the data to be sent. + + +### ``dsmr_datalogger.signals.raw_telegram`` +Called by dsmr_datalogger when receiving or reading a telegram string. The ``data`` kwarg contains the raw telegram string. + + +### ``dsmr_notification.signals.notification_sent`` +Called by dsmr_notification just after dispatching a notification. The ``title`` kwarg contains the notification title, ``message`` contains the message body. + + +### ``django.db.models.signals.post_save`` +Called by Django after saving new records to the database. Can be bound to the ``DayStatistics`` model for example, to process daily statistics elsewhere. + + +### Other signals +More signals may be available for use, please be careful when binding Django save-signals as it may impact performance. + + +## Examples: + +### Example #1: Upload data to second PVOutput account +This is an example of issue `#407 `_, requesting the feature to upload data to a second PVOuput account. + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + + ```ini + DSMRREADER_PLUGINS=dsmr_plugins.modules.secondary_pvoutput_upload + ``` + + +Plugin file ``dsmr_plugins/modules/secondary_pvoutput_upload.py`` (new file): + +```python +import requests + +from django.dispatch import receiver + +from dsmr_pvoutput.models.settings import PVOutputAddStatusSettings +from dsmr_pvoutput.signals import pvoutput_upload + + +@receiver(pvoutput_upload) +def handle_secondary_pvoutput_upload(**kwargs): + print(' - Uploading the same data to PVOutput using plugin: {}'.format(kwargs['data'])) + + response = requests.post( + PVOutputAddStatusSettings.API_URL, + headers={ + 'X-Pvoutput-Apikey': 'XXXXX', + 'X-Pvoutput-SystemId': 'YYYYY', + }, + data=kwargs['data'] + ) + + if response.status_code != 200: + print(' [!] PVOutput upload failed (HTTP {}): {}'.format(response.status_code, response.text)) + +``` +!!! tip + + Note that the ``XXXXX`` and ``YYYYY`` variables should be replace by your second set of PVOutput API credentials. + + +### Example #2: Forwarding raw telegram data to another serial port +This is an example of issue [#557](https://github.com/dsmrreader/dsmr-reader/issues/557), allowing raw DSMR telegrams to be forwarded to another serial port. + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + + ```ini + DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_raw_telegram_to_serial + ``` + + +Plugin file ``dsmr_plugins/modules/forward_raw_telegram_to_serial.py`` (new file): + +```python +import serial + +from django.dispatch import receiver + +from dsmr_datalogger.signals import raw_telegram +import dsmr_datalogger.services.datalogger + + +@receiver(raw_telegram) +def handle_forward_raw_telegram_to_serial(**kwargs): + DEST_PORT = '/dev/ttyUSBvA' + connection_parameters = dsmr_datalogger.services.datalogger.get_dsmr_connection_parameters() + + serial_handle = serial.Serial() + serial_handle.port = DEST_PORT + serial_handle.baudrate = connection_parameters['baudrate'] + serial_handle.bytesize = connection_parameters['bytesize'] + serial_handle.parity = connection_parameters['parity'] + serial_handle.stopbits = serial.STOPBITS_ONE + serial_handle.xonxoff = 1 + serial_handle.rtscts = 0 + serial_handle.timeout = 1 + serial_handle.write_timeout = 0.2 + + try: + serial_handle.open() + bytes_sent = serial_handle.write(bytes(kwargs['data'], 'utf-8')) + except Exception as error: + print(error) + else: + print(' >>> Sent {} bytes to {}'.format(bytes_sent, DEST_PORT)) + + serial_handle.close() + +``` + +!!! tip + + Note that the ``/dev/ttyUSBvA`` variable should be changed to the serial port used in your own situation. + + +### Example #3: Forwarding raw telegram data to another instance by API +This can be quite handy if you run multiple instances of DSMR-reader (i.e.: RaspberryPI + somewhere in cloud). + + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + + ```ini + DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_raw_telegram_to_api + ``` + +Plugin file ``dsmr_plugins/modules/forward_raw_telegram_to_api.py`` (new file): + +```python +import requests +import logging + +from django.dispatch import receiver + +from dsmr_datalogger.signals import raw_telegram + + +@receiver(raw_telegram) +def handle_forward_raw_telegram_to_api(**kwargs): + API_HOST = 'https://YOUR-DSMR-HOST' # Note: Check whether you use HTTP or SSL (HTTPS). + API_KEY = 'YOUR-API-KEY' + TIMEOUT = 5 # A low timeout prevents the application from hanging, when the server is unavailable. + + try: + # Register telegram by simply sending it to the application with a POST request. + response = requests.post( + '{}/api/v1/datalogger/dsmrreading'.format(API_HOST), + headers={'X-AUTHKEY': API_KEY}, + data={'telegram': kwargs['data']}, + timeout=TIMEOUT + ) + except Exception as error: + return logging.error(error) + + if response.status_code != 201: + logging.error('Server Error forwarding telegram: {}'.format(response.text)) + +``` + +!!! tip + + Note that the ``API_HOST``, ``API_KEY`` and ``TIMEOUT`` variables should be changed to your own preferences. + + +### Example #4: Forwarding DSMR readings in JSON format to some API +Use this to send DSMR readings in JSON format to some (arbitrary) API. + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + + ```ini + DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_json_dsmrreading_to_api + ``` + + +Plugin file ``dsmr_plugins/modules/forward_json_dsmrreading_to_api.py`` (new file): + +```python + +import requests +import json + +from django.dispatch import receiver +from django.core import serializers +from django.utils import timezone +import django.db.models.signals + +from dsmr_datalogger.models.reading import DsmrReading + +@receiver(django.db.models.signals.post_save, sender=DsmrReading) +def handle_forward_json_dsmrreading_to_api(sender, instance, created, raw, **kwargs): + if not created or raw: + return + + instance.timestamp = timezone.localtime(instance.timestamp) + + if instance.extra_device_timestamp: + instance.extra_device_timestamp = timezone.localtime(instance.extra_device_timestamp) + + serialized = json.loads(serializers.serialize('json', [instance])) + json_string = json.dumps(serialized[0]['fields']) + + try: + requests.post( + 'https://YOUR-DSMR-HOST/api/endpoint/', + data=json_string, + # A low timeout prevents DSMR-reader from hanging, when the remote server is unreachable. + timeout=5 + ) + except Exception as error: + print('forward_json_dsmrreading_to_api:', error) + +``` + +### Example #5: Read telegrams using DSMRloggerWS API + + +!!! info + + Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information see [Environment variables](./environment-variables.md). + + ```ini + DSMRREADER_PLUGINS=dsmr_plugins.modules.poll_dsmrloggerws_api + ``` + +Plugin file ``dsmr_plugins/modules/poll_dsmrloggerws_api.py`` (new file): + +```python +import requests + +from django.dispatch import receiver + +from dsmr_backend.signals import backend_called +import dsmr_datalogger.services.datalogger + + +# Preverve a low timeout to prevent the entire backend process from hanging too long. +DSMRLOGGERWS_ENDPOINT = 'http://localhost/api/v1/sm/telegram' +DSMRLOGGERWS_TIMEOUT = 5 + + +@receiver(backend_called) +def handle_backend_called(**kwargs): + response = requests.get(DSMRLOGGERWS_ENDPOINT, + timeout=DSMRLOGGERWS_TIMEOUT) + + if response.status_code != 200: + print(' [!] DSMRloggerWS plugin: Telegram endpoint failed (HTTP {}): {}'.format( + response.status_code, + response.text + )) + return + + dsmr_datalogger.services.datalogger.telegram_to_reading(data=response.text) + +``` + +!!! tip + + Note that you might need to update the ``http://localhost`` value to your own situation. diff --git a/legacy-docs/how-to/development.rst b/legacy-docs/how-to/development.rst deleted file mode 100644 index 2fcb4fb7a..000000000 --- a/legacy-docs/how-to/development.rst +++ /dev/null @@ -1,173 +0,0 @@ -Developing: Localhost -===================== - - -.. contents:: - :depth: 2 - - -Setting up a development environment using Docker -------------------------------------------------- - -.. note:: - - I'm using JetBrain's PyCharm IDE for local development, which has builtin support for Git and Docker. - Therefor some steps or information below may or may not match your own development stack. - -- Install Docker on your system. E.g. Ubuntu: https://docs.docker.com/engine/install/ubuntu/ and consider rootless: https://docs.docker.com/engine/security/rootless/ - -- Clone DSMR-reader repository from GitHub:: - - git clone ... (your fork) - cd dsmr-reader/ - -- Symlink Docker files required (or just copy them):: - - # Either symlink - ln -s provisioning/container/compose.dev.yml compose.yml - - # Or copy - cp provisioning/container/compose.dev.yml compose.yml - -- Copy Django settings template:: - - cp provisioning/django/settings.py.template dsmrreader/settings.py - -- Open the settings template ``dsmrreader/settings.py`` you've copied and **replace**:: - - from dsmrreader.config.production import * - -- With:: - - from dsmrreader.config.development import * - -- Try running Docker (compose):: - - # This should build all the containers for local development - docker-compose up -d - -- Containers built? See if this command works:: - - docker exec -it dev-dsmr-app poetry run /app/manage.py check - - # Expected output: "System check identified no issues (0 silenced)" - -- Now check whether tests run well in SQLite:: - - ./tools/test-quick.sh - -.. tip:: - - Other DB engines can be tested as well, but the CI will take care of it anyway. The SQLite engine matches 99%% of the features DSMR-reader requires and it also runs in-memory, speeding up tests. - -- When using PyCharm, you can add a new Interpreter using Docker Compose. Just select ``dev-dsmr-app`` and set ``/opt/venv/bin/python`` as interpreter path. It should now map all dependencies used/installed in the container. - - -Initial data to develop with ----------------------------- - -To be honest, the best initial/fixture data is simply a backup of your own system in production. - -.. danger:: - - Please note that you should not run any (backend) processes in your DSMR-reader development environment, **until you've unlinked all external services**. - - After importing the backup of your production system, simply run:: - - docker exec -it dev-dsmr-app poetry run /app/manage.py development_reset - - This will remove all API keys and other links to externals systems, as well as reset the admin user credentials to ``admin / admin`` (user / password). - -Just import it as you should on your RaspberryPi. Copy a database backup to ``/var/lib/postgresql/`` on your PC and import it:: - - # NOTE: Some what broken when using Docker. But this was the legacy method: - # Create empty database if it does not exist yet. - sudo -u postgres createdb -O dsmrreader dsmrreader - - # For .SQL - sudo -u postgres psql dsmrreader -f - - # For .SQL.GZ - zcat | sudo -u postgres psql dsmrreader - - -Fake datalogger ---------------- - -.. tip:: - - There is a builtin command that can somewhat fake a datalogger:: - - docker exec -it dev-dsmr-app poetry run /app/manage.py dsmr_fake_datasource --with-gas --with-electricity-returned - -It will generate random data every second in a certain pattern and should be fine for basic testing. - -Please note that it only inserts unprocessed readings, so you'll still have to run the following command to have the readings processed:: - - docker exec -it dev-dsmr-app poetry run /app/manage.py dsmr_backend --run-once - - -Running DSMR-reader locally ---------------------------- - -When running it with the default Docker compose config, the ``dev-dsmr-app`` `Django Development Server application `_ will be accessible at: ``http://localhost:8000/``. - -Any Python code changes you make will cause the Django Development Server to reload itself automatically. - - -Tests and coverage ------------------- - -DSMR-reader's test coverage should remain as high as possible, however this does not guarantee the quality of tests, so find a sweet spot for coverage whenever possible. - -The easiest way to run tests is to use the SQLite (in-memory) tests:: - - docker exec -it dev-dsmr-app poetry run ./tools/quick-test.sh - -To test a single app within DSMR-reader, just append it:: - - docker exec -it dev-dsmr-app poetry run ./tools/quick-test.sh dsmr_frontend - -The test coverage should be visible in the terminal after running tests. -There are detailed HTML pages available as well, after each test run, in ``coverage_report/html/index.html``. -Just open it with your browser to view the test coverage of each file and line. - -.. note:: - - A side effect of running tests is that it may also regenerate .PO files from the ``docs/`` folder. - If you did not make any changes there, your should just ignore those changed files and revert them. - - -Translations ------------- - -You can find the translations (.PO files) for the main application in ``dsmrreader/locales/``. -To regenerate them, just execute the ``docker exec -it dev-dsmr-app poetry run ./tools/check-translations.sh`` script. - - -Editing documentation ---------------------- - -The documentation is part of the repository and can be generated (automatically) with Sphinx. - -By default the Docker compose file should create and run a docs container for each language supported. - -- English:: - - http://127.0.0.1:10000 - -- Dutch:: - - http://127.0.0.1:10001 - -Any changes you make will be reflected instantly in the browser, as Sphinx continuously checks for changed files. - - -Translating documentation -------------------------- - -Translations are done using gettext and .PO files. Regenerate the .PO files with:: - - docker exec -it dsmr-docs bash -c 'poetry run make gettext && poetry run sphinx-intl update --line-width=-1 -p _build/locale -l nl' - -The .PO files in ``docs/locale`` should be regenerated now. You can use the open-source tool ``poedit`` to view and translate the files. diff --git a/legacy-docs/how-to/donate/thanks.rst b/legacy-docs/how-to/donate/thanks.rst deleted file mode 100644 index 2e0ab3677..000000000 --- a/legacy-docs/how-to/donate/thanks.rst +++ /dev/null @@ -1,11 +0,0 @@ -Thanks! -======= - - -If you really like DSMR-reader, feel free to use it without anything in return, you're welcome! ❤ - ----- - -When for some reason you still want to give a small token of appreciation, you can: - -- `Donate with PayPal `_ diff --git a/legacy-docs/reference/api.rst b/legacy-docs/reference/api.rst deleted file mode 100644 index 3d15df781..000000000 --- a/legacy-docs/reference/api.rst +++ /dev/null @@ -1,56 +0,0 @@ -API -=== - -The application has an API allowing you to insert/create readings and retrieve statistics. - - -.. contents:: - :depth: 2 - - -Changelog ---------- - -.. versionadded:: 4.7.0 - - - Added new endpoint to create day statistics - -.. versionadded:: 4.4.0 - - - Added new endpoint for application monitoring - -.. versionchanged:: 4.0.0 - - - Removed ``/api/v2/application/status`` - -.. deprecated:: 3.12 - - - Deprecated ``/api/v2/application/status`` - - -Documentation -------------- - -.. hint:: - - The API-documentation has been moved to your local installation since DSMR-reader ``v3.1``. - - You can access it by selecting the ``About & Support`` menu item in DSMR-reader. - - -Configuration -------------- - -Enable API -^^^^^^^^^^ - -The API is disabled by default in the application. You may enable it :doc:`in the configuration`. - -.. image:: ../_static/screenshots/v5/admin/apisettings.png - :target: ../_static/screenshots/v5/admin/apisettings.png - :alt: API admin settings - -API key -^^^^^^^ - -A random API key is generated for you by default. You can always view or update it :doc:`in the configuration`. diff --git a/legacy-docs/reference/env-settings.rst b/legacy-docs/reference/env-settings.rst index 5d3f60877..17d4bb214 100644 --- a/legacy-docs/reference/env-settings.rst +++ b/legacy-docs/reference/env-settings.rst @@ -13,7 +13,7 @@ Django settings/overrides ------------------------- DSMR-reader utilizes the Python Django framework. -All settings below directly affect or override Django, and therefor your DSMR-reader installation as well. +All settings below directly affect or override Django. .. tip:: diff --git a/legacy-docs/reference/plugins.rst b/legacy-docs/reference/plugins.rst deleted file mode 100644 index a1d74b2bc..000000000 --- a/legacy-docs/reference/plugins.rst +++ /dev/null @@ -1,300 +0,0 @@ -Plugins -======= - -The application allows you to create and add plugins, hooking on certain events triggered. - - -.. contents:: - -Configuration -~~~~~~~~~~~~~ - -You can create plugins in their own file in ``dsmr_plugins/modules/plugin_name.py``, -where ``plugin_name`` is the name of your plugin. - -Please make sure the ``plugin_name``, - -* is lowercase (``plugin_name`` and **not** ``PLUGIN_NAME``), -* does not contains spaces or dashes, only use underscores and do not start the name with a digit. - -.. seealso:: - - Add the **dotted** path as ``DSMRREADER_PLUGINS`` env var. For more information :doc:`see DSMRREADER_PLUGINS in Env Settings<./env-settings>`. - -Your plugin file is imported once, so you should make sure to hook any events you want. - -And finally, make sure to **restart the application** to reflect the changes. -Do so by executing the following as **root user or sudoer**:: - - sudo supervisorctl restart all - - -Events / Signals ----------------- -These are either dispatched by the Django framework or the application at some point. - -``dsmr_backend.signals.backend_called`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Called by each iteration of the backend. Please use with caution, as it may block all backend operations when used improperly. - - -``dsmr_pvoutput.signals.pvoutput_upload`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Called by dsmr_pvoutput just before uploading data to PVOutput. The ``data`` kwarg contains the data to be sent. - - -``dsmr_datalogger.signals.raw_telegram`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Called by dsmr_datalogger when receiving or reading a telegram string. The ``data`` kwarg contains the raw telegram string. - - -``dsmr_notification.signals.notification_sent`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Called by dsmr_notification just after dispatching a notification. The ``title`` kwarg contains the notification title, ``message`` contains the message body. - - -``django.db.models.signals.post_save`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Called by Django after saving new records to the database. Can be bound to the ``DayStatistics`` model for example, to process daily statistics elsewhere. - - -Other -^^^^^ -More signals may be available for use, please be careful when binding Django save-signals as it may impact performance. - - -Examples: -~~~~~~~~~ - -Example #1: Upload data to second PVOutput account --------------------------------------------------- -This is an example of issue `#407 `_, requesting the feature to upload data to a second PVOuput account. - -.. seealso:: - - :doc:`DSMRREADER_PLUGINS configuration<./env-settings>`:: - - DSMRREADER_PLUGINS=dsmr_plugins.modules.secondary_pvoutput_upload - - -Plugin file ``dsmr_plugins/modules/secondary_pvoutput_upload.py`` (new file):: - - import requests - - from django.dispatch import receiver - - from dsmr_pvoutput.models.settings import PVOutputAddStatusSettings - from dsmr_pvoutput.signals import pvoutput_upload - - - @receiver(pvoutput_upload) - def handle_secondary_pvoutput_upload(**kwargs): - print(' - Uploading the same data to PVOutput using plugin: {}'.format(kwargs['data'])) - - response = requests.post( - PVOutputAddStatusSettings.API_URL, - headers={ - 'X-Pvoutput-Apikey': 'XXXXX', - 'X-Pvoutput-SystemId': 'YYYYY', - }, - data=kwargs['data'] - ) - - if response.status_code != 200: - print(' [!] PVOutput upload failed (HTTP {}): {}'.format(response.status_code, response.text)) - -.. attention:: - - Note that the ``XXXXX`` and ``YYYYY`` variables should be replace by your second set of PVOutput API credentials. - - -Example #2: Forwarding raw telegram data to another serial port ---------------------------------------------------------------- -This is an example of issue `#557 `_, allowing raw DSMR telegrams to be forwarded to another serial port. - -.. seealso:: - - :doc:`DSMRREADER_PLUGINS configuration<./env-settings>`:: - - DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_raw_telegram_to_serial - - -Plugin file ``dsmr_plugins/modules/forward_raw_telegram_to_serial.py`` (new file):: - - import serial - - from django.dispatch import receiver - - from dsmr_datalogger.signals import raw_telegram - import dsmr_datalogger.services.datalogger - - - @receiver(raw_telegram) - def handle_forward_raw_telegram_to_serial(**kwargs): - DEST_PORT = '/dev/ttyUSBvA' - connection_parameters = dsmr_datalogger.services.datalogger.get_dsmr_connection_parameters() - - serial_handle = serial.Serial() - serial_handle.port = DEST_PORT - serial_handle.baudrate = connection_parameters['baudrate'] - serial_handle.bytesize = connection_parameters['bytesize'] - serial_handle.parity = connection_parameters['parity'] - serial_handle.stopbits = serial.STOPBITS_ONE - serial_handle.xonxoff = 1 - serial_handle.rtscts = 0 - serial_handle.timeout = 1 - serial_handle.write_timeout = 0.2 - - try: - serial_handle.open() - bytes_sent = serial_handle.write(bytes(kwargs['data'], 'utf-8')) - except Exception as error: - print(error) - else: - print(' >>> Sent {} bytes to {}'.format(bytes_sent, DEST_PORT)) - - serial_handle.close() - - -.. attention:: - - Note that the ``/dev/ttyUSBvA`` variable should be changed to the serial port used in your own situation. - - -Example #3: Forwarding raw telegram data to another instance by API -------------------------------------------------------------------- -This can be quite handy if you run multiple instances of DSMR-reader (i.e.: RaspberryPI + somewhere in cloud). - -.. seealso:: - - :doc:`DSMRREADER_PLUGINS configuration<./env-settings>`:: - - DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_raw_telegram_to_api - - -Plugin file ``dsmr_plugins/modules/forward_raw_telegram_to_api.py`` (new file):: - - import requests - import logging - - from django.dispatch import receiver - - from dsmr_datalogger.signals import raw_telegram - - - @receiver(raw_telegram) - def handle_forward_raw_telegram_to_api(**kwargs): - API_HOST = 'https://YOUR-DSMR-HOST' # Note: Check whether you use HTTP or SSL (HTTPS). - API_KEY = 'YOUR-API-KEY' - TIMEOUT = 5 # A low timeout prevents the application from hanging, when the server is unavailable. - - try: - # Register telegram by simply sending it to the application with a POST request. - response = requests.post( - '{}/api/v1/datalogger/dsmrreading'.format(API_HOST), - headers={'X-AUTHKEY': API_KEY}, - data={'telegram': kwargs['data']}, - timeout=TIMEOUT - ) - except Exception as error: - return logging.error(error) - - if response.status_code != 201: - logging.error('Server Error forwarding telegram: {}'.format(response.text)) - - -.. attention:: - - Note that the ``API_HOST``, ``API_KEY`` and ``TIMEOUT`` variables should be changed to your own preferences. - - -Example #4: Forwarding DSMR readings in JSON format to some API ---------------------------------------------------------------- -Use this to send DSMR readings in JSON format to some (arbitrary) API. - -.. seealso:: - - :doc:`DSMRREADER_PLUGINS configuration<./env-settings>`:: - - DSMRREADER_PLUGINS=dsmr_plugins.modules.forward_json_dsmrreading_to_api - - -Plugin file ``dsmr_plugins/modules/forward_json_dsmrreading_to_api.py`` (new file):: - - import requests - import json - - from django.dispatch import receiver - from django.core import serializers - from django.utils import timezone - import django.db.models.signals - - from dsmr_datalogger.models.reading import DsmrReading - - @receiver(django.db.models.signals.post_save, sender=DsmrReading) - def handle_forward_json_dsmrreading_to_api(sender, instance, created, raw, **kwargs): - if not created or raw: - return - - instance.timestamp = timezone.localtime(instance.timestamp) - - if instance.extra_device_timestamp: - instance.extra_device_timestamp = timezone.localtime(instance.extra_device_timestamp) - - serialized = json.loads(serializers.serialize('json', [instance])) - json_string = json.dumps(serialized[0]['fields']) - - try: - requests.post( - 'https://YOUR-DSMR-HOST/api/endpoint/', - data=json_string, - # A low timeout prevents DSMR-reader from hanging, when the remote server is unreachable. - timeout=5 - ) - except Exception as error: - print('forward_json_dsmrreading_to_api:', error) - - -Example #5: Read telegrams using DSMRloggerWS API -------------------------------------------------- - -.. seealso:: - - :doc:`DSMRREADER_PLUGINS configuration<./env-settings>`:: - - DSMRREADER_PLUGINS=dsmr_plugins.modules.poll_dsmrloggerws_api - - -Plugin file ``dsmr_plugins/modules/poll_dsmrloggerws_api.py`` (new file):: - - import requests - - from django.dispatch import receiver - - from dsmr_backend.signals import backend_called - import dsmr_datalogger.services.datalogger - - - # Preverve a low timeout to prevent the entire backend process from hanging too long. - DSMRLOGGERWS_ENDPOINT = 'http://localhost/api/v1/sm/telegram' - DSMRLOGGERWS_TIMEOUT = 5 - - - @receiver(backend_called) - def handle_backend_called(**kwargs): - response = requests.get(DSMRLOGGERWS_ENDPOINT, - timeout=DSMRLOGGERWS_TIMEOUT) - - if response.status_code != 200: - print(' [!] DSMRloggerWS plugin: Telegram endpoint failed (HTTP {}): {}'.format( - response.status_code, - response.text - )) - return - - dsmr_datalogger.services.datalogger.telegram_to_reading(data=response.text) - -.. attention:: - - Note that you might need to update the ``http://localhost`` value to your own situation.