Define MQTT and sensor configuration separately from implementation

.gitignore

@@ -1,6 +1,6 @@
 .idea
 .venv*
-.coverage
+.coverage*
 coverage.xml
 *.egg-info
 *.pyc

CHANGES.md

@@ -4,6 +4,11 @@
 - Tests: Make sensor tests work, using a fake sysfs filesystem
 - Tests: Add basic test case for CLI interface
 - Remove support for Python 3.7
+- Define MQTT and sensor configuration separately from implementation.
+  The data logger uses a YAML file now, for example like `etc/mois.yaml`.
+- Added subcommand `make-config`, for creating a configuration blueprint
+- Added subcommand `make-dashboard`, for creating a Grafana Dashboard
+- Added subcommand `read`, for acquiring and displaying a reading on STDOUT
 
 ## v0.0.2 - 2024-04-15
 - Publish as `ds18b20-datalogger` package

MANIFEST.in

@@ -1,4 +1,4 @@
 include *.md
 recursive-include docs *.md
-recursive-include ds18b20_datalogger *.py *.json
+recursive-include ds18b20_datalogger *.py *.json *.yaml
 prune tests

README.md

@@ -3,7 +3,7 @@
 A data logger specializing in reading an array of DS18B20 sensors.
 
 A temperature sensor matrix with heatmap visualization for bee hive monitoring,
-using Raspberry Pi, Python, DS18B20, MQTT, Kotori DAQ, and Grafana. 
+using Raspberry Pi, Linux, Python, DS18B20, MQTT, Kotori DAQ, and Grafana.
 
 
 | View from outside | View from inside (sensor tip details) |
@@ -37,83 +37,113 @@ https://community.hiveeyes.org/t/laborprotokoll-4x5-temp-matrix-mit-ds18b20/5102
 ## What's Inside
 
 - The `ds18b20-datalogger` program, reading DS18B20 sensors and
-  publishing readings to MQTT.
-- JSON representation for a corresponding [Grafana Dashboard],
-  when acquired through [Kotori DAQ].
+  publishing readings to MQTT in JSON format.
+- Configuration file in YAML format, like [`datalogger.yaml`].
+- JSON representation for a corresponding Grafana Dashboard
+  [`grafana-dashboard.json`], when measurement data is submitted
+  and acquired through [Kotori DAQ].
 
 
-## Setup
-We recommend to install the program into a Python virtualenv.
+## Synopsis
+Acquire single reading, and echo it on STDOUT in JSON format.
+```shell
+ds18b20-datalogger read datalogger.yaml
+```
+
+Take a single reading, and publish it to the configured MQTT topic.
+```shell
+ds18b20-datalogger run datalogger.yaml
+```
+
+## Live
+In order to see a running system in action, please enjoy inspecting the
+[Live Grafana Dashboard].
+
+
+## Installation
+Install the `ds18b20-datalogger` package from PyPI using `pip`.
 ```shell
-python3 -m venv .venv
-source .venv/bin/activate
 pip install --upgrade ds18b20-datalogger
 ```
+See also alternative installation methods and hands-on walkthroughs at
+[development sandbox] and [production setup].
+
+## Configuration
+In order to operate the data logger successfully, you will need to configure
+two important details:
 
-In this spirit, you keep the installation separate from your system Python, so
-you can easily nuke it and start from scratch in case anything goes south.
+- Sensors: Map one-wire sensor sysfs paths to self-assigned sensor names.
+- Telemetry: Adjust MQTT connection settings and MQTT topic.
 
+You can create a blueprint configuration file by using the `make-config`
+subcommand.
+```shell
+ds18b20-datalogger make-config > datalogger.yaml
+```
 
-## Operations
+### Appliance: Sensor Wiring and Sensor Mapping
 
-### Sensor Wiring
 Be aware that you might have to adjust your resistors size.
 With 30 sensors i had erratic sensor mapping using a 4.7k resistor.
 I am getting valid mapping using a 2.2k resistor.
 
-### Sensor Mapping
-https://community.hiveeyes.org/t/ds18b20-temperatur-sensoren-am-one-wire-bus-anordnen/1399
+Please read more about [sensor mapping] on our community forum. In practice,
+just edit the `one-wire` section within the configuration file according to
+your setup.
+
+### Backend: Telemetry and Visualization
+
+The data logger will publish measurements to an MQTT topic, where [Kotori DAQ]
+can pick it up, in order to converge into a timeseries database, and displays
+it on a Grafana Dashboard.
+
+The package includes a corresponding Grafana Dashboard, which can be created
+by invoking the `make-dashboard` subcommand.
 
-### Data Publishing
 ```shell
-ssh youruser@yourpi
-screen
-source /path/to/ds18b20-datalogger/.venv/bin/activate
-ds18b20-datalogger
+ds18b20-datalogger make-dashboard > dashboard.json
 ```
 
-### MQTT data upload to Hiveeyes
-https://community.hiveeyes.org/t/daten-per-mqtt-und-python-ans-backend-auf-swarm-hiveeyes-org-ubertragen/94/6
-
-### Format your array
-https://community.hiveeyes.org/t/how-to-visualize-2-dimensional-temperature-data-in-grafana/974/9
-```python
-matrix = [[temp_ir_1_1, temp_ir_1_2, temp_ir_1_3, temp_ir_1_4, temp_ir_1_5, temp_ir_1_6],
-          [temp_ir_2_1, temp_ir_2_2, temp_ir_2_3, temp_ir_2_4, temp_ir_2_5, temp_ir_2_6],
-          [temp_ir_3_1, temp_ir_3_2, temp_ir_3_3, temp_ir_3_4, temp_ir_3_5, temp_ir_3_6],
-          [temp_ir_4_1, temp_ir_4_2, temp_ir_4_3, temp_ir_4_4, temp_ir_4_5, temp_ir_4_6],
-          [temp_ir_5_1, temp_ir_5_2, temp_ir_5_3, temp_ir_5_4, temp_ir_5_5, temp_ir_5_6]]
-```
+On our community forum, you can find relevant discussions about this topic.
 
-### Data visualization in Grafana
-https://swarm.hiveeyes.org/grafana/d/Y9PcgE4Sz/mois-ex-wtf-test-ir-sensor-svg-pixmap-copy
+- [MQTT data upload to Hiveeyes]
+- [Data visualization in Grafana]
+- [Sensor offsets] (optional)
 
-### Bonus: Sensor offsets
-https://community.hiveeyes.org/t/temperatursensoren-justieren-kalibrieren/1744/2
 
-### Cron Configuration
-```
-*/5 * * * * cd /path/to/data-directory && /path/to/ds18b20-datalogger/.venv/bin/ds18b20-datalogger
-```
+## Acknowledgements
+
+The original code this implementation has been derived from has been discovered
+on the element14 community forum at [Multiple DS18B20 Temp sensors interfacing
+with Raspberry Pi], shared by [@laluha]. Thanks!
 
 
 ## Contributing
 
 In order to learn how to start hacking on this program, please have a look at the
-documentation about how to install a [development sandbox](./docs/sandbox.md).
+documentation about how to install a [development sandbox].
 
 Contributions of any kind are always welcome and appreciated. Thank you.
 
 
 
-[Grafana Dashboard]: https://swarm.hiveeyes.org/grafana/d/T49wHSaIk/mois-ex-wtf-test-ds18b20-5x6-temp-matrix-svg-pixmap?orgId=2&from=1712771622514&to=1712807415379
+[@laluha]: https://github.com/laluha
+[`datalogger.yaml`]: https://github.com/hiveeyes/ds18b20-datalogger/raw/main/ds18b20_datalogger/datalogger.yaml
+[Data visualization in Grafana]: https://swarm.hiveeyes.org/grafana/d/Y9PcgE4Sz/mois-ex-wtf-test-ir-sensor-svg-pixmap-copy
+[`grafana-dashboard.json`]: https://github.com/hiveeyes/ds18b20-datalogger/raw/main/ds18b20_datalogger/grafana-dashboard.json
 [Kotori DAQ]: https://kotori.readthedocs.io
+[Live Grafana Dashboard]: https://swarm.hiveeyes.org/grafana/d/T49wHSaIk/mois-ex-wtf-test-ds18b20-5x6-temp-matrix-svg-pixmap?orgId=2&from=1712771622514&to=1712807415379
+[MQTT data upload to Hiveeyes]: https://community.hiveeyes.org/t/daten-per-mqtt-und-python-ans-backend-auf-swarm-hiveeyes-org-ubertragen/94/6
+[Multiple DS18B20 Temp sensors interfacing with Raspberry Pi]: https://community.element14.com/products/raspberry-pi/raspberrypi_projects/b/blog/posts/multiple-ds18b20-temp-sensors-interfacing-with-raspberry-pi?CommentId=9470e4e9-b054-4dd3-9a3f-ac9d1fe38087
+[Sensor offsets]: https://community.hiveeyes.org/t/temperatursensoren-justieren-kalibrieren/1744/2
+[sensor mapping]: https://community.hiveeyes.org/t/ds18b20-temperatur-sensoren-am-one-wire-bus-anordnen/1399
 
 [Changelog]: https://github.com/hiveeyes/ds18b20-datalogger/blob/main/CHANGES.md
-[development documentation]: https://ds18b20-datalogger.readthedocs.io/en/latest/sandbox.html
+[development sandbox]: https://github.com/hiveeyes/ds18b20-datalogger/blob/main/docs/sandbox.md
 [Documentation]: https://ds18b20-datalogger.readthedocs.io/
 [Issues]: https://github.com/hiveeyes/ds18b20-datalogger/issues
 [License]: https://github.com/hiveeyes/ds18b20-datalogger/blob/main/LICENSE
+[production setup]: https://github.com/hiveeyes/ds18b20-datalogger/blob/main/docs/production.md
 [PyPI]: https://pypi.org/project/ds18b20-datalogger/
 [Source code]: https://github.com/hiveeyes/ds18b20-datalogger
 

docs/backlog.md

@@ -1,8 +1,17 @@
 # Backlog for ds18b20-datalogger
 
 ## Iteration +1
+
+## Iteration +2
+- By default, probe configuration at `/etc/ds18b20-datalogger/config.yaml`
+- Add possibility to connect to MQTT/SSL
+
+## Done
 - Better software tests
 - Break out sensor mapping configuration from code
   to make it re-usable across different setups
-- Improve documentation
 - Publish to PyPI
+- Subcommand `make-dashboard`
+- Subcommand `read`
+- Test coverage 100%?
+- Improve documentation

docs/mois.md

@@ -0,0 +1,9 @@
+# Measurement reading and publishing for mois
+
+```shell
+ssh youruser@yourpi
+screen
+source /path/to/ds18b20-datalogger/.venv/bin/activate
+wget https://github.com/hiveeyes/ds18b20-datalogger/raw/main/etc/mois.yaml
+ds18b20-datalogger run mois.yaml
+```

docs/production.md

@@ -0,0 +1,75 @@
+# Production Setup
+
+A walkthrough how to install the program on a Raspberry Pi machine running
+Linux. In order to learn about more details, please also consult the [README]
+document.
+
+## Installation
+```shell
+sudo su -
+python3 -m venv /opt/ds18b20-datalogger
+/opt/ds18b20-datalogger/bin/pip install ds18b20-datalogger
+/opt/ds18b20-datalogger/bin/ds18b20-datalogger make-config > /etc/ds18b20-datalogger/config.yaml
+/opt/ds18b20-datalogger/bin/ds18b20-datalogger run /etc/ds18b20-datalogger/config.yaml
+```
+
+We recommend to install the program into a Python virtualenv.
+In this spirit, you keep the installation separate from your system Python, so
+you can easily nuke it and start from scratch in case anything goes south.
+
+## Periodic Measurements
+If you want to run the program periodically, for example each five minutes, use
+Cron or, alternatively, Systemd Timers.
+
+### Cron
+This should go into `/etc/cron.d/ds18b20-datalogger`.
+```
+*/5 * * * *  root  /opt/ds18b20-datalogger/bin/ds18b20-datalogger run /etc/ds18b20-datalogger/config.yaml
+```
+
+### Systemd Timers
+
+This should go into `/etc/systemd/system/ds18b20-datalogger.service`.
+```ini
+# Systemd service unit file for ds18b20-datalogger.
+# https://github.com/hiveeyes/ds18b20-datalogger
+
+[Unit]
+Description=DS18B20 data logger
+Wants=ds18b20-datalogger.timer
+
+[Service]
+Type=oneshot
+ExecStart=/opt/ds18b20-datalogger/bin/ds18b20-datalogger run /etc/ds18b20-datalogger/config.yaml
+
+[Install]
+WantedBy=multi-user.target
+```
+
+This should go into `/etc/systemd/system/ds18b20-datalogger.timer`.
+```ini
+# Systemd timer unit file for ds18b20-datalogger.
+# https://github.com/hiveeyes/ds18b20-datalogger
+
+[Unit]
+Description=DS18B20 data logger
+Requires=ds18b20-datalogger.service
+
+[Timer]
+Unit=ds18b20-datalogger.service
+OnCalendar=*-*-* *:0/5
+
+[Install]
+WantedBy=timers.target
+```
+
+Start and enable the service and the timer units.
+```shell
+systemctl start ds18b20-datalogger.service
+systemctl enable ds18b20-datalogger.timer
+systemctl list-timers
+```
+
+
+[README]: https://github.com/hiveeyes/ds18b20-datalogger/blob/main/README.md
+[Systemd Timers]: https://opensource.com/article/20/7/systemd-timers

docs/sandbox.md

@@ -1,13 +1,21 @@
 # Development Sandbox
 
 ## Install
+Acquire the source code from the repository, install a Python virtualenv,
+and install the package in development mode.
 ```shell
 git clone https://github.com/hiveeyes/ds18b20-datalogger
+cd ds18b20-datalogger
 python3 -m venv .venv
 source .venv/bin/activate
-pip install --editable='.[develop,docs,release,test]'
+pip install --upgrade --editable='.[develop,docs,release,test]'
 ```
 
+We recommend to install the program into a Python virtualenv.
+In this spirit, you keep the installation separate from your system Python, so
+you can easily nuke it and start from scratch in case anything goes south.
+
+
 ## Software tests
 Run all linters, and invoke the test suite.
 ```shell

ds18b20_datalogger/cli.py

@@ -1,7 +1,44 @@
+import json
+import sys
+from pathlib import Path
+
 from ds18b20_datalogger.core import read_ds18b20_sensor_matrix, send_measurement_mqtt
+from ds18b20_datalogger.model import Settings
+
+if sys.version_info < (3, 9):
+    from importlib_resources import files  # pragma: nocover
+else:
+    from importlib.resources import files
 
 
 def main():
-    reading = read_ds18b20_sensor_matrix()
-    send_measurement_mqtt(reading)
-    # print(strftime("%Y-%m-%d %H:%M:%S", time.localtime())," Done sending. Going to sleep for 15min.")   # noqa: ERA001
+
+    # Sanity checks.
+    if not sys.argv[1:]:
+        raise ValueError("Program needs a subcommand")
+    subcommand = sys.argv[1]
+
+    # Evaluate subcommand.
+    if subcommand in ["run", "read"]:
+        if not sys.argv[2:]:
+            raise ValueError("Program needs a configuration file")
+        configfile = Path(sys.argv[2])
+        if not configfile.exists():
+            raise ValueError(f"Configuration file does not exist: {configfile}")
+        settings = Settings.from_file(configfile)
+        reading = read_ds18b20_sensor_matrix(settings.devicemap)
+        if subcommand == "read":
+            print(json.dumps(reading.to_dict(), indent=2))  # noqa: T201
+        elif subcommand == "run":
+            send_measurement_mqtt(settings.mqtt, reading)
+
+    elif subcommand == "make-config":
+        config_template = files("ds18b20_datalogger") / "datalogger.yaml"
+        print(config_template.read_text(), file=sys.stdout)  # noqa: T201
+
+    elif subcommand == "make-dashboard":
+        dashboard = files("ds18b20_datalogger") / "grafana-dashboard.json"
+        print(dashboard.read_text(), file=sys.stdout)  # noqa: T201
+
+    else:
+        raise ValueError(f"Subcommand unknown: {subcommand}")