[CrateDB] Add documentation across the board

CHANGES.rst

@@ -25,6 +25,7 @@ in progress
  a single TTN Application. Thanks, @thiasB, @einsiedlerkrebs, and @ClemensGruber.
 - [core] Fix error when connecting to MQTT broker without authentication credentials
 - [docs] Refactor "decoders" section to "integrations", and improve index/overview page
+- [CrateDB] Add support for data acquisition and data export
 
 
 .. _kotori-0.27.0:

README.rst

@@ -76,6 +76,10 @@ Kotori
  :target: https://github.com/grafana/grafana
  :alt: Supported Grafana versions
 
+ .. image:: https://img.shields.io/badge/CrateDB-4.2%20--%205.x-blue.svg
+ :target: https://github.com/crate/crate
+ :alt: Supported CrateDB versions
+
  .. image:: https://img.shields.io/badge/InfluxDB-1.6%2C%201.7%2C%201.8-blue.svg
  :target: https://github.com/influxdata/influxdb
  :alt: Supported InfluxDB versions
@@ -105,7 +109,7 @@ Details
 
 Kotori takes the role of the `data historian`_ component within a `SCADA`_ / `MDE`_
 system, exclusively built upon industry-grade `free and open-source software`_
-like Grafana_, Mosquitto_, or InfluxDB_. It is written in Python_,
+like Grafana_, Mosquitto_, CrateDB_, or InfluxDB_. It is written in Python_,
 and uses the Twisted_ networking library.
 
 The best way to find out what you can do with Kotori, is by looking at
@@ -222,6 +226,7 @@ The project is licensed under the terms of the GNU AGPL license, see `LICENSE`_.
 
 .. _Autobahn: https://crossbar.io/autobahn/
 .. _contributors: https://getkotori.org/docs/project/contributors.html
+.. _CrateDB: https://github.com/crate/crate
 .. _Create an issue: https://github.com/daq-tools/kotori/issues/new
 .. _data historian: https://en.wikipedia.org/wiki/Operational_historian
 .. _development sandbox: https://getkotori.org/docs/setup/sandbox.html

doc/source/_meta.rst

@@ -16,5 +16,5 @@
  :keywords lang=es: adquisición de datos, gráficas, conjunto de herramientas, telemetría
  :keywords lang=fr: acquisition de données, graphiques, boîte à outils, télémétrie
  :keywords: m2m, iot, mqtt, wamp, http, rest, udp, binary struct,
- Kotori, Grafana, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP
+ Kotori, Grafana, CrateDB, InfluxDB, Mosquitto, MongoDB, XMPP, SMTP
 

doc/source/_resources.rst

@@ -170,6 +170,7 @@
 .. _Raumfahrtagentur: http://www.raumfahrtagentur.org/
 
 .. NEW
+.. _CrateDB: https://github.com/crate/crate
 .. _curl: https://en.wikipedia.org/wiki/CURL
 .. _Flux data scripting language: https://docs.influxdata.com/flux/
 .. _Funky v3: https://harizanov.com/product/funky-v3/
@@ -183,6 +184,7 @@
 .. _OpenXC: https://openxcplatform.com/
 .. _OpenXC for Python: http://python.openxcplatform.com/
 .. _multi-tenant: https://en.wikipedia.org/wiki/Multitenancy
+.. _near real-time: https://en.wikipedia.org/wiki/Near_real-time#Near_real-time
 .. _NodeUSB: https://web.archive.org/web/20210621192219/http://www.nodeusb.com/
 .. _trunking: https://en.wikipedia.org/wiki/Trunking
 .. _webhook: https://en.wikipedia.org/wiki/Webhook

doc/source/about/scenarios.rst

@@ -85,8 +85,8 @@ We are standing on the shoulders of giants:
 - Leverage the open infrastructure based on Twisted_ - an event-driven networking engine -
  to implement custom software components.
 - Listen and talk M2M_ using the *MQ Telemetry Transport* connectivity protocol and software bus (MQTT_).
-- Store data points into InfluxDB_, a leading open source time series database suitable
- for realtime analytics and sensor data storage.
+- Store data points into CrateDB_, InfluxDB_, or other open source time series databases
+ suitable for realtime analytics and sensor data storage.
 - Automate dashboard management in the context of data arriving on different data channels
  using Grafana_, an open source, feature rich metrics dashboard and graph editor.
 - Make Browser clients first-class citizens of the underpinning software bus framework

doc/source/about/technologies.rst

@@ -26,14 +26,16 @@ Infrastructure components
 
 - Kotori_, a data acquisition, graphing and telemetry toolkit
 - Grafana_, a graph and dashboard builder for visualizing time series metrics
-- InfluxDB_, a time-series database
+- CrateDB_, a time-series database ¹
+- InfluxDB_, a time-series database ¹
 - Mosquitto_, an MQTT message broker
-- MongoDB_, a document store (optional) ¹²
+- MongoDB_, a document store (optional) ²³
 
-| ¹ MongoDB is only required when doing CSV data acquisition, so it is completely
+| ¹ Kotori can either use CrateDB or InfluxDB as timeseries database.
+| ² MongoDB is only required when doing CSV data acquisition, so it is completely
 | optional for regular operations of Kotori.
-| ² As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
-| it will probably be phased out gradually and replaced by PostgreSQL_.
+| ³ As MongoDB - strictly speaking - stopped being free software recently (2018/2019),
+| it will probably be phased out gradually and replaced by PostgreSQL_ or CrateDB_.
 
 
 Supported protocols

doc/source/database/cratedb.rst

@@ -0,0 +1,18 @@
+.. include:: ../_resources.rst
+
+.. _database-cratedb:
+
+#######
+CrateDB
+#######
+
+.. admonition::
+
+ .. todo:: Content on this page may need an update.
+
+*****
+About
+*****
+
+`CrateDB`_ is a distributed and scalable SQL database for storing and analyzing massive
+amounts of data in `near real-time`_. It is PostgreSQL-compatible, and based on Lucene.

doc/source/database/index.md

@@ -14,6 +14,7 @@ with Kotori. Adding more adapters is possible.
 :maxdepth: 1
 :hidden:
 
+cratedb
 influxdb
 mongodb
 ```
@@ -24,6 +25,34 @@ mongodb
 :padding: 0
 
 
+:::::{grid-item-card}
+::::{grid} 2
+:margin: 0
+:padding: 0
+
+:::{grid-item}
+:columns: 8
+#### [](#database-cratedb)
+
+CrateDB is a distributed and scalable SQL database for storing and analyzing massive
+amounts of data in near real-time. It is PostgreSQL-compatible, and based on Lucene.
+
+<small>
+<strong>Categories:</strong> timeseries-database, multi-modal database
+</small>
+:::
+:::{grid-item}
+:columns: 4
+{bdg-primary-line}`eth` {bdg-primary-line}`wifi` {bdg-primary-line}`http` {bdg-primary-line}`postgresql`
+
+{bdg-success-line}`sql` {bdg-success-line}`blob`
+
+{bdg-secondary-line}`amd64` {bdg-secondary-line}`arm64`
+:::
+::::
+:::::
+
+
 :::::{grid-item-card}
 ::::{grid} 2
 :margin: 0

doc/source/development/releasing/foundation.rst

@@ -34,6 +34,24 @@ Download packages
  scp mosquitto*.deb [email protected]:/srv/packages/organizations/elmyra/foss/aptly/public/incoming
 
 
+*******
+CrateDB
+*******
+- https://github.com/crate/crate
+- https://crate.io/docs/crate/tutorials/
+
+Download packages
+=================
+::
+
+ # Download amd64 packages
+ wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~buster_amd64.deb
+ wget https://cdn.crate.io/downloads/apt/stable/pool/main/c/crate/crate_5.3.2-1~bullseye_amd64.deb
+
+ # Upload to "incoming" directory
+ scp crate_*.deb [email protected]:/srv/packages/organizations/elmyra/foss/aptly/public/incoming
+
+
 ********
 InfluxDB
 ********
@@ -100,7 +118,7 @@ Publish packages
 
  # Add packages to repository
  aptly repo add -config=$APTLY_CONFIG -remove-files=true $APTLY_REPOSITORY \
- $PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb
+ $PACKAGES_INCOMING/crate*.deb $PACKAGES_INCOMING/influxdb*.deb $PACKAGES_INCOMING/grafana*.deb $PACKAGES_INCOMING/*mosquitto*.deb
 
  # Publish repository
  aptly publish update -config=$APTLY_CONFIG -gpg-key=2543A838 -passphrase=esp $APTLY_DISTRIBUTION

doc/source/development/releasing/index.rst

@@ -4,7 +4,7 @@ Releasing and packaging
 
 This section describes how to cut a new release, how to build
 and publish Kotori packages and how to add foundation packages
-like InfluxDB and Grafana to the same package repositories.
+like CrateDB, InfluxDB, and Grafana to the same package repositories.
 
 .. toctree::
  :maxdepth: 1

doc/source/development/systemd.rst

@@ -169,12 +169,12 @@ systemd troubleshooting
 
 ::
 
- systemctl status influxdb mosquitto grafana-server kotori
+ systemctl status crate influxdb mosquitto grafana-server kotori
 
 
 ::
 
- systemctl list-unit-files kotori* influx* grafana* mosquitto*
+ systemctl list-unit-files kotori* crate* influx* grafana* mosquitto*
 
 
 ::

doc/source/development/tests.rst

@@ -11,7 +11,7 @@ The tests are mostly full integration tests. They are testing the whole system
 and the interactions between the subsystems.
 
 Messages will get published to the MQTT bus by shelling out to ``mosquitto_pub``.
-After that, InfluxDB will be checked to contain the right data and Grafana will
+After that, the database will be checked to contain the right data and Grafana will
 be checked to be accurately provisioned.
 
 While the shell-out can well be optimized for efficiency, it is also pleasant
@@ -32,12 +32,12 @@ Install some needed packages::
 Foundation services
 ===================
 
-The test suite will assume running instances of Mosquitto, InfluxDB, MongoDB
+The test suite will assume running instances of Mosquitto, CrateDB, InfluxDB, MongoDB
 and Grafana and fire up an in-process instance of Kotori to complement these. Please
 have a look at :ref:`setup-docker` in order to get the complementing services
 up and running in a quick and ad hoc manner.
 
-Run Mosquitto, InfluxDB, MongoDB and Grafana as Docker containers::
+Run Mosquitto, CrateDB, InfluxDB, MongoDB and Grafana as Docker containers::
 
  make start-foundation-services
 

doc/source/handbook/acquisition/error-signalling.rst

@@ -48,13 +48,24 @@ the system will respond with a MQTT "response" on the corresponding topic with s
  "description": "Error processing MQTT message \"{\"value\": 42.42\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"."
  }
 
-Database error
-==============
-When sending a payload with an existing field already seeded in a different data type::
+Database errors
+===============
+When sending a payload with an existing field already materialized using a different data type::
 
  echo '{"value": "invalid"}' | mosquitto_pub -h kotori.example.org -t mqttkit-1/testdrive/area-42/node-1/data.json -l
 
-the system will respond with::
+the system will respond correspondingly.
+
+CrateDB::
+
+ mqttkit-1/testdrive/area-42/node-1/error.json {
+ "type": "<class 'crate.client.exceptions.ProgrammingError'>",
+ "message": "SQLParseException[Cannot cast object element `humidity` with value `invalid` to type `double precision`]",
+ "description": "Error processing MQTT message \"b'{\"temperature\": 42.84, \"humidity\": \"invalid\"}'\" from topic \"mqttkit-2/foo/bar/1/data.json\".",
+ "timestamp": "2023-06-20T12:00:08+00:00"
+ }
+
+InfluxDB::
 
  mqttkit-1/testdrive/area-42/node-1/error.json {
  "message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n",
@@ -73,11 +84,4 @@ and then sending an invalid payload like::
 
  echo '2017-05-01 22:39:09,invalid' | http POST http://kotori.example.org/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv
 
-The system will also respond over MQTT with::
-
- mqttkit-1/testdrive/area-42/node-1/error.json {
- "message": "400: {\"error\":\"field type conflict: input field \\\"value\\\" on measurement \\\"area_42_node_1_sensors\\\" is type string, already exists as type float dropped=1\"}\n",
- "type": "<class 'influxdb.exceptions.InfluxDBClientError'>",
- "description": "Error processing MQTT message \"{\"time\": \"2017-05-01 22:40:09\", \"value\": \"invalid\"}\" from topic \"mqttkit-1/testdrive/area-42/node-1/data.json\"."
- }
-
+The system will also respond over MQTT correspondingly, like outlined above.

doc/source/handbook/index.rst

@@ -49,4 +49,5 @@ Usage
 
  usage/kotori
  usage/grafana
+ usage/cratedb
  usage/influxdb

doc/source/handbook/usage/cratedb.rst

@@ -0,0 +1,14 @@
+.. include:: ../../_resources.rst
+
+.. _cratedb-handbook:
+
+################
+CrateDB handbook
+################
+
+.. admonition::
+
+ .. todo:: Content on this page may need an update.
+
+This section of the documentation will give a short and concise summary about
+how to operate and work with CrateDB.

doc/source/integration/tts-ttn.rst

@@ -45,6 +45,7 @@ Overview
 
  subgraph backend
  direction LR
+ Kotori --> CrateDB
  Kotori --> InfluxDB
  Kotori --> Grafana
  end

doc/source/setup/docker.rst

@@ -26,7 +26,7 @@ its configuration at :ref:`getting-started`.
 Introduction
 ************
 
-This section outlines how to conveniently run Mosquitto, InfluxDB,
+This section outlines how to conveniently run Mosquitto, CrateDB, InfluxDB,
 MongoDB, Grafana and Kotori using Docker.
 
 The repository provides two files ``docker-compose.yml`` and ``.env``. They
@@ -48,15 +48,15 @@ Those images are published to Docker Hub.
 
  Please note that this Docker Compose configuration is primarily suited for
  evaluation and development purposes. As it either disables authentication
- or uses insecure authentication credentials for Mosquitto, InfluxDB,
+ or uses insecure authentication credentials for Mosquitto, CrateDB, InfluxDB,
  and Grafana, it is not prepared for production setups.
 
 
 *************
 Prerequisites
 *************
 
-This will give you Mosquitto, InfluxDB, MongoDB, Grafana, an improved
+This will give you Mosquitto, CrateDB, InfluxDB, MongoDB, Grafana, an improved
 Grafana map panel plugin, and a command alias for invoking Kotori.
 
 In order to invoke the auxiliary services, run::
@@ -94,6 +94,34 @@ Testdrive
 This is a basic test walkthrough, to check if data is correctly routed from the
 telemetry message bus to the database.
 
+CrateDB
+=======
+
+This example uses CrateDB as timeseries-database.
+
+Invoke Kotori::
+
+ kotori --config /etc/kotori/docker/docker-cratedb.ini
+
+Publish single reading using MQTT::
+
+ export CHANNEL_TOPIC=sensorwan-cratedb/foo/bar/1/data.json
+ docker run \
+ --network kotori_default \
+ -it --rm eclipse-mosquitto \
+ mosquitto_pub -d -h mosquitto -t $CHANNEL_TOPIC -m '{"temperature": 42.84, "humidity": 83.1}'
+
+Check if reading has been stored in CrateDB::
+
+ docker run \
+ --network kotori_default \
+ -it --rm crate \
+ crash --hosts cratedb -c 'SELECT * FROM sensorwan_cratedb_foo.bar_1_sensors;'
+
+Go to Grafana and visit the dashboard just created::
+
+ open "http://localhost:3000/?orgId=1&search=open&query=sensorwan-cratedb"
+
 InfluxDB
 ========
 

doc/source/setup/index.rst

@@ -16,7 +16,7 @@ from Docker Hub, from the Python Package Index (PyPI) or from the Git
 repository.
 
 For running Kotori in a full configuration, you will need some other
-infrastructure services like Mosquitto_, InfluxDB_, Grafana_ and optionally
+infrastructure services like Mosquitto_, CrateDB_, InfluxDB_, Grafana_ and optionally
 MongoDB_.
 
 Have fun and enjoy your data acquisition!
@@ -62,7 +62,7 @@ Details
 
 
 ¹ When choosing to install from the Debian package repository, you will also be
-able to receive appropriate Debian packages for Mosquitto, InfluxDB and Grafana
+able to receive appropriate Debian packages for Mosquitto, CrateDB, InfluxDB, and Grafana
 through the `DaqZilla package repository <https://packages.elmyra.de/elmyra/foss/debian/>`_.
 This makes it easy to setup the complete DAQ system from a single package source.
 

doc/source/setup/linux-arch.rst

@@ -59,6 +59,7 @@ MongoDB is provided through the Arch User Repository (AUR)::
  cd mongodb-tools-bin
  makepkg -si
 
+.. todo:: Add walkthrough how to install CrateDB on Arch Linux.
 
 Kotori
 ======