Support for InfluxDB 2.x

[nadolny]

Kotori is a great tool. Is an update that supports InfluxDB 2.x in your future plans?

[amotl]

Dear Aurélio,

thank you for writing in and for the kind words ¹. Well, it would be sweet if Kotori would have support for InfluxDB 2.x, and maybe also other databases as storage backends.

While we might come back to this feature request in a few months or so, we would like to outline the places ²³⁴, where some adjustments would need to be applied, in case anyone would like to start working on implementing this feature.

We can imagine implementing and shipping the feature in two subsequent steps. We think implementing the first step would be pretty valuable already, at least for some users.

1. Receive/ingest data payloads and write them to InfluxDB 2.x buckets.
2. Add support for the graphing subsystem on behalf of the convenient "instant dashboards" in Grafana.

What do you think about it? Would you need both parts to be implemented, or would step one be favorable to you already?

With kind regards,
Andreas.

Footnotes

1. May we humbly ask you if you could share some words in which scenario Kotori is being used on your end? It is always nice to see what others are doing, and if it is not a secret thing, it could also find its way into the use case gallery. ↩

2. https://github.com/daq-tools/kotori/blob/0.26.12/kotori/daq/services/mig.py ↩

3. https://github.com/daq-tools/kotori/blob/0.26.12/kotori/daq/storage/influx.py ↩

4. https://github.com/daq-tools/kotori/blob/0.26.12/kotori/daq/graphing/grafana/manager.py ↩

[nadolny]

Dear Andreas,

Thank you for your quick response.
At the moment evaluating Kotori to be used in an IoT solution for use in Industry.
In the scenario of this evaluation, only the first step would be excellent

1.Receive/ingest data payloads and write them to InfluxDB 2.x buckets.

Let me know if in addition to the points you mentioned, there would also be an update on the project

https://github.com/daq-tools/pyinfluxql => Flux

With kind regards,
Aurelius

[amotl]

Hi again,

In addition to the points you mentioned, [there would also be the export feature, which would need to be prepared for InfluxDB 2.x, no?].

You are absolutely right. ¹ and ² are two other spots which would need to be adjusted. Thank you for mentioning that!

Would we need to make the PyInfluxQL package understand Flux?

I am not sure about it. Maybe the semantics don't fit well together, as InfluxDB 2.x has different addressing (database vs. bucket) and different query language structure (InfluxQL vs. Flux). Maybe it would become a whole mess when trying to squeeze all into the same package. We would have to evaluate this detail when spending some more thoughts on this topic.

With kind regards,
Andreas.

Footnotes

1. https://github.com/daq-tools/kotori/blob/0.26.12/kotori/io/protocol/influx.py ↩

2. https://github.com/daq-tools/kotori/blob/0.26.12/kotori/io/export/influx.py ↩

[nadolny]

Hi again,

We could update the "influxdb-python client library" module by the "influxdb-client-python" module

In the link https://github.com/influxdata/influxdb-client-python We check a note:

Note: Use this client library (influxdb-client-python) with InfluxDB 2.x and InfluxDB 1.8+.

The fact that kotori is using influxdb 1.8 wouldn't have some facilities?

With kind regards,
Aurelius

[amotl]

Hi Aurelius,

thank you for sharing this detail. Yes, using that client library was planned for supporting InfluxDB 2.x. The section at [1] has some more details about compatibility with InfluxDB 1.8, where it is stated that it will also use the new API based on the "buckets" addressing scheme.

Currently, Kotori supports InfluxDB 1.6 - 1.8, through the previous "database/collection" addressing scheme. I am not objecting against dropping support for the InfluxDB 1.x series, but I would favor to have them both supported in parallel. Actually, we are still running InfluxDB 1.7 on the majority of our production servers.

With kind regards,
Andreas.

[1] https://github.com/influxdata/influxdb-client-python#influxdb-1-8-api-compatibility

[nadolny]

Dear Andreas,

This project can serve as an inspiration (Coded in JS): https://github.com/ioBroker/ioBroker.influxdb

This adapter saves state history into InfluxDB.
The adapter supports InfluxDB 1.x and 2.x

The following observation caught my attention:

If you have an InfluxDB 1.x installation (preferably 1.8.x or 1.9.x) then you choose " 1.x" in the adapter configuration.

With kind regards,
Aurelius

[amotl]

Hi again,

I just want to share some additional thoughts on this topic. I believe, when starting to work on this feature/improvement, we should aim for converging all that InfluxDB-specific code scattered throughout the code base into a subsystem with a more well-defined interface.

Based on that interface, data source adapters for other databases could be implemented more easily. For example, I would like to add support for CrateDB ¹ or other PostgreSQL-compatible databases in the future as well.

With kind regards,
Andreas.

Footnotes

1. https://github.com/crate/crate ↩