btmon MQTT

[ghost]

I am using btmon 3.2.0 with IP in and MQTT out and it is working, but I have a few issues/comments:

I have OBFUSCATE_SERIALS = 1, but the serial is not Obfuscated. I believe it works for OEM output. I can Obfuscate the serial with MQTT mapping, but that's just a workaround.

It would be useful if there was an option for all ECM-1240 data in one payload.

I set INCLUDE_CURRENT = 1, assuming the current would be output, but I saw in the comments it is only for GEM?

I noticed a timestamp is not included in MQTT out, even if upload period is set to sampling time. Can it be added?

Thanks.

[mrguessed]

I made a prototype of some of the changes, it's in the btmon.py in my fork:

• https://github.com/mrguessed/mtools/tree/mqtt_flexformat

The code defaults to using the existing single-value payload format, and can be reconfigured for both plain-text and json style as needed. See the examples inside the code for more details.

I have OBFUSCATE_SERIALS = 1, but the serial is not Obfuscated. I believe it works for OEM output. I can Obfuscate the serial with MQTT mapping, but that's just a workaround.

This code will now follow the OBFUSCATE_SERIALS setting.

It would be useful if there was an option for all ECM-1240 data in one payload.

Please file a separate ticket to track. Not sure when I'll get to that one, as it'll require a bunch of changes to be made for row-oriented format.

I set INCLUDE_CURRENT = 1, assuming the current would be output, but I saw in the comments it is only for GEM?

If your device supports it, then it can be used. Looks like only the GEM devices support it in their format. You'd need to check with Brultech to see if there's an equivalent for ECM-1240 models.

That said, I've added support to MQTT output so it'll send this over if the data is there & it's been enabled.

I noticed a timestamp is not included in MQTT out, even if upload period is set to sampling time. Can it be added?

In MQTT, it's never clear where this stuff is expected to go. There are certain conventions like Graphite's plaintext format, where they'll encode it. These rely upon you being able to peel apart the values at the other end.

I've prototyped the addition of a mqtt-value-format & mqtt-value-escape parameters. Together these can be used to either plaintext (mqtt-value-escape=raw) or json (mqtt-value-escape=json) the payload using a variety of formats. The formats themselves (mqtt-value-format) use a Python str.format value, with a bunch of pre-defined tokens.

See the doc in bymon.py for common examples.

Once it's gone through some functional review, I'll submit a PR to @matthewwall for inclusion. Over time, I'd like to add per-measurement level metadata, and I plan to use this mechanism to do it (for JSON-encoded values)

eg. room=kitchen

[ghost]

Thanks for your updates. I am running on your update now and will continue testing over the coming days. Obfuscate is working now. Current shows if I use the -p option and it is included in the data transmitted by the ECM-1240, but it isn't showing up in the MQTT data.

[mrguessed]

With the current feature enabled in btmon.py, I can see current values from my GEM in the MQTT stream (using mosquitto_sub):

house/current/microwave 0.100
house/current/kitchen_island 0.120
house/current/wine_fridge 1.120
house/current/main_fridge 1.080
house/current/kitchen_outlets 0.120

If the ECM-1240 has the values, I'm not sure why they wouldn't appear in the stream.

[ghost]

I can see the current variables are allocated in the DB_SCHEMA's, but unfortunately I don't understand Python. I assume Matthew will know. Regardless, MQTT out is working great and even without current it is a great addition to btmon.

[mrguessed]

@tgmaxx,
When you're seeing values on the ECM-1240, do you only see them as ch1_a and ch2_a, and aux_1..n are missing?

In either case, you you mind including a copy of your -p output?

My guess is that it's due to a missing FILTER_CURRENT section from btmon.py, inside ECM1240BinaryPacket.channels().

I think it should look like the following (similar to class GEM48PBinaryPacket.channels()):

    def channels(self, fltr):
        ...
        if fltr == FILTER_PE_LABELS:
            c = ['ch1', 'ch2', 'aux1', 'aux2', 'aux3', 'aux4', 'aux5']
        elif fltr == FILTER_CURRENT:
            c = ['ch1_a', 'ch2_a']
        elif fltr == FILTER_POWER:
            c = ['ch1_w', 'ch2_w', 'aux1_w', 'aux2_w', 'aux3_w', 'aux4_w', 'aux5_w']
        ...

My code relies upon the filters to extract what I deliver to MQTT.

[ghost]

@mrguessed,
I can see that btmon does not have FILTER_CURRENT for the ECM1220BinaryPacket and the ECM1240BinaryPacket. FYI, the ECM1240 is a ECM1220 with the addition of Aux channels (as far as I understand it).

Anyway, here is a screen capture of your updated btmon startup and one packet using the -p option:

[mrguessed]

Thx. I pushed a new version of my prototype that includes the support for current on both ECM-1220 and ECM-1240 units. Please try it out and let me know.

For now I bundled it with my prototype, but I may break it out as a separate PR so it can be reviewed/included without the rest of my MQTT changes (as my changes may not be correct)

Ref: Current support added in a4e4b12

[ghost]

Thanks Mark. I will check it out tomorrow and let you know how it goes.

[ghost]

I am running your new version and getting current now. Thanks!

[mrguessed]

Good stuff. I'll file a separate PR in a few days to cover fixing that part.

[mrguessed]

@tgmaxx,
I'd like to turn my prototype flexible format code into another PR. How is your testing going on the alternative formats going?

eg. emitting data with timestamps included.

PS: I filed PR #9 to separately bring the Current functionality to the Brultech ECM units.

[ghost]

@mrguessed,
Sorry it took me some time to get back to you. I've been upgrading some of my hardware....
I tried the Value and Grafana payloads with success. For the JSON payload, btmon fails on startup. I used the following (and a few other alternates) without success:
MQTT_VALUE_FORMAT = "{{"value":{metric.value:.3f}, "timestamp":{metric.timestamp}}}"

Thanks, Tom