Create EnOceanStorage for storing learned devices

[kipe]

More specific issue, derived from #15.

@romor wrote:

• in EnOcean, the sender address can be either a Unique ID (manufacturer chip-ID) or an address based on the Base-ID
• the Base-ID can be set and read by COMMAN_COMMAND CO_WR_IDBASE (07) and CO_RD_IDBASE (08) and must be in range 0xFF800000-0xFFFFFF80 with the last 7 bits set to zero
• these seven bits are used to emulate 128 IDs and are usually incremented for each device learnt, so one module can emulate 128 senders using the Base-ID
• the actuator learns this unique Sender-ID during teach-in and uses this for addressing

So, I see two possible approaches for this enocean library:

1. the callback specified above, which leaves the user the obligation to determine the address (or the offset to the base-ID) based on the destination address
2. the enocean library maintains this mapping, which implies the need to store, read, reset,... this

I'll start working on a storage, which would take care of this. According to some tests, JSON seems to be fastest way of storing the data, so I propose we use it to implement the storage.

My proposed structure is something similar to this:

{
    "storage_version": 1,
    "used_transmitter_ids": [1, 4, 12],
    "devices": {
        "0xF6": {},
        "0xD5": {
            "01:64:18:39": {
                "func": "0x00",
                "type": "0x01",
                "transmitter_id": 4
            }
        },
        "0xA5": {
            "01:94:E3:B9": {
                "func": "0x02",
                "type": "0x05",
                "transmitter_id": 12
            },
            "01:94:E4:B9": {
                "func": "0x02",
                "type": "0x02",
                "transmitter_id": 1
            }
        }
    }
}

Is there anything that should be altered or added in the first implementation phase? This structure should be created in a way that future needs can be quite easily added. Also, I included a storage_version, which can be used to create any required conversions.

[TheMeaningfulEngineer]

"transmitter_id" is a arbitrary value?
Could it be something more human readable like "name":"STM 320 Thermometer Bedroom"

[kipe]

Actually, it's better to change Transmitter ID to Sender ID. It is number from 0 to 128, used as offset from the Base ID.
And no, I'm not going to include anything like names etc. to this file, as it's not meant for the uses. The user should stuff like storing names in his/hers application.

[kipe]

Basic implementation started at storage.

[kipe]

The current implementation doesn't actually implement the storage anywhere.
In my opinion, the storage should be tied to Communicator, as that's where we're going to handle the packages (as per #34). So basically, Storage.save_device() should be called, if RadioPacket.learn is set or UTETeachIn is received.
Furthermore Storage.load_device() should be called to try and find the saved EEP when receiving a RadioPacket, so we have an idea where to start parsing?

@romor comments?

[romor]

I agree to your proposal.

Since we probably will not require to have all devices learnt before their usage there is the need to also add devices with their EEP information externally. So there needs to be something like Communicator.storage.define_device(), Communicator.storage.get_devices() and Communicator.storage.remove_device().

[Ethal]

Hi,
Could we add also the manufacturer id ?

and i would proposed for structure

        "01:64:18:39": {
            "rorg":"0xD5",
            "func": "0x00",
            "type": "0x01",
            "manufacturer":"0x0B",
            "transmitter_id": 4
        },

[kipe]

@Ethal Manufacturer ID is already defined in my current version, defaulting to None.

@romor There will be those functions, as Storage.save_device(Device), device = Storage.load_device(device_id), [devices] = Storage.get_devices() and Storage.remove_device(device_id).

So in essence, I've now implemented storage.Device -object, which will be used for saving, loading, and listing devices.

[kipe]

Hmm, I'm struggling a bit with this.
The original proposal saved the data as hex strings. However, this is a compromise for speed, as more conversions would be required (hex -> int -> hex in many cases).

Therefore I'd like to restructure this to use

devices = {
    "01:64:18:39": {
        "eep_rorg": 213,
        "eep_func": 0,
        "eep_type": 0,
        "manufacturer_id": 124,
        "transmitter_offset": 4
    }
}

This makes the implementation a bit simpler, while making it less human-readable (as the documentation is in hex). In my own opinion, there shouldn't be much reason to handle the file directly, and therefore I'd go with this solution.

What do you think?

[romor]

I also would prefer the integer storage against the string version. I guess everyone going into such detail is also able to convert hex values.
I would even consider storing the device address as integer.

[kipe]

I would also like to store it as an integer, but unfortunately integer keys aren't allowed in JSON ;)

[Ethal]

Hi,
Why transmitter_offset instead of transmitter_id ?

[kipe]

Sorry for the late reply, been extremely busy.
transmitter_offset is used, as it's easier (and faster) to handle with the related lists (used_transmitter_offsets etc).

I might however remove the used_transmitter_offsets and just use a generator to find the next available offset, not sure if we need to save the same data in multiple places...

[kipe]

My current storage with the latest commits in storage-branch looks like this:

{
    "used_transmitter_offsets": [],
    "storage_version": 1,
    "devices": {
        "01:81:B7:44": {
            "eep_type": 5,
            "eep_rorg": 165,
            "eep_func": 2,
            "manufacturer_id": null,
            "id": [1, 129, 183, 68],
            "transmitter_offset": null
        },
        "00:29:4A:50": {
            "eep_type": 2,
            "eep_rorg": 246,
            "eep_func": 2,
            "manufacturer_id": null,
            "id": [0, 41, 74, 80],
            "transmitter_offset": null
        },
        "01:94:B9:46": {
            "eep_type": 1,
            "eep_rorg": 212,
            "eep_func": 1,
            "manufacturer_id": 62,
            "id": [1, 148, 185, 70],
            "transmitter_offset": null
        },
        "01:80:A1:A0": {
            "eep_type": 1,
            "eep_rorg": 213,
            "eep_func": 0,
            "manufacturer_id": null,
            "id": [1, 128, 161, 160],
            "transmitter_offset": null
        },
        "01:94:E3:B9": {
            "eep_type": 1,
            "eep_rorg": 212,
            "eep_func": 1,
            "manufacturer_id": 62,
            "id": [1, 148, 227, 185],
            "transmitter_offset": null
        },
        "01:82:5D:AB": {
            "eep_type": 1,
            "eep_rorg": 213,
            "eep_func": 0,
            "manufacturer_id": null,
            "id": [1, 130, 93, 171],
            "transmitter_offset": null
        }
    }
}

So in essence, EEP-information is saved once they're found (typically in teach-in messages).

If they're not found, the Device can be fetched using Storage.load_device(device_id). This can in turn be modified using Device.update(**kwargs) and then saved using Storage.save_device(device). (or manually editing the file, when the system is not running).