Entity creation from UI

[farmio]

We'd like to be able to create HA KNX entities from the UI instead of YAML.

Let's see if we can come up with ideas on how this should look like and work.

[probert94]

A manual configuration using the UI would already make it a lot easier to use the integration, at least for users not confortable with YAML.
The first idea, that comes to my mind is to first select the entity type (Binary Sensor, Cover, Light etc.).
Depending on the selected type the next step shows more or less fields. For the binary sensor, for example, only a "state_address" is required, a light instead is way more complex and even depends on more details (dimmable, tunable white, rgb(w) etc.).
That way a user would not need to remember all of the property names.
The downside of this approach is, that an experienced user will probably be way slower then in YAML. Some people even create custom scripts that convert a group address export into a YAML.

So the UI should probably allow some kind of patch creation. Usually the structure in ETS follows some standards, so addresses of the same entity are either grouped in blocks or in several middle groups.
Example:
Light 1:

• switch: 1/0/0
• state: 1/1/0 or 1/0/1
• relative dimming: 1/2/0 or 1/0/2
• absolute dimming: 1/3/0 or 1/3/0
• dimming state: 1/4/0 or 1/0/4
• ...

Of course there are quite a few things to consider, for example TW / RGB(W) might be separated in multiple blocks (one for each color).
Also, in the "block" system one might not reserve the dimming address for non dimmable lights etc.
So adding this kind of logic will already be pretty complex.

Last but not least, the ideal solution would of course be an import of a KNX project with auto detection of the entities.
Unfortunately this is pretty complex and probably won't work in all situations / with all projects.
However, I found a pretty interesting (ggerman) video from PEAKnx, where they explain how the project should be structured in order to detect the entities:
https://www.youtube.com/watch?v=c-fKEkp9LBs

[farmio]

The possibility to script your config creation will probably not go away.

Also the UI created config will have to be stored somewhere, and this will probably be some json in .storage/knx as it is easy to work with and HA provides good tooling for that. So if someone liked to, this json file can be created / extended by custom scripts just like yaml can currently.

That said, I don't think we have to get rid of YAML (at least in the near future) so this will also be an option still.

[farmio]

Imposing naming or project structure conventions for our users seems a bit harsh to me.

The most common case for our users is that they already have a working installations when they start to use HA. Some users even don't own ETS licenses, so they wouldn't even be able to change their projects accordingly. Others may use different systems that already impose some structure on their own (like your example) that may be incompatible.

For those who are committed to a specific GA numbering schema, scripting could be a viable option though.

[probert94]

Imposing naming or project structure conventions for our users seems a bit harsh to me.

I would not force them to use an exact structure and namig, the manual configuration always needs to be possible.
But in order to provide a simple auto import you probably must make some assumptions about the project structure and naming.
As KNXA itself provides some guidelines regarding ETS projects and there are a few other popular structures teached by books like "ETS Schnellkurs" and so on. So (hopefully) most projects will follow some rules and at least some entities could be auto detected.

So my idea is to make the configuration simpler (as automatic as possible) for users with well-structured ETS Project while still providing a simple manual configuration for all others.

[marvin-w]

I honestly doubt that there is a unified way (as of now) of how we can achieve automatically configuring entities.

Instead I'd opt for an (improved) ETS Look&Feel where users can see their project in the UI and then we could potentially work with context menus to start creating entities based on this user input. This would work really well for entities that have one or two DPTs but not so much for our bigger ones like climate and light. For those we definetly need dedicated views.

In this views we can use the parsed project data to prefilter the available DPTs during selection to increase the user experience while still allowing the user to manually switch the filtering off (i.e. if the DPTs are not maintained accordingly in the project file).

[farmio]

Looking into peaKnx it seems they not only expect certain numbering schema, but also maintain a database of devices and their possible parameters to be able to parse correctly.
And from what I read it doesn't sound that reliable.

See (German) https://knx-user-forum.de/forum/supportforen/peaknx/1837925-import-geräte-parsing-buggy?p=1868544#post1868544

[probert94]

I did not know about that, I just did a quick research on how other visualizations deal with the project import and it looked pretty good at first glance.
However, it seems like parsing the project is even more complex that I thought.
So it will probably be better to focus on a easy manual entity creation instead of an automatic one.

[ztardik]

I was working with a visualisation SW that offers auto-import (Divus). They approach is to offer filters to the user. For example, the following GA's can be imported easily:

0/0/0 [Kitchen] Spot /c
0/0/1 [Kitchen] Spot /sta
0/0/2 [Kitchen] LED /c
0/0/3 [Kitchen] LED /sta
0/0/4 [Kitchen] LED /dimm

User set's the filter as - command: '/c', status: '/sta', dimming brighter/darker: '/dimm'
The GA description is used then to determine what belongs together as one entity.

In small projects the manual selection is ok, but in case of bigger ones I would like to see some kind of preselection. Example would be an apartment building I was working on some times ago:

0/0 Building A, Floor 0, Apt1, Lights,
0/1 Building A, Floor 0, Apt1, HVAC,
0/2 Building A, Floor 0, Apt2, Lights,
0/3 Building A, Floor 0, Apt2, HVAC,
0/4 Building A, Floor 1, Apt1, Lights,
0/5 Building A, Floor 1, Apt1, HVAC,
0/4 Building A, Floor 1, Apt2, Lights,
0/5 Building A, Floor 1, Apt2, HVAC

GA's from the example above. What I would like to see is the option to choose only one area to work with at any given moment, instead of scrolling trough all addresses or having to manually write each into a search box.
It would be great if when se selecting 0/0/2 [Kitchen] LED /c you got automatically shown 0/0/3 and 0/0/4 as related GAs.

[probert94]

@ztardik thats an interesting approach. So instead of needing a specific structure and naming-convention to auto-import all at once, you import one entity-type alone and give hints about the used names.
I could imagine something like this:

Basically you specify some kind of address range, which could be a main-group, a middle-group or maybe a range (f.e. 1/0/0 - 1/0/10).
In the example it is the main-group "1/" which is used for lighting.
This might be needed, when you have other GAs with similar naming. For example, covers have a state address too.
Then you specify a pattern for each type of GA needed for the entities. The example uses dimmable lights, so there are 4 patterns (switch, state, brightness and brightness state). Each pattern consists of a type (starts with, ends with, contains, regex or whatever makes sense) and the value string.
In the example GAs for the switching addresses end with Switch, GAs for the state address end with Switch State etc.
I also added a checkbox to specify if non connected GAs should be ignored. The reason for this is, that in my project, I reserved the GAs for dimming, even if the light is switchable only. But of course they should be imported as switchable, not dimmable.
Then there are the additional properties, like the "synchronize" property, which can't be read out of the project. So the values set here are defaults for all imported entities.

Of course this is just a quick idea and needs a lot more elaboration. For example it is not clear how to connect different GAs to each other. One possibility would be the name, so that Living Room Ceiling Switch and Living Room Ceiling Switch State would result in an entity Living Room Ceiling with the GAs for switch and switch state set to the respective values.

Also this approach could become quite complex for RGB(W) lights as there you could have 1 GA for the color or all GAs (switch, state, brightness, brightness state) for each color.

[ztardik]

This is an example how most of my projects look like (everybody do it differently). Sometimes the main groups is the floor separator, sometimes the middle group. It depends from the size of the project. But basically you always have something fixed describing what it is and where it is.
The approach you described is good, flexible enough for most types of control. RGB(W) may be more complex, but how many times you have to deal with it? I think it's better to focus on the most used types and later add others.

Because all projects are different, automatic grouping is hard to make, but with implementation of filters, some clever sorting could check for same text in description which is not in filters (for example [Room 01] but ignore /c, /sta, /dimm ), then do something with it.
Type of control must be determined from the group type in the project. In this example there is two/three types of "/c":

 0/0/1 [Entry Hall] Spot /c
 0/2/1 [Room 01] SPV /c
 0/2/7 [Room 01] Cooling Output /c

In this case 0/0/1 is clearly a 1.001 switch, 0/2/1 is a 9.001 Temperature, and 0/2/7 can be 1.001 switch or 5.001 percentage.

[farmio]

For the more complex platforms, namely

• climate
• cover
• light

I think we could look into ETS Functions (and DPT lookup) for GA suggestions.

Although they don't provide all information we'd need for an entity to be created, we could speed up the config process with these.

[empx92]

are climate and cover already integrated in the ui?

[farmio]

No.

[probert94]

I just had a quick look at another KNX tool, the Gira Project Assistant. It seems like it does not auto create entities but instead just import the GAs and let you cinnect them to entities.
This would of course be a great first (or second) step.
Wirh some additional logic it should then be possible to give better suggestions to which GAs are suitable for other addresses of the same entity (similar name etc.)

[probert94]

I created a quick Figma prototype for the entity configuration in the UI:
https://www.figma.com/proto/0XQAVAZY0kFSYOvS61XnXp/KNX?type=design&node-id=1-165&scaling=min-zoom&page-id=0%3A1&starting-point-node-id=1%3A165
In this prototyp the configuration is done in the configuraion dialog from the integrations panel, similar to what other integrations do.
However, I had a quick discussion about it with @farmio and we are not sure if the entity-configuration is not too complex for those dialogs. Especially when it comes to RGB(W) lights with different addresses per color it will get quite complex (4 addresses per color + the entity configuration).
However, I think the principle of the configuration could stay more or less the same no matter where it is located.
The basic idea is, that the addresses are configured using a combobox. This allows you to give suggestions for addresses (based on DTP, names etc.), search for addresses (using address or name) and enter custom values if needed.
I personally would avoid a drag and drop configuration like in the ETS as it seems pretty slow to me. Generally using the mouse is way slower then the keyboard.

[farmio]

SelectSelector style drop-down boxes with custom value option (and multi-select option, as most of our GA-config-keys accept multiple addresses) are used for the interface device trigger in automations too - I think it's a pretty nice way to configure GAs.

They also inherently support internal GAs (but these aren't available in the suggestions).

It's also possible to use that on mobile, whereas drag&drop would be a challenge.

[philippwaller]

@probert94 & @farmio, fantastic concept! Clustering entities into devices in Home Assistant seems like a solid strategy. The main challenge, though, is incorporating device-specific functions into the device object, like automatic shading for certain roller shutter actuators. Ideally, the device object should be versatile enough to include not only main functions but also link to device-specific entities. If possible, there should also be a generic device option (possibly at the HA Core level?), to cover specific use cases.

Currently, my YAML configuration is set up as if KNX devices were already integrated into Home Assistant. Each file represents a distinct device within Home Assistant. The schema works great and helps to keep consistency. However, managing all this through the UI would indeed be taking it to the next level.

Great work! Thank you!

[probert94]

I recently added another light source which I inserted in the middle of my lighting group addresses. So I had to move half of the addresses by one.
As the GAs are used as some kind of ID in Home Assistant, all the entity configuration (Name, Area etc.) as well as the history was wrong after that. The easiest fix I found was to delete all entities and recreate them.
So maybe a possibility to update the GAs of existing entities should be considered as well.

[farmio]

This can be done when we have UI config. We can simply use a UUID or an incremental counter to use for unique_id instead of having to rely on user input.

But then still your GAs will not auto-update. If you change GA numbers for whatever reason, you'll probably have to do that in the UI config too.

[heidle78]

Short question, in my KNX folder I can´t see the ETS project information - did I miss something? Monitor works great. Im imported my ETS project file and I can see all the information from KNX.

Beside that, good job!!!

[farmio]

Hi 👋! It's not released yet. We are still working on it.

[heidle78]

Thanks for your quick reply, was not sure if I made a mistake as the screenshots looked really promising and real. Can´t wait to get the new release with that feature - hope that simplifies the entity creation a little bit.

[farmio]

Currently it just shows a list of group addresses. We wanted to add some kind of "creation feature" before releasing. Still a lot to do though.

[Julius2342]

Probably a longshot here:

im still thinking if we one day might configure KNX devices via XKNX - without the use of ETS. My idea is to have a more-or-less standard configuration binary blob, optimized for HASS use. During the automated configuration process just the KNX addresses are replaced within the blob.

The System then knows which KNX addresses were assigned. this data then may be used for configuration.

:-)

[farmio]

Longshot for sure 😂

What devices would you think are good candidates for that?

Do you know if it was legally okay to ship a pre-defined KNX device application blob with xknx (or some sub-project)?

[Julius2342]

I would say any super-simple device, like e.g. https://katalog.gira.de/de_DE/datenblatt.html?id=658305 Program it, record the configuration. Change one single parameter e.g. von 1/23/42 to 1/42/23 and record the difference.

Regarding shipping: I would leave it up to the user to download the knxprod-zip files.

[ztardik]

I don't this this is possible or even worth spending time trying. You are saying it's a simple device, but look at the configuration options:

Each input (of two) has six different functions, and each function has different parameters. How many 'blobs' you have to create so that this become usable?
Many other devices are even more complicated. Of course one can make some basic setups available, but the usability would stay very low/basic.
What I think is worth trying is to make an alternative to ETS. A software compatible with ETS in any way possible, but with better/different workflow and usability.

[Damageplan1980]

Hi there,

although I don't use the Hager Domova TJA470 anymore, I always liked the way of adding devices to it.
There's a simple GUI where you can choose the type of device you want to add.
I give you a small example with a light:
You first have to choose the features your light supports.
There's a basic config where you can just set the group addresses for switching and the status.
You also can choose whether the light has color, is dimmable and whether it has color temperatur settings.
The KNX-Konfiguration section changes accordingly to the settings you choose.
At the input fields for entering the group addresses you can see what DPT is needed for every function.
If you click on the magniefier for searching your imported KNX project (group addresses) a popup opens where all the possible devices are filtered for the needed DPT.

Many greetings,
Matthias

Here are some screenshot:

[evercape]

Well done Matthias for the KNX UI integration into HA 2008.8 looks really nice and I think it will permit new users that dont like YAML coding to get on board with HA quicker.

Questions I have:

1. Whatever the UI does, where is that stored in YAML? I was hoping/assuming that the KNX UI front end has similar behaviour like HA UI front ends such as automation. So you could configure something in front end, and somewhere it stores it in YAML which one can copy and edit or paste elsewhere. It would be nice to have a switch UI-YAML mode button.

2. I have many KNX entities already configured via YAML. I prefer yaml to UI, but I also been programming for most of my life. Is there a way to "scrape" the YAML and have those KNX entities listed in the UI frontend? Like HA native automations shown (and cant be edited from UI as they are coded in YAML packages) ?

3. I like the idea of "grouping" KNX entities into devices. Prior to this new UI feature all KNX entities are just entities and not devices (except for the main interface). What I am looking for is to bundle a KNX switch (that has a switch and status binary object) together with an analogue sensor for current measurement into one device. Same applies to dimmable lights with current measurement. Or a climate control (aircon) object with current monitoring. I am a big fan of current monitoring as it provides a real confirmation if something is actually working or not (how many times did one assume some entity/device is on mean time a breaker or earth leakage has popped). So I almost exclusively use actuators with current monitoring (all from our German friends at mdt.de) and would like to bundle those in one device.

4. Lastly, I just used the UI for the first time. create a switch entity, added it to a new device light. then deleted the KNX entity. cannot delete the light device. Odd?

thanks for your help,
Martin

[evercape]

I kind of agree with @sbiela if you compare the KNX UI vs YAML with the HA native automation UI vs YAML the automation side behaves like one system - whereas on the KNX side you can end up with two different way of configuring the same concepts.

It does not matter if an automation is created in YAML - the UI shows it, and vice versa, if created via UI you can get the YAML in the automation files.

I suggest to consider the same approach for KNX UI vs YAML. By doing this, it would uplift the existing YAML files into configurable UI, and vice versa use UI to create YAML files.

Lastly, a point I made before, it would be amazing to create devices in HA (ideally via YAML) that permit various entities to be grouped together, ie a switch with status for a light together with a current sensor --> light with current measurement. Or to give another example an aircon climate entity with a fan entity, creating a modern aircon & fan device. I looked into templating and the old templating used to support some of these concepts, but not the new way of doing it. Any ideas outside custom integrations?

[evercape]

to not have two different systems

I don't see an issue with that, so I guess it comes down to personal preference.

Will it probably be possible to add the entities to devices afterwards

I think yaml entities will stay non-device... unless

• someone comes up with an acceptable way to do devices for those
• or someone makes a yaml-to-Ui migration

@farmio thanks Matthias for amazing work around KNX integration. Really makes a difference to have this all integrated. I wonder if UI for KNX is so important. People who use KNX probably have a decent tech degree and understanding, and should probably be rather comfortable with YAML?

Anyone working with KNX & HA I suggest to either choose YAML or KNX UI, but not mix both as it can probably get confusing.

I personally would vote for extending KNX YAML integration and spending more efforts on this, than on KNX UI. However, that is my personal opinion.

[sbiela]

I don't see an issue with that, so I guess it comes down to personal preference.

Yes, probably just my inner monk 📐

[probert94]

if you compare the KNX UI vs YAML with the HA native automation UI vs YAML the automation side behaves like one system - whereas on the KNX side you can end up with two different way of configuring the same concepts.

It seems like there is a difference between automation + dashboard configuration and integrations. While automations and dashboards can be configured in the UI to generate YAML instead of writing it, for integrations the YAML seems to be optional. So the integrations that can be configured with a UI don't generate YAML so thats probably the reason why they are separated in the KNX UI as well.

People who use KNX probably have a decent tech degree and understanding, and should probably be rather comfortable with YAML?

I don't think that KNX is a system made for people with technical understanding. It's probably quite the opposite. Usually you pay someone to install and configure a working KNX system, especially in the industries, where KNX actually comes from.
People who configure (or even plan and install) the KNX system themselfs are a minority for sure.
Also most people who use Home Assistant are probably pretty tech savvy but Home Assistant is trying to become a solution for "normal" end users as well. So in my opinion it is the right choice to also make the KNX integration configurable with the UI.
However, I have to agree that most of the times, UI configurations are slower for tech savvy people so there might be a need for some kind of "batch" creation or maybe a YAML import or something like that.

[farmio]

Oops, I dropped my previous reply at the wrong place, so here is a reference. Github phone app is wired sometimes 😬
#50 (comment)

[farmio]

it would be amazing to create devices in HA (ideally via YAML)

I don't think that would be allowed in HA Core. It would mean to accept user-defined (yaml) unique_ids.

I wonder if UI for KNX is so important.

It was the most requested feature for a long time - and probably still is. It also has huge potential when leveraging knxproj file data.

People who use KNX probably have a decent tech degree and understanding, and should probably be rather comfortable with YAML?

I think there is a very mixed user base. The general plan was to get more user-friendly.

[evercape]

Thanks @farmio yes - your contribution with the KNX integration definitely serves a large user base.

Do you think there is merit in considering to extend the existing KNX integration by adding certain data to existing KNX/HA devices?

As example, a lot of the actuators that I use (mostly MDT.de) have a functionality to BLOCK a device from working (including a BLOCK_STATE) whilst in block state. That is something I have not seen much within HA, yet is a great way of disabling functionality for security reason or whilst in season/out of season (heating, irrigation) and for occupation (home vs away mode).

Another example is to extend a light, switch, climate, fan object with power consumption sensor data. Obviously, a power sensor can work besides any of the those devices, however, there is also a good enough argument to bundle it. One advantage would be to then use the power consumption sensor data to be able to show on the tile card as state content.

Would be interesting to know what you think about this or if this is against any of the core principles of HA / KNX Integration?

Thanks
Martin

[farmio]

In HA there is a preference for creating separate entities over additional state attributes. It makes sense since those can have state_class, device_class, units etc. So a light entity shall not have a power attribute, but an extra power entity (ideally both bundled by a HA device).

Regarding the block / lock thing, there is no HA entity functionality I know of, that could be used for this.