How to query properties and attributes of components?

[threefoldo]

5fca344 When traversing the element tree this way, I could query the "x,y" (origin) and "w,h" (size) of each widget, but how to get more information, the properties and attributes?

The code in process_key_input in windows.rs:

    let x = item.geometry().origin.x;
    let w = item.geometry().size.width;
    println!("x={x}");
    ......
    let mut child = item.first_child();
    while let Some(c) = child {
        self.dump_sub_tree(Some(node_id.clone()), &c, component_names, focus_item.clone());
        child = c.next_sibling();
    }

Sample slint:

Example := Window {    
    Rectangle {
        x: 10px;
        y: 10px;
        width: 50px;
        height: 50px;
        background: blue;
        Text {
                text: "Hello World";
                color: red;
            }
        Image {
            source: @image-url("https://slint-ui.com/logo/slint-logo-full-light.svg");
            // image-fit default is `contain` when in layout, preserving aspect ratio
        }
    }
}

My goal is to get the following output:
{['rectangle', x, y, w, h, ''], ['text', x, y, w, h, 'Hello Word'], ['image', x, y, w, h, '']}
Right now, only "x,y,w,h" are available, but not clear how to get the type of components, and how to read their properties or attributes.

[threefoldo]

907b581 Maybe I could use this "init" callback to save properties of all components.

[ogoffart]

What you're doing sounds quite interesting and I wonder what exactly you are trying to achieve.

Unfortunately, this is not really possible to do what you're asking.
One very important conceptual thing about Slint is that there is a compilation phase, so the Element tree and its property are compiled down into another representation that is used at runtime (Items). In between, there are optimization are removes some properties, and elements, inline them or move them around, and they no longer corresponds to what's in the code.

It would be nice to have an understanding of what you really want to achieve with that information so we could find a solution that works for you.
Could you elaborate on the specifics of what you are doing?

[threefoldo]

In simple terms, we need to find out the exact position (x,y,w,h) and type and text of all components. I almost get it done:

MainWindow := Window {
    preferred-width: 400px;
    preferred-height: 600px;

    property <int> px : self.x / 1px;
    property <int> py : self.y / 1px;
    property <int> pw : self.width / 1px;
    property <int> ph : self.height / 1px;
    init => {
        debug("window, \{px}, \{py}, \{pw}, \{ph}, \{self.title}");
    }

    VerticalBox {

        HorizontalBox {

            text-edit := LineEdit {
                placeholder-text: "What needs to be done?";
                property <int> px : self.x / 1px;
                property <int> py : self.y / 1px;
                property <int> pw : self.width / 1px;
                property <int> ph : self.height / 1px;
                property <string> ptext: text;
                init => { debug("edit, \{px}, \{py}, \{pw}, \{ph}, \{ptext}"); }
            }
            btn := Button {
                text: "Add New Entry";
                property <int> px : self.x / 1px;
                property <int> py : self.y / 1px;
                property <int> pw : self.width / 1px;
                property <int> ph : self.height / 1px;
                property <string> ptext: text;
                init => { debug("button, \{px}, \{py}, \{pw}, \{ph}, \{ptext}"); }
            }
        }

        list-view := ListView {
        }
        HorizontalBox {
            Button {
                text: "Remove Done Items";
                property <int> px : self.x / 1px;
                property <int> py : self.y / 1px;
                property <int> pw : self.width / 1px;
                property <int> ph : self.height / 1px;
                property <string> ptext: text;
                init => { debug("button, \{px}, \{py}, \{pw}, \{ph}, \{ptext}"); }
            }
        }
    }
}

The only problem is that x,y are relative position, not absolute position, and the code is a little bit messy.

[threefoldo]

Our goal is to generate many UI screenshot with annotations, then we can use them to train deep learning models. That's why we need information of widgets, only these properties: type, x,y, w,h, text.

I haven't read the code of slint.internal.core, but the concept of compilation process is clear to me. The .slint file is a specification, not exactly the same as runtime. For example, some components can switch between visible and invisible state, the position will change accordingly. The task is a little bit difficult, but I guess it's still doable.

In the compilation stage (internal/compiler/load_builtins.rs), each built-in widget is registered with their properties, so the widget name and properties are stored somewhere. If the runtime item could be connected with widgets by UUID or something similar, then the runtime item can attach back to the original widgets, thus get all the information. Actually, not all the information, just the above six properties.

[ogoffart]

Interesting. I wonder what kind of training you are doing. What is the goal of this AI?

So to answer the question, I believe you could add a function to the Item trait that print the properties. The item trait is defined there:

slint/internal/core/items.rs

Line 110 in ad5b7be

pub struct ItemVTable {

(It is defined as a vtable using the vtable macro because we want the functions to be callable from C++, but despite the wired declaration, adding function in that struct just add functions in the trait)
Then you can re implement that function for each item and there you can print each properties since you have access to them.

That would be one way to do it.

Another way to do it, without adding a fn to the Item trait, would be to try to downcast into each possible item. (Example, there we downcast the root item to a window to access it's background property:

slint/internal/core/software_renderer.rs

Line 240 in ad5b7be

if let Some(window_item) = crate::items::ItemRef::downcast_pin::<crate::items::WindowItem>(

and you can do the same for each possible item and get all their properties)

[threefoldo]

Thank you very much, I will try them later.

The general idea is as follow, given an UI screenshot, recognize all items and relationship between them. There are automatically generated data for iOS and Android apps, but no data for PC and web applications. That's why we want to do this.