Add segmentation painting and MLSD support.

[CodeMonkey80]

Would love to see segmentation painting as an added feature. It seems that most of the brush painting groundwork is already done, so it's a matter of figuring out how to use the color presets: https://www.reddit.com/r/StableDiffusion/comments/11e4vxl/paint_by_color_numbers_with_controlnet/

Since this wonderful app can already draw straight lines how about adding MLSD model support?

Good job on the app! Amazing how fast it's developing.

EDIT: Ok, so MLSD model apparently works when you add it to the model list in config.json but MLSD detector doesn't seem to be doing its job. I get a blank canvas when switching to it with an image already generated on the left side. The other detectors show proper results.

[Danamir]

It seems you are using MLSD as a detector, ie. the modules (preprocessors) part of ControlNet, not the models (rendering). The detectors/preprocessors are used to detect features in an image, or do some computations before the image can be used by the models for rendering.

Check your http://127.0.0.1:7860/controlnet/model_list URL to find the correct model to add to config.json controlnet_models list. For example on my system it's control_v11p_sd15_mlsd_fp16.

Once added to the configuration, restart the application and use the shortcut shift + m to select the MLSD model. Check your webui log, it should display something like ControlNet model control_v11p_sd15_mlsd_fp16 [77b5ad24] loaded before rendering.

There seems to be a bug with the mlsd detector at the moment. I can detect features in an image with the webui, but calling the same detector with ctrl + d gets a white image. It may be due to the image format sent by the API.

[CodeMonkey80]

No, no, you misread. MLSD model loaded and worked for me and the detector doesn't seem to do anything, just as you replied. It's all good when it comes to generating images but can't trace them. I guess that's something to solve.

Any comment on segmentation painting? Is it doable?

[Danamir]

Ok I was not sure you were using MLSD for the rendering part also.

In theory segmentation painting should work, but would necessitate at least a color picker and a paint filler of some kind. It's not something I expect to code anytime soon, but maybe someone will. 😅

Another approach could be to modify the img2img mode to send the image data to ControlNet instead, and use an external editor with the correct editing capabilities.

[Danamir]

It's all good when it comes to generating images but can't trace them. I guess that's something to solve.

The problem was with the default threshold values used. MLSD detector is now working after #47 .