Merge pull request #84 from houseofsecrets/develop

LCM LoRA quick mode handling

README.md

@@ -5,49 +5,49 @@ the canvas when the image is generated.
 
 ## Controls
 
-| Key / Mouse button            | Control                                         |
-| ----------------------------- | ----------------------------------------------- |
-| Left button                   | Draw with the current brush size                |
-| Middle button                 | Draw with a white color brush                   |
-| `e` + Left button             | Eraser brush (bigger)                           |
-| Scroll up / down              | Increase / decrease brush size                  |
-| `1` to `9`                    | Set brush size                                  |
-| `backspace`                   | Erase the entire sketch                         |
-| `shift` + Left button         | Draw a line between two clicks                  |
-| `RETURN` or `ENTER`           | Request image rendering                         |
-| `ctrl` + `i`                  | Interrupt image rendering                       |
-| `c`                           | Display current configuration while pressed     |
-| `p`                           | Edit prompt                                     |
-| `alt` + `p`                   | Edit negative prompt                            |
-| `a`                           | Toggle autosave                                 |
-| `shift` + `t`                 | Cycle render wait time (+0.5s, or off)          |
-| `ctrl` + `p`                  | Pause dynamic rendering                         |
-| `q`                           | Toggle quick rendering : low steps & HR fix off |
-| `n`                           | Random seed value                               |
-| `ctrl` + `n`                  | Edit seed value                                 |
-| `UP` / `DOWN`                 | Increase / decrease seed by 1                   |
-| `ctrl`+ `s`                   | Save the current generated image                |
-| `ctrl`+ `o`                   | Open an image file as sketch                    |
-| `ctrl`+ `d`                   | Call ControlNet detector (replace sketch)       |
-| `shift` + `ctrl`+ `d`         | Cycle ControlNet detectors                      |
-| `h`                           | Toggle HR fix                                   |
-| `shift` + `h`                 | Cycle HR fix scale                              |
-| `shift` + `u`                 | Cycle HR upscalers                              |
-| `shift` + `d`                 | Cycle denoising strengths                       |
-| `shift` + `s`                 | Cycle samplers                                  |
-| `b`                           | Toggle batch rendering                          |
-| `shift` + `b`                 | Cycle batch sizes                               |
-| `shift` + `c`                 | Cycle CLIP skip settings                        |
-| `shift` + `m`                 | Cycle ControlNel models                         |
-| `shift` + `w`                 | Cycle ControlNel weights                        |
-| `shift` + `g`                 | Cycle ControlNel guidance ends                  |
-| `shift` + `ctrl` + `g`        | Toggle ControlNel pixel perfect mode            |
-| `keypad 0`                    | Restore starting settings                       |
-| `keypad 1-9`                  | Load custom rendering preset                    |
-| `ctrl` + `keypad 1-9`         | Save custom rendering preset                    |
-| `alt` + `keypad 1-9`          | Load custom ControlNet preset                   |
-| `ctrl` + `alt` + `keypad 1-9` | Save custom ControlNet preset                   |
-| `x` or `ESC`                  | Quit                                            |
+| Key / Mouse button            | Control                                            |
+| ----------------------------- |----------------------------------------------------|
+| Left button                   | Draw with the current brush size                   |
+| Middle button                 | Draw with a white color brush                      |
+| `e` + Left button             | Eraser brush (bigger)                              |
+| Scroll up / down              | Increase / decrease brush size                     |
+| `1` to `9`                    | Set brush size                                     |
+| `backspace`                   | Erase the entire sketch                            |
+| `shift` + Left button         | Draw a line between two clicks                     |
+| `RETURN` or `ENTER`           | Request image rendering                            |
+| `ctrl` + `i`                  | Interrupt image rendering                          |
+| `c`                           | Display current configuration while pressed        |
+| `p`                           | Edit prompt                                        |
+| `alt` + `p`                   | Edit negative prompt                               |
+| `a`                           | Toggle autosave                                    |
+| `shift` + `t`                 | Cycle render wait time (+0.5s, or off)             |
+| `ctrl` + `p`                  | Pause dynamic rendering                            |
+| `q`                           | Toggle quick rendering, with LCM LoRA if available |
+| `n`                           | Random seed value                                  |
+| `ctrl` + `n`                  | Edit seed value                                    |
+| `UP` / `DOWN`                 | Increase / decrease seed by 1                      |
+| `ctrl`+ `s`                   | Save the current generated image                   |
+| `ctrl`+ `o`                   | Open an image file as sketch                       |
+| `ctrl`+ `d`                   | Call ControlNet detector (replace sketch)          |
+| `shift` + `ctrl`+ `d`         | Cycle ControlNet detectors                         |
+| `h`                           | Toggle HR fix                                      |
+| `shift` + `h`                 | Cycle HR fix scale                                 |
+| `shift` + `u`                 | Cycle HR upscalers                                 |
+| `shift` + `d`                 | Cycle denoising strengths                          |
+| `shift` + `s`                 | Cycle samplers                                     |
+| `b`                           | Toggle batch rendering                             |
+| `shift` + `b`                 | Cycle batch sizes                                  |
+| `shift` + `c`                 | Cycle CLIP skip settings                           |
+| `shift` + `m`                 | Cycle ControlNel models                            |
+| `shift` + `w`                 | Cycle ControlNel weights                           |
+| `shift` + `g`                 | Cycle ControlNel guidance ends                     |
+| `shift` + `ctrl` + `g`        | Toggle ControlNel pixel perfect mode               |
+| `keypad 0`                    | Restore starting settings                          |
+| `keypad 1-9`                  | Load custom rendering preset                       |
+| `ctrl` + `keypad 1-9`         | Save custom rendering preset                       |
+| `alt` + `keypad 1-9`          | Load custom ControlNet preset                      |
+| `ctrl` + `alt` + `keypad 1-9` | Save custom ControlNet preset                      |
+| `x` or `ESC`                  | Quit                                               |
 
 _Note_ : "Cycle" shortcuts type will wait for the `shift` key to be released before launching the rendering.
 
@@ -95,6 +95,18 @@ models folder of the controlnet extension.
     ".\stable-diffusion-webui\extensions\sd-webui-controlnet\models"
 ```
 
+### Quick mode / LCM LoRA
+
+The quick mode is able to use a very low number of steps with [Latent Consistency Models LoRAs](https://huggingface.co/blog/lcm_lora) .
+To be able to use it fully, please download the LCM LoRAs for SD 1.5 and/or SDXL on huggingface :
+- [LCM LoRA for SD 1.5](https://huggingface.co/latent-consistency/lcm-lora-sdv1-5)
+- [LCM LoRA for SDXL](https://huggingface.co/latent-consistency/lcm-lora-sdxl)
+
+Save the safetensors files as `LCM_LoRA_Weights.safetensors` in the `./models/LoRA/` directory of your Stable Diffusion WebUI installation.
+The name of the LoRA is configurable in the JSON configuration files `controlnet.json` and `img2img.json` with the `quick.lora` entry.
+You can also configure the steps, cfg scale, sampler, and LoRA weight used. The default quick setting gives a good quality/performance ratio
+with DPM++ SDE Karras and 6 steps. HRFix is disabled when first entering quick mode, but you can activate it if wanted by pressing the `h` key.
+
 ### Autosave
 
 The images can be auto-saved after each rendering into `outputs` and `outputs/autosave` directories. The maximum

configs/controlnet-high.json-dist

@@ -5,8 +5,14 @@
     "seed": 42,
     "batch_size": 1,
     "steps": 16,
-    "quick_steps": 12,
     "cfg_scale": 7,
+    "quick": {
+        "steps": 6,
+        "cfg_scale": 3,
+        "lora":  "LCM_LoRA_Weights",
+        "lora_weight": 0.4,
+        "sampler": "DPM++ SDE Karras"
+    },
     "width": 640,
     "height": 720,
     "override_settings": {

configs/controlnet.json-dist

@@ -5,8 +5,14 @@
     "seed": 42,
     "batch_size": 1,
     "steps": 16,
-    "quick_steps": 12,
     "cfg_scale": 7,
+    "quick": {
+        "steps": 6,
+        "cfg_scale": 3,
+        "lora":  "LCM_LoRA_Weights",
+        "lora_weight": 0.4,
+        "sampler": "DPM++ SDE Karras"
+    },
     "width": 512,
     "height": 512,
     "override_settings": {

configs/img2img.json-dist

@@ -4,8 +4,14 @@
     "seed": 42,
     "batch_size": 1,
     "steps": 16,
-    "quick_steps": 12,
     "cfg_scale": 7,
+    "quick": {
+        "steps": 6,
+        "cfg_scale": 3,
+        "lora":  "LCM_LoRA_Weights",
+        "lora_weight": 0.4,
+        "sampler": "DPM++ SDE Karras"
+    },
     "override_settings": {
         "CLIP_stop_at_last_layers": 1
     },

scripts/common/state.py

@@ -26,6 +26,8 @@ class State:
     render = {
         "checkpoint": None,
         "vae": None,
+        "steps": 16,
+        "cfg_scale": 7,
         "hr_scales": [],
         "hr_scale": 1.0,
         "hr_scale_prev": 1.25,
@@ -45,6 +47,7 @@ class State:
         "use_invert_module": True,
         "quick_mode": False,
         "clip_skip_setting": 'clip_skip',
+        "quick": {}
     }
     control_net = {
         "config_file": "configs/controlnet.json",
@@ -162,6 +165,9 @@ def update_config(self, preload=False):
             hr_scales.insert(0, 1.0)
         hr_scale = hr_scales[0]
 
+        self.render['steps'] = self.configuration["config"].get("steps", 16)
+        self.render['cfg_scale'] = self.configuration["config"].get("cfg_scale", 7)
+        self.render['quick'] = self.configuration["config"].get("quick", {})
         self.render["hr_scale_prev"] = hr_scales[1]
         self.render["denoising_strengths"] = self.configuration["config"].get("denoising_strengths", [0.6])
         self.render["denoising_strength"] = self.render["denoising_strengths"][0]
@@ -219,6 +225,10 @@ def update_settings(self):
             self.render["hr_scale"] = self.render["hr_scales"][1]
             self.render["batch_hr_scale_prev"] = self.render["hr_scale"]
 
+        self.render['steps'] = settings.get("steps", 16)
+        self.render['cfg_scale'] = settings.get("cfg_scale", 7)
+        self.render['quick'] = settings.get("quick", {})
+
         self.gen_settings["prompt"] = settings.get('prompt', '')
         self.gen_settings["negative_prompt"] = settings.get('negative_prompt', '')
 

scripts/common/utils.py

@@ -233,9 +233,12 @@ def payload_submit(state, image_string):
     with open(state.json_file, "r") as f:
         json_data = json.load(f)
 
-    if state.render["quick_mode"]:
+    quick_mode = state.render["quick_mode"] and json_data.get('quick', None) is not None
+
+    if quick_mode:
         # use quick_steps setting, or halve steps if not set
-        json_data['steps'] = json_data.get('quick_steps', json_data['steps'] // 2)
+        json_data['steps'] = json_data['quick'].get('steps', json_data['steps'] // 2)
+        json_data['cfg_scale'] = json_data['quick'].get('cfg_scale', json_data['cfg_scale'])
 
     if not json_data.get('controlnet_units', None):
         json_data['controlnet_units'] = [{}]
@@ -253,7 +256,7 @@ def payload_submit(state, image_string):
         json_data['controlnet_units'][0]['module'] = 'invert'
     if not state.render["pixel_perfect"]:
         json_data['controlnet_units'][0]['processor_res'] = min(state.render["width"], state.render["height"])
-    json_data['hr_second_pass_steps'] = max(8, math.floor(int(json_data['steps']) * state.render["denoising_strength"]))  # at least 8 steps
+    json_data['hr_second_pass_steps'] = max(4, math.floor(int(json_data['steps']) * state.render["denoising_strength"]))  # at least 4 steps
 
     if state.render["hr_scale"] > 1.0:
         json_data['enable_hr'] = 'true'
@@ -263,11 +266,16 @@ def payload_submit(state, image_string):
     json_data['batch_size'] = state.render["batch_size"]
     json_data['seed'] = state.gen_settings["seed"]
     json_data['prompt'] = state.gen_settings["prompt"]
+    if quick_mode and json_data['quick'].get('lora', None):
+        json_data['prompt'] += f" <lora:{json_data['quick']['lora']}:{json_data['quick']['lora_weight']}>"
     json_data['negative_prompt'] = state.gen_settings["negative_prompt"]
     json_data['hr_scale'] = state.render["hr_scale"]
     json_data['hr_upscaler'] = state.render["hr_upscaler"]
     json_data['denoising_strength'] = state.render["denoising_strength"]
-    json_data['sampler_name'] = state.samplers["sampler"]
+    if quick_mode and json_data['quick'].get('sampler', None):
+        json_data['sampler_name'] = json_data['quick']['sampler']
+    else:
+        json_data['sampler_name'] = state.samplers["sampler"]
 
     if json_data.get('override_settings', None) is None:
         json_data['override_settings'] = {}
@@ -303,22 +311,31 @@ def get_img2img_json(state):
             json_data['width'] = im.width
             json_data['height'] = im.height
 
+    quick_mode = state.render["quick_mode"] and json_data.get('quick', None) is not None
+
     json_data['init_images'] = [data]
 
     json_data['seed'] = state.gen_settings["seed"]
     json_data['prompt'] = state.gen_settings["prompt"]
+    if quick_mode and json_data['quick'].get('lora', None):
+        json_data['prompt'] += f" <lora:{json_data['quick']['lora']}:{json_data['quick']['lora_weight']}>"
     json_data['negative_prompt'] = state.gen_settings["negative_prompt"]
     json_data['denoising_strength'] = state.render["denoising_strength"]
-    json_data['sampler_name'] = state.samplers["sampler"]
+    if quick_mode and json_data['quick'].get('sampler', None):
+        json_data['sampler_name'] = json_data['quick']['sampler']
+    else:
+        json_data['sampler_name'] = state.samplers["sampler"]
 
     if json_data.get('override_settings', None) is None:
         json_data['override_settings'] = {}
 
     json_data['override_settings'][state.render['clip_skip_setting']] = state.render["clip_skip"]
 
-    if state.render["quick_mode"]:
+    if quick_mode:
         # use quick_steps setting, or halve steps if not set
-        json_data['steps'] = json_data.get('quick_steps', json_data['steps'] // 2)
+        json_data['steps'] = json_data['quick'].get('steps', json_data['steps'] // 2)
+        json_data['cfg_scale'] = json_data['quick'].get('cfg_scale', json_data['cfg_scale'])
+
     return json_data
 
 

scripts/views/PygameView.py

@@ -743,8 +743,8 @@ def display_configuration(self, wrap=True):
             'state/samplers/sampler',
             'state/render/vae',
             'state/render/render_size',
-            'settings.steps',
-            'settings.cfg_scale',
+            'state/render/steps',
+            'state/render/cfg_scale',
             'state/render/hr_scale',
             'state/render/hr_upscaler',
             'state/render/denoising_strength',
@@ -765,8 +765,8 @@ def display_configuration(self, wrap=True):
         text = ''
 
         for field in fields:
-            if field == 'settings.steps' and self.state.render["quick_mode"]:
-                field = 'settings.quick_steps'
+            if field in ('state/render/steps', 'state/samplers/sampler', 'state/render/cfg_scale', 'state/gen_settings/prompt') and self.state.render["quick_mode"]:
+                field = 'state/render/quick/'+field[field.rfind('/')+1:]
 
             # Display separator
             if field.startswith('--'):
@@ -777,7 +777,14 @@ def display_configuration(self, wrap=True):
             label = ''
             value = ''
 
-            if '.' in field:
+            if field == 'state/render/quick/prompt':
+                field_components = field.replace('state/', '').split('/')
+                label = field_components[2]
+                value = self.state.gen_settings['prompt']
+                if self.state.render['quick'].get('lora', None) and self.state.render['quick'].get('lora_weight', None):
+                    value += f" <lora:{self.state.render['quick']['lora']}:{self.state.render['quick']['lora_weight']}>"
+
+            elif '.' in field:
                 field = field.split('.')
                 var = globals().get(field[0], locals().get(field[0], None))
                 if var is None:
@@ -797,6 +804,11 @@ def display_configuration(self, wrap=True):
                     field_components = field.replace('state/', '').split('/')
                     label = field_components[1]
                     field_value = getattr(self.state, field_components[0])[field_components[1]]
+                    if isinstance(field_value, dict) and len(field_components) > 2:
+                        label = field_components[2]
+                        field_value = field_value.get(field_components[2], None)
+                        if field_components[1] == 'quick':
+                            field_value = f'{field_value} -quick-'
                 else:
                     label = field
                     field_value = globals().get(field, locals().get(field, None))