Skip to content

Commit

Permalink
Merge branch 'main' into licence-update
Browse files Browse the repository at this point in the history
  • Loading branch information
Fannovel16 authored Aug 28, 2024
2 parents 149e78c + 07cad1c commit 648a5d7
Show file tree
Hide file tree
Showing 550 changed files with 647 additions and 612 deletions.
Binary file added NotoSans-Regular.ttf
Binary file not shown.
79 changes: 8 additions & 71 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,13 @@
# ComfyUI's ControlNet Auxiliary Preprocessors
![](./examples/example_mesh_graphormer.png)
Plug-and-play [ComfyUI](https://github.com/comfyanonymous/ComfyUI) node sets for making [ControlNet](https://github.com/lllyasviel/ControlNet/) hint images

The code is copy-pasted from the respective folders in https://github.com/lllyasviel/ControlNet/tree/main/annotator and connected to [the 🤗 Hub](https://huggingface.co/lllyasviel/Annotators).
"anime style, a protest in the street, cyberpunk city, a woman with pink hair and golden eyes (looking at the viewer) is holding a sign with the text "ComfyUI ControlNet Aux" in bold, neon pink" on Flux.1 Dev

All credit & copyright goes to https://github.com/lllyasviel.
![](./examples/CNAuxBanner.jpg)

# Marigold
Check out Marigold Depth Estimator which can generate very detailed and sharp depth map from high-resolution still images. The mesh created by it is even 3D-printable. Due to diffusers, it can't be implemented in this extension but there is an Comfy implementation by Kijai
https://github.com/kijai/ComfyUI-Marigold
The code is copy-pasted from the respective folders in https://github.com/lllyasviel/ControlNet/tree/main/annotator and connected to [the 🤗 Hub](https://huggingface.co/lllyasviel/Annotators).

![](./examples/example_marigold_flat.jpg)
![](./examples/example_marigold.png)
All credit & copyright goes to https://github.com/lllyasviel.

# Updates
Go to [Update page](./UPDATES.md) to follow updates
Expand Down Expand Up @@ -179,71 +175,12 @@ for o in history['outputs']:
# Examples
> A picture is worth a thousand words
Credit to https://huggingface.co/thibaud/controlnet-sd21 for most examples below. You can get the same kind of results from preprocessor nodes of this repo.
## Line Extractors
### Canny Edge
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_canny.png)
### HED Lines
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_hed.png)
### Realistic Lineart
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_lineart.png)
### Scribble/Fake Scribble
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_scribble.png)
### TEED Soft-Edge Lines
![](./examples/example_teed.png)
### Anyline Lineart
![](./examples/example_anyline.png)

## Normal and Depth Map
### Depth (idk the preprocessor they use)
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_depth.png)
## Zoe - Depth Map
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_zoedepth.png)
## BAE - Normal Map
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_normalbae.png)
## MeshGraphormer
![](./examples/example_mesh_graphormer.png)
## Depth Anything & Zoe Depth Anything
![](./examples/example_depth_anything.png)
## DSINE
![](./examples/example_dsine.png)
## Metric3D
![](./examples/example_metric3d.png)
## Depth Anything V2
![](./examples/example_depth_anything_v2.png)

## Faces and Poses
### OpenPose
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_openpose.png)
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_openposev2.png)

### Animal Pose (AP-10K)
![](./examples/example_animal_pose.png)

### DensePose
![](./examples/example_densepose.png)

## Semantic Segmantation
### OneFormer ADE20K Segmentor
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_ade20k.png)

### Anime Face Segmentor
![](./examples/example_anime_face_segmentor.png)

## T2IAdapter-only
### Color Pallete for T2I-Adapter
![](https://huggingface.co/thibaud/controlnet-sd21/resolve/main/example_color.png)

## Optical Flow
### Unimatch
![](./examples/example_unimatch.png)

## Recolor
![](./examples/example_recolor.png)
![](./examples/ExecuteAll1.jpg)
![](./examples/ExecuteAll2.jpg)

# Testing workflow
https://github.com/Fannovel16/comfyui_controlnet_aux/blob/master/tests/test_cn_aux_full.json
![](https://github.com/Fannovel16/comfyui_controlnet_aux/blob/master/tests/pose.png?raw=true)
https://github.com/Fannovel16/comfyui_controlnet_aux/blob/main/examples/ExecuteAll.png
Input image: https://github.com/Fannovel16/comfyui_controlnet_aux/blob/main/examples/comfyui-controlnet-aux-logo.png

# Q&A:
## Why some nodes doesn't appear after I installed this repo?
Expand Down
1 change: 1 addition & 0 deletions UPDATES.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,4 @@
* Added Depth Anything V2 (16/06/2024)
* Added Union model of ControlNet and preprocessors
![345832280-edf41dab-7619-494c-9f60-60ec1f8789cb](https://github.com/user-attachments/assets/aa55f57c-cad7-48e6-84d3-8f506d847989)
* Refactor INPUT_TYPES and add Execute All node during the process of learning [Execution Model Inversion](https://github.com/comfyanonymous/ComfyUI/pull/2666)
77 changes: 70 additions & 7 deletions __init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
import sys, os
from .utils import here, create_node_input_types
from .utils import here, define_preprocessor_inputs, INPUT
from pathlib import Path
import threading
import traceback
import warnings
import importlib
from .log import log, blue_text, cyan_text, get_summary, get_label
from .hint_image_enchance import NODE_CLASS_MAPPINGS as HIE_NODE_CLASS_MAPPINGS
from .hint_image_enchance import NODE_DISPLAY_NAME_MAPPINGS as HIE_NODE_DISPLAY_NAME_MAPPINGS
#Ref: https://github.com/comfyanonymous/ComfyUI/blob/76d53c4622fc06372975ed2a43ad345935b8a551/nodes.py#L17
sys.path.insert(0, str(Path(here, "src").resolve()))
for pkg_name in ["controlnet_aux", "custom_mmpkg"]:
for pkg_name in ["custom_controlnet_aux", "custom_mmpkg"]:
sys.path.append(str(Path(here, "src", pkg_name).resolve()))

#Enable CPU fallback for ops not being supported by MPS like upsample_bicubic2d.out
Expand Down Expand Up @@ -62,8 +60,10 @@ def load_nodes():

AUX_NODE_MAPPINGS, AUX_DISPLAY_NAME_MAPPINGS = load_nodes()

AIO_NOT_SUPPORTED = ["InpaintPreprocessor"]
#For nodes not mapping image to image
#For nodes not mapping image to image or has special requirements
AIO_NOT_SUPPORTED = ["InpaintPreprocessor", "MeshGraphormer+ImpactDetector-DepthMapPreprocessor", "DiffusionEdge_Preprocessor"]
AIO_NOT_SUPPORTED += ["SavePoseKpsAsJsonFile", "FacialPartColoringFromPoseKps", "UpperBodyTrackingFromPoseKps", "RenderPeopleKps", "RenderAnimalKps"]
AIO_NOT_SUPPORTED += ["Unimatch_OptFlowPreprocessor", "MaskOptFlow"]

def preprocessor_options():
auxs = list(AUX_NODE_MAPPINGS.keys())
Expand All @@ -79,7 +79,10 @@ def preprocessor_options():
class AIO_Preprocessor:
@classmethod
def INPUT_TYPES(s):
return create_node_input_types(preprocessor=(PREPROCESSOR_OPTIONS, {"default": "none"}))
return define_preprocessor_inputs(
preprocessor=INPUT.COMBO(PREPROCESSOR_OPTIONS, default="none"),
resolution=INPUT.RESOLUTION()
)

RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"
Expand Down Expand Up @@ -116,6 +119,63 @@ def execute(self, preprocessor, image, resolution=512):

return getattr(aux_class(), aux_class.FUNCTION)(**params)

class ControlNetAuxSimpleAddText:
@classmethod
def INPUT_TYPES(s):
return dict(
required=dict(image=INPUT.IMAGE(), text=INPUT.STRING())
)

RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"
CATEGORY = "ControlNet Preprocessors"
def execute(self, image, text):
from PIL import Image, ImageDraw, ImageFont
import numpy as np
import torch

font = ImageFont.truetype(str((here / "NotoSans-Regular.ttf").resolve()), 40)
img = Image.fromarray(image[0].cpu().numpy().__mul__(255.).astype(np.uint8))
ImageDraw.Draw(img).text((0,0), text, fill=(0,255,0), font=font)
return (torch.from_numpy(np.array(img)).unsqueeze(0) / 255.,)

class ExecuteAllControlNetPreprocessors:
@classmethod
def INPUT_TYPES(s):
return define_preprocessor_inputs(resolution=INPUT.RESOLUTION())
RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"

CATEGORY = "ControlNet Preprocessors"

def execute(self, image, resolution=512):
try:
from comfy_execution.graph_utils import GraphBuilder
except:
raise RuntimeError("ExecuteAllControlNetPreprocessor requries [Execution Model Inversion](https://github.com/comfyanonymous/ComfyUI/commit/5cfe38). Update ComfyUI/SwarmUI to get this feature")

graph = GraphBuilder()
curr_outputs = []
for preprocc in PREPROCESSOR_OPTIONS:
preprocc_node = graph.node("AIO_Preprocessor", preprocessor=preprocc, image=image, resolution=resolution)
hint_img = preprocc_node.out(0)
add_text_node = graph.node("ControlNetAuxSimpleAddText", image=hint_img, text=preprocc)
curr_outputs.append(add_text_node.out(0))

while len(curr_outputs) > 1:
_outputs = []
for i in range(0, len(curr_outputs), 2):
if i+1 < len(curr_outputs):
image_batch = graph.node("ImageBatch", image1=curr_outputs[i], image2=curr_outputs[i+1])
_outputs.append(image_batch.out(0))
else:
_outputs.append(curr_outputs[i])
curr_outputs = _outputs

return {
"result": (curr_outputs[0],),
"expand": graph.finalize(),
}

class ControlNetPreprocessorSelector:
@classmethod
Expand All @@ -141,11 +201,14 @@ def get_preprocessor(self, preprocessor: str):
"AIO_Preprocessor": AIO_Preprocessor,
"ControlNetPreprocessorSelector": ControlNetPreprocessorSelector,
**HIE_NODE_CLASS_MAPPINGS,
"ExecuteAllControlNetPreprocessors": ExecuteAllControlNetPreprocessors,
"ControlNetAuxSimpleAddText": ControlNetAuxSimpleAddText
}

NODE_DISPLAY_NAME_MAPPINGS = {
**AUX_DISPLAY_NAME_MAPPINGS,
"AIO_Preprocessor": "AIO Aux Preprocessor",
"ControlNetPreprocessorSelector": "Preprocessor Selector",
**HIE_NODE_DISPLAY_NAME_MAPPINGS,
"ExecuteAllControlNetPreprocessors": "Execute All ControlNet Preprocessors"
}
2 changes: 1 addition & 1 deletion dev_interface.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
import sys
sys.path.append(str(Path(here, "src")))

from controlnet_aux import *
from custom_controlnet_aux import *
Binary file added examples/CNAuxBanner.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/ExecuteAll.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/ExecuteAll1.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/ExecuteAll2.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/comfyui-controlnet-aux-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
21 changes: 8 additions & 13 deletions node_wrappers/anime_face_segment.py
Original file line number Diff line number Diff line change
@@ -1,22 +1,17 @@
from ..utils import common_annotator_call, create_node_input_types
from ..utils import common_annotator_call, define_preprocessor_inputs, INPUT
import comfy.model_management as model_management
import torch
from einops import rearrange

class AnimeFace_SemSegPreprocessor:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"image": ("IMAGE",)
},
"optional": {
#This preprocessor is only trained on 512x resolution
#https://github.com/siyeong0/Anime-Face-Segmentation/blob/main/predict.py#L25
"remove_background_using_abg": ("BOOLEAN", {"default": True}),
"resolution": ("INT", {"default": 512, "min": 512, "max": 512, "step": 64})
}
}
#This preprocessor is only trained on 512x resolution
#https://github.com/siyeong0/Anime-Face-Segmentation/blob/main/predict.py#L25
return define_preprocessor_inputs(
remove_background_using_abg=INPUT.BOOLEAN(True),
resolution=INPUT.RESOLUTION(default=512, min=512, max=512)
)

RETURN_TYPES = ("IMAGE", "MASK")
RETURN_NAMES = ("IMAGE", "ABG_CHARACTER_MASK (MASK)")
Expand All @@ -25,7 +20,7 @@ def INPUT_TYPES(s):
CATEGORY = "ControlNet Preprocessors/Semantic Segmentation"

def execute(self, image, remove_background_using_abg=True, resolution=512, **kwargs):
from controlnet_aux.anime_face_segment import AnimeFaceSegmentor
from custom_controlnet_aux.anime_face_segment import AnimeFaceSegmentor

model = AnimeFaceSegmentor.from_pretrained().to(model_management.get_torch_device())
if remove_background_using_abg:
Expand Down
35 changes: 15 additions & 20 deletions node_wrappers/anyline.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import comfy.utils

# Requires comfyui_controlnet_aux funcsions and classes
from ..utils import common_annotator_call, MAX_RESOLUTION
from ..utils import common_annotator_call, INPUT, define_preprocessor_inputs

def get_intensity_mask(image_array, lower_bound, upper_bound):
mask = image_array[:, :, 0]
Expand All @@ -21,19 +21,14 @@ def combine_layers(base_layer, top_layer):
class AnyLinePreprocessor:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"image": ("IMAGE",),
"merge_with_lineart": (["lineart_standard", "lineart_realisitic", "lineart_anime", "manga_line"], {"default": "lineart_standard"}),
"resolution": ("INT", {"default": 1280, "min": 512, "max": MAX_RESOLUTION, "step": 8})
},
"optional": {
"lineart_lower_bound": ("FLOAT", {"default": 0, "min": 0, "max": 1, "step": 0.01}),
"lineart_upper_bound": ("FLOAT", {"default": 1, "min": 0, "max": 1, "step": 0.01}),
"object_min_size": ("INT", {"default": 36, "min": 1, "max": MAX_RESOLUTION}),
"object_connectivity": ("INT", {"default": 1, "min": 1, "max": MAX_RESOLUTION}),
}
}
return define_preprocessor_inputs(
merge_with_lineart=INPUT.COMBO(["lineart_standard", "lineart_realisitic", "lineart_anime", "manga_line"], default="lineart_standard"),
resolution=INPUT.RESOLUTION(default=1280, step=8),
lineart_lower_bound=INPUT.FLOAT(default=0),
lineart_upper_bound=INPUT.FLOAT(default=1),
object_min_size=INPUT.INT(default=36, min=1),
object_connectivity=INPUT.INT(default=1, min=1)
)

RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("image",)
Expand All @@ -44,8 +39,8 @@ def INPUT_TYPES(s):
def __init__(self):
self.device = model_management.get_torch_device()

def get_anyline(self, image, merge_with_lineart, resolution, lineart_lower_bound=0, lineart_upper_bound=1, object_min_size=36, object_connectivity=1):
from controlnet_aux.teed import TEDDetector
def get_anyline(self, image, merge_with_lineart="lineart_standard", resolution=512, lineart_lower_bound=0, lineart_upper_bound=1, object_min_size=36, object_connectivity=1):
from custom_controlnet_aux.teed import TEDDetector
from skimage import morphology
pbar = comfy.utils.ProgressBar(3)

Expand All @@ -58,14 +53,14 @@ def get_anyline(self, image, merge_with_lineart, resolution, lineart_lower_bound

# Process the image with the lineart standard preprocessor
if merge_with_lineart == "lineart_standard":
from controlnet_aux.lineart_standard import LineartStandardDetector
from custom_controlnet_aux.lineart_standard import LineartStandardDetector
lineart_standard_detector = LineartStandardDetector()
lineart_result = common_annotator_call(lineart_standard_detector, image, guassian_sigma=2, intensity_threshold=3, resolution=resolution, show_pbar=False).numpy()
del lineart_standard_detector
else:
from controlnet_aux.lineart import LineartDetector
from controlnet_aux.lineart_anime import LineartAnimeDetector
from controlnet_aux.manga_line import LineartMangaDetector
from custom_controlnet_aux.lineart import LineartDetector
from custom_controlnet_aux.lineart_anime import LineartAnimeDetector
from custom_controlnet_aux.manga_line import LineartMangaDetector
lineart_detector = dict(lineart_realisitic=LineartDetector, lineart_anime=LineartAnimeDetector, manga_line=LineartMangaDetector)[merge_with_lineart]
lineart_detector = lineart_detector.from_pretrained().to(self.device)
lineart_result = common_annotator_call(lineart_detector, image, resolution=resolution, show_pbar=False).numpy()
Expand Down
11 changes: 6 additions & 5 deletions node_wrappers/binary.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
from ..utils import common_annotator_call, create_node_input_types
from ..utils import common_annotator_call, INPUT, define_preprocessor_inputs
import comfy.model_management as model_management

class Binary_Preprocessor:
@classmethod
def INPUT_TYPES(s):
return create_node_input_types(
bin_threshold=("INT", {"default": 100, "min": 0, "max": 255, "step": 1})
return define_preprocessor_inputs(
bin_threshold=INPUT.INT(default=100, max=255),
resolution=INPUT.RESOLUTION()
)

RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"

CATEGORY = "ControlNet Preprocessors/Line Extractors"

def execute(self, image, bin_threshold, resolution=512, **kwargs):
from controlnet_aux.binary import BinaryDetector
def execute(self, image, bin_threshold=100, resolution=512, **kwargs):
from custom_controlnet_aux.binary import BinaryDetector

return (common_annotator_call(BinaryDetector(), image, bin_threshold=bin_threshold, resolution=resolution), )

Expand Down
13 changes: 7 additions & 6 deletions node_wrappers/canny.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
from ..utils import common_annotator_call, create_node_input_types
from ..utils import common_annotator_call, INPUT, define_preprocessor_inputs
import comfy.model_management as model_management

class Canny_Edge_Preprocessor:
@classmethod
def INPUT_TYPES(s):
return create_node_input_types(
low_threshold=("INT", {"default": 100, "min": 0, "max": 255, "step": 1}),
high_threshold=("INT", {"default": 200, "min": 0, "max": 255, "step": 1})
return define_preprocessor_inputs(
low_threshold=INPUT.INT(default=100, max=255),
high_threshold=INPUT.INT(default=200, max=255),
resolution=INPUT.RESOLUTION()
)

RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"

CATEGORY = "ControlNet Preprocessors/Line Extractors"

def execute(self, image, low_threshold, high_threshold, resolution=512, **kwargs):
from controlnet_aux.canny import CannyDetector
def execute(self, image, low_threshold=100, high_threshold=200, resolution=512, **kwargs):
from custom_controlnet_aux.canny import CannyDetector

return (common_annotator_call(CannyDetector(), image, low_threshold=low_threshold, high_threshold=high_threshold, resolution=resolution), )

Expand Down
Loading

0 comments on commit 648a5d7

Please sign in to comment.