From 56c454cab4a3db6909d9559bce123f64981b9635 Mon Sep 17 00:00:00 2001 From: Tim Pietrusky Date: Mon, 19 Aug 2024 13:27:14 +0200 Subject: [PATCH] docs: added models --- README.md | 54 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index f8311dd..78e8b3f 100644 --- a/README.md +++ b/README.md @@ -49,15 +49,14 @@ Read our article here: https://blib.la/blog/comfyui-on-runpod ## Quickstart -- 🐳 Choose one of the three available images for your serverless endpoint: - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-base`: doesn't contain any checkpoints, just a clean ComfyUI - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-sdxl`: contains the checkpoints and VAE for Stable Diffusion XL - - Checkpoint: [sd_xl_base_1.0.safetensors](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) - - VAEs: - - [sdxl_vae.safetensors](https://huggingface.co/stabilityai/sdxl-vae/) - - [sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/) - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-sd3`: contains the [sd3_medium_incl_clips_t5xxlfp8.safetensors](https://huggingface.co/stabilityai/stable-diffusion-3-medium) checkpoint for Stable Diffusion 3 +- 🐳 Choose one of the five available images for your serverless endpoint: + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-base`: doesn't contain anything, just a clean ComfyUI + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-flux1-schnell`: contains the checkpoint, text encoders and VAE for [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell) + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-flux1-dev`: contains the checkpoint, text encoders and VAE for [FLUX.1 dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-sdxl`: contains the checkpoint and VAE for [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-sd3`: contains the checkpoint for [Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium) - ℹ️ [Use the Docker image on RunPod](#use-the-docker-image-on-runpod) +- 🧪 Pick an [example workflow](./test_resources/workflows/) & [send it to your deployed endpoint](#interact-with-your-runpod-api) ## Features @@ -66,10 +65,15 @@ Read our article here: https://blib.la/blog/comfyui-on-runpod - The generated image is either: - Returned as base64-encoded string (default) - Uploaded to AWS S3 ([if AWS S3 is configured](#upload-image-to-aws-s3)) -- There are three different Docker images to choose from: - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-base`: doesn't contain anything, just a clean ComfyUI - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-sdxl`: contains the checkpoint and VAE for Stable Diffusion XL - - `timpietruskyblibla/runpod-worker-comfy:3.0.0-sd3`: contains the checkpoint for Stable Diffusion 3 +- There are a few different Docker images to choose from: + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-flux1-schnell`: contains the [flux1-schnell.safetensors](https://huggingface.co/black-forest-labs/FLUX.1-schnell) checkpoint, the [clip_l.safetensors](https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors) + [t5xxl_fp8_e4m3fn.safetensors](https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp8_e4m3fn.safetensors) text encoders and [ae.safetensors](https://huggingface.co/black-forest-labs/FLUX.1-schnell/resolve/main/ae.safetensors) VAE for FLUX.1-schnell + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-flux1-dev`: contains the [flux1-dev.safetensors](https://huggingface.co/black-forest-labs/FLUX.1-dev) checkpoint, the [clip_l.safetensors](https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors) + [t5xxl_fp8_e4m3fn.safetensors](https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp8_e4m3fn.safetensors) text encoders and [ae.safetensors](https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/ae.safetensors) VAE for FLUX.1-dev + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-sdxl`: contains the checkpoints and VAE for Stable Diffusion XL + - Checkpoint: [sd_xl_base_1.0.safetensors](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) + - VAEs: + - [sdxl_vae.safetensors](https://huggingface.co/stabilityai/sdxl-vae/) + - [sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/) + - `timpietruskyblibla/runpod-worker-comfy:3.1.0-sd3`: contains the [sd3_medium_incl_clips_t5xxlfp8.safetensors](https://huggingface.co/stabilityai/stable-diffusion-3-medium) checkpoint for Stable Diffusion 3 medium - [Bring your own models](#bring-your-own-models) - Based on [Ubuntu + NVIDIA CUDA](https://hub.docker.com/r/nvidia/cuda) @@ -99,30 +103,46 @@ This is only needed if you want to upload the generated picture to AWS S3. If yo ## Use the Docker image on RunPod +### Create your template (optional) + - Create a [new template](https://runpod.io/console/serverless/user/templates) by clicking on `New Template` - In the dialog, configure: - Template Name: `runpod-worker-comfy` (it can be anything you want) - Template Type: serverless (change template type to "serverless") - - Container Image: `/:tag`, in this case: `timpietruskyblibla/runpod-worker-comfy:3.0.0-sd3` (or `-base` for a clean image or `-sdxl` for Stable Diffusion XL) + - Container Image: `/:tag`, in this case: `timpietruskyblibla/runpod-worker-comfy:3.1.0-sd3` (or `-base` for a clean image or `-sdxl` for Stable Diffusion XL or `-flex1-schnell` for FLUX.1 schnell) - Container Registry Credentials: You can leave everything as it is, as this repo is public - Container Disk: `20 GB` - (optional) Environment Variables: [Configure S3](#upload-image-to-aws-s3) - Note: You can also not configure it, the images will then stay in the worker. In order to have them stored permanently, [we have to add the network volume](https://github.com/blib-la/runpod-worker-comfy/issues/1) - Click on `Save Template` + +### Create your endpoint + - Navigate to [`Serverless > Endpoints`](https://www.runpod.io/console/serverless/user/endpoints) and click on `New Endpoint` - In the dialog, configure: + - Endpoint Name: `comfy` - - Select Template: `runpod-worker-comfy` (or whatever name you gave your template) + - Worker configuration: Select a GPU that can run the model you have chosen (see [GPU recommendations](#gpu-recommendations)) - Active Workers: `0` (whatever makes sense for you) - Max Workers: `3` (whatever makes sense for you) + - GPUs/Worker: `1` - Idle Timeout: `5` (you can leave the default) - Flash Boot: `enabled` (doesn't cost more, but provides faster boot of our worker, which is good) + - Select Template: `runpod-worker-comfy` (or whatever name you gave your template) - (optional) Advanced: If you are using a Network Volume, select it under `Select Network Volume`. Otherwise leave the defaults. - - Select a GPU that has some availability - - GPUs/Worker: `1` + - Click `deploy` - Your endpoint will be created, you can click on it to see the dashboard +### GPU recommendations + +| Model | Image | Minimum VRAM Required | Container Size | +| ------------------------- | --------------- | --------------------- | -------------- | +| Stable Diffusion XL | `sdxl` | 8 GB | 15 GB | +| Stable Diffusion 3 Medium | `sd3` | 5 GB | 20 GB | +| FLUX.1 Schnell | `flux1-schnell` | 24 GB | 30 GB | +| FLUX.1 dev | `flux1-dev` | 24 GB | 30 GB | + ## API specification The following describes which fields exist when doing requests to the API. We only describe the fields that are sent via `input` as those are needed by the worker itself. For a full list of fields, please take a look at the [official documentation](https://docs.runpod.io/docs/serverless-usage). @@ -193,7 +213,7 @@ The API expects a [JSON in this form](#json-request-body), where `workflow` is t Please also take a look at the [test_input.json](./test_input.json) to see how the API input should look like. -#### Example request with cURL +#### Example request for SDXL with cURL ```bash curl -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"input":{"workflow":{"3":{"inputs":{"seed":1337,"steps":20,"cfg":8,"sampler_name":"euler","scheduler":"normal","denoise":1,"model":["4",0],"positive":["6",0],"negative":["7",0],"latent_image":["5",0]},"class_type":"KSampler"},"4":{"inputs":{"ckpt_name":"sd_xl_base_1.0.safetensors"},"class_type":"CheckpointLoaderSimple"},"5":{"inputs":{"width":512,"height":512,"batch_size":1},"class_type":"EmptyLatentImage"},"6":{"inputs":{"text":"beautiful scenery nature glass bottle landscape, purple galaxy bottle,","clip":["4",1]},"class_type":"CLIPTextEncode"},"7":{"inputs":{"text":"text, watermark","clip":["4",1]},"class_type":"CLIPTextEncode"},"8":{"inputs":{"samples":["3",0],"vae":["4",2]},"class_type":"VAEDecode"},"9":{"inputs":{"filename_prefix":"ComfyUI","images":["8",0]},"class_type":"SaveImage"}}}}' https://api.runpod.ai/v2//runsync