enable multiple prompts for T2I #125
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rickstaa
reviewed
Jul 17, 2024
There was a problem hiding this comment.
@ad-astra-video thanks for implementing this so quickly. I just updated the code a bit and am now testing to see if the logic works with all models without throwing errors. After that we can merge 🚀.
rickstaa
approved these changes
Jul 17, 2024
This commit ensures that the prompt splitting logic implemented in the previous commits can be reused in multiple pipelines.
rickstaa
force-pushed
the
av-enable-multiple-prompts
branch
from
e021b8a to
3b19060
Compare
eliteprox
pushed a commit
to eliteprox/ai-worker
that referenced
this pull request
Jul 26, 2024
* enable multiple prompts for T2I * refactor(runner): make prompt splitting more general This commit ensures that the prompt splitting logic implemented in the previous commits can be reused in multiple pipelines. --------- Co-authored-by: Rick Staa <[email protected]>
eliteprox
pushed a commit
to eliteprox/ai-worker
that referenced
this pull request
Jul 26, 2024
* enable multiple prompts for T2I * refactor(runner): make prompt splitting more general This commit ensures that the prompt splitting logic implemented in the previous commits can be reused in multiple pipelines. --------- Co-authored-by: Rick Staa <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables requests to use multiple prompts supported by SDXL and SD3 with no API updates needed. Diffusers uses the same prompt in all text_encoders if the prompts are not split. Multiple prompts is supported by SDXL and SD3 models and can benefit from specializing the prompts to the different text_ecnoders if tuned properly.
The prompts are split by including a
|at the point where want the prompts split.For SDXL see diffusers pipeline documentation here:https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl
Specifically the
prompt_2input to the__call__: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl#diffusers.StableDiffusionXLPipeline.__call__For SD3 see diffusers pipeline documentation here: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_3
Specifically the
prompt_2andprompt_3inputs to the__call__: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_3#diffusers.StableDiffusion3Pipeline.__call__