Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: sdbds/magic-animate-for-windows
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 1.0
Choose a base ref
...
head repository: sdbds/magic-animate-for-windows
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Dec 5, 2023

  1. debug mode

    cocktailpeanut committed Dec 5, 2023
    Copy the full SHA
    d734e60 View commit details
  2. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    318a14b View commit details

Commits on Dec 6, 2023

  1. add install option

    sdbds committed Dec 6, 2023
    Copy the full SHA
    c8cce8a View commit details
  2. Update README.md

    sdbds committed Dec 6, 2023
    Copy the full SHA
    58aa5c3 View commit details
  3. Update README.md

    sdbds committed Dec 6, 2023
    Copy the full SHA
    a8999f1 View commit details
  4. add submodule

    sdbds committed Dec 6, 2023
    Copy the full SHA
    6fe2f31 View commit details
  5. add vid2pose

    sdbds committed Dec 6, 2023
    Copy the full SHA
    a0d2d7b View commit details
  6. add video2openpose

    sdbds committed Dec 6, 2023
    Copy the full SHA
    12315e4 View commit details

Commits on Dec 7, 2023

  1. disable queue

    zcxu-eric committed Dec 7, 2023
    Copy the full SHA
    5cde154 View commit details
  2. change demo img

    zcxu-eric committed Dec 7, 2023
    Copy the full SHA
    c6c3b19 View commit details
  3. change demo img

    zcxu-eric committed Dec 7, 2023
    Copy the full SHA
    32484c1 View commit details

Commits on Dec 8, 2023

  1. update README

    zcxu-eric committed Dec 8, 2023
    Copy the full SHA
    5f969d8 View commit details
  2. add video_controlnet_aux

    sdbds committed Dec 8, 2023
    Copy the full SHA
    ffef1e4 View commit details

Commits on Dec 9, 2023

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    d61200a View commit details
  2. update magic anime

    sdbds committed Dec 9, 2023
    Copy the full SHA
    e1416b5 View commit details
  3. 2
    Copy the full SHA
    8657110 View commit details
  4. Update gradio_animate.py

    sdbds committed Dec 9, 2023
    Copy the full SHA
    4c8f9f5 View commit details
  5. update for install cn

    sdbds committed Dec 9, 2023
    Copy the full SHA
    525b176 View commit details

Commits on Dec 12, 2023

  1. Update README.md

    sdbds authored Dec 12, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    39d7fad View commit details
  2. Copy the full SHA
    e83ea98 View commit details

Commits on Dec 19, 2023

  1. add lfs clone

    sdbds committed Dec 19, 2023
    Copy the full SHA
    bc45ca2 View commit details

Commits on Jan 13, 2024

  1. Update README.md

    sdbds authored Jan 13, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    bc1dc07 View commit details
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -4,11 +4,13 @@ samples
xformers
src
third_party
backup
pretrained_models
*.nfs*
./*.png
./*.mp4
demo/tmp
demo/outputs
huggingface/
venv/
venv/
outputs/
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "video_controlnet_aux"]
path = video_controlnet_aux
url = https://github.com/sdbds/video_controlnet_aux
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -30,24 +30,18 @@

<table align="center">
<tr>
<td>
<img src="assets/teaser/t1.gif">
</td>
<td>
<img src="assets/teaser/t4.gif">
</td>
</tr>
<tr>
<td>
<img src="assets/teaser/t3.gif">
</td>
<td>
<img src="assets/teaser/t2.gif">
</td>
</tr>
</table>

## 📢 News
* **[2023.12.8]** Add loading local safetensors or ckpt,you can change `config/prompts/animation.yaml` about `pretrained_model_path` for your local SD1.5 model.
such as `"D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"`
* **[2023.12.4]** Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
* **[2023.11.23]** Release MagicAnimate paper and project page.

@@ -71,8 +65,17 @@ Give unrestricted script access to powershell so venv can work:
- Type `Set-ExecutionPolicy Unrestricted` and answer A
- Close admin powershell window

```
git clone --recurse-submodules https://github.com/sdbds/magic-animate-for-windows/
```

Install with Powershell run `install.ps1` or `install-cn.ps1`(for Chinese)

### Use local model

Add loading local safetensors or ckpt,you can change `config/prompts/animation.yaml` about `pretrained_model_path` for your local SD1.5 model.
such as `"D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"`

## 🎨 Gradio Demo

#### Online Gradio Demo:
Binary file removed assets/teaser/t1.gif
Binary file not shown.
Binary file removed assets/teaser/t3.gif
Binary file not shown.
3 changes: 3 additions & 0 deletions configs/prompts/animation.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
pretrained_model_path: "pretrained_models/stable-diffusion-v1-5"
pretrained_vae_path: ""
pretrained_controlnet_path: "pretrained_models/MagicAnimate/densepose_controlnet"
openpose_path: "pretrained_models/control_v11p_sd15_openpose"
pretrained_appearance_encoder_path: "pretrained_models/MagicAnimate/appearance_encoder"
pretrained_unet_path: ""

@@ -40,3 +41,5 @@ max_length: null
video_type: "condition"
invert_video: false
save_individual_videos: false

openpose: false
106 changes: 75 additions & 31 deletions demo/animate.py
Original file line number Diff line number Diff line change
@@ -29,7 +29,7 @@
from magicanimate.models.controlnet import ControlNetModel
from magicanimate.models.appearance_encoder import AppearanceEncoderModel
from magicanimate.models.mutual_self_attention import ReferenceAttentionControl
from magicanimate.models.model_util import load_models
from magicanimate.models.model_util import load_models, torch_gc
from magicanimate.pipelines.pipeline_animation import AnimationPipeline
from magicanimate.utils.util import save_videos_grid
from accelerate.utils import set_seed
@@ -43,20 +43,33 @@
import math
from pathlib import Path


class MagicAnimate:
def __init__(self, config="configs/prompts/animation.yaml") -> None:
def __init__(
self, config="configs/prompts/animation.yaml", controlnet_model="densepose"
) -> None:
print("Initializing MagicAnimate Pipeline...")
*_, func_args = inspect.getargvalues(inspect.currentframe())
func_args = dict(func_args)

self.config = config

config = OmegaConf.load(config)

inference_config = OmegaConf.load(config.inference_config)

motion_module = config.motion_module

self.controlnet_model = controlnet_model

### >>> create animation pipeline >>> ###
tokenizer, text_encoder, unet, noise_scheduler, vae = load_models(
(
self.tokenizer,
self.text_encoder,
self.unet,
noise_scheduler,
self.vae,
) = load_models(
config.pretrained_model_path,
scheduler_name="",
v2=False,
@@ -69,15 +82,15 @@ def __init__(self, config="configs/prompts/animation.yaml") -> None:
# config.pretrained_model_path, subfolder="text_encoder"
# )
if config.pretrained_unet_path:
unet = UNet3DConditionModel.from_pretrained_2d(
self.unet = UNet3DConditionModel.from_pretrained_2d(
config.pretrained_unet_path,
unet_additional_kwargs=OmegaConf.to_container(
inference_config.unet_additional_kwargs
),
)
else:
unet = UNet3DConditionModel.from_pretrained_2d(
unet.config,
self.unet = UNet3DConditionModel.from_pretrained_2d(
self.unet.config,
subfolder=None,
unet_additional_kwargs=OmegaConf.to_container(
inference_config.unet_additional_kwargs
@@ -93,38 +106,45 @@ def __init__(self, config="configs/prompts/animation.yaml") -> None:
fusion_blocks=config.fusion_blocks,
)
self.reference_control_reader = ReferenceAttentionControl(
unet,
self.unet,
do_classifier_free_guidance=True,
mode="read",
fusion_blocks=config.fusion_blocks,
)

if config.pretrained_vae_path:
vae = AutoencoderKL.from_pretrained(config.pretrained_vae_path)
self.vae = AutoencoderKL.from_pretrained(config.pretrained_vae_path)
# else:
# vae = AutoencoderKL.from_pretrained(
# config.pretrained_model_path, subfolder="vae"
# )

### Load controlnet
controlnet = ControlNetModel.from_pretrained(config.pretrained_controlnet_path)
if "openpose" in self.controlnet_model:
self.controlnet = ControlNetModel.from_pretrained(config.openpose_path)
print("Using OpenPose ControlNet")
else:
self.controlnet = ControlNetModel.from_pretrained(
config.pretrained_controlnet_path
)
print("Using Densepose ControlNet")

vae.to(torch.float16)
unet.to(torch.float16)
text_encoder.to(torch.float16)
controlnet.to(torch.float16)
self.vae.to(torch.float16)
self.unet.to(torch.float16)
self.text_encoder.to(torch.float16)
self.controlnet.to(torch.float16)
self.appearance_encoder.to(torch.float16)

unet.enable_xformers_memory_efficient_attention()
self.unet.enable_xformers_memory_efficient_attention()
self.appearance_encoder.enable_xformers_memory_efficient_attention()
controlnet.enable_xformers_memory_efficient_attention()
self.controlnet.enable_xformers_memory_efficient_attention()

self.pipeline = AnimationPipeline(
vae=vae,
text_encoder=text_encoder,
tokenizer=tokenizer,
unet=unet,
controlnet=controlnet,
vae=self.vae,
text_encoder=self.text_encoder,
tokenizer=self.tokenizer,
unet=self.unet,
controlnet=self.controlnet,
scheduler=DDIMScheduler(
**OmegaConf.to_container(inference_config.noise_scheduler_kwargs)
),
@@ -165,7 +185,7 @@ def __init__(self, config="configs/prompts/animation.yaml") -> None:
_tmp_[_key] = motion_module_state_dict[key]
else:
_tmp_[key] = motion_module_state_dict[key]
missing, unexpected = unet.load_state_dict(_tmp_, strict=False)
missing, unexpected = self.unet.load_state_dict(_tmp_, strict=False)
assert len(unexpected) == 0
del _tmp_
del motion_module_state_dict
@@ -175,9 +195,31 @@ def __init__(self, config="configs/prompts/animation.yaml") -> None:

print("Initialization Done!")

def reset_init(instance, *args, **kwargs):
instance.__init__(*args, **kwargs)

def __call__(
self, source_image, motion_sequence, random_seed, step, guidance_scale, size=512
self,
source_image,
motion_sequence,
random_seed,
step,
guidance_scale,
debug,
controlnet_model="densepose",
size=512,
):
if self.controlnet_model != controlnet_model:
self.vae.to("cpu")
self.unet.to("cpu")
self.text_encoder.to("cpu")
self.controlnet.to("cpu")
self.appearance_encoder.to("cpu")
torch_gc()
self.reset_init(
config="configs/prompts/animation.yaml",
controlnet_model=controlnet_model,
)
prompt = n_prompt = ""
random_seed = int(random_seed)
step = int(step)
@@ -229,16 +271,18 @@ def __call__(
source_image=source_image,
).videos

source_images = np.array([source_image] * original_length)
source_images = (
rearrange(torch.from_numpy(source_images), "t h w c -> 1 c t h w") / 255.0
)
samples_per_video.append(source_images)
if debug:
source_images = np.array([source_image] * original_length)
source_images = (
rearrange(torch.from_numpy(source_images), "t h w c -> 1 c t h w")
/ 255.0
)
samples_per_video.append(source_images)

control = control / 255.0
control = rearrange(control, "t h w c -> 1 c t h w")
control = torch.from_numpy(control)
samples_per_video.append(control[:, :, :original_length])
control = control / 255.0
control = rearrange(control, "t h w c -> 1 c t h w")
control = torch.from_numpy(control)
samples_per_video.append(control[:, :, :original_length])

samples_per_video.append(sample[:, :, :original_length])

Loading