Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875

juancopi81 · 2024-07-17T01:38:08Z

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD 1.5 with controlnet and inpainting models (StableDiffusionControlNetPAGInpaintPipeline )
Continuation of #8710. It was not mentioned on #8710 but I think this pipeline is pretty cool also 😄

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@a-r-r-o-w @yiyixuxu
Anyone in the community is free to review the PR once the tests have passed.

Code example

import cv2
from diffusers import AutoPipelineForInpainting, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image
import numpy as np
from PIL import Image
import torch

init_image = load_image("https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy.png")
init_image = init_image.resize((512, 512))

mask_image = load_image("https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy_mask.png")
mask_image = mask_image.resize((512, 512))

def make_canny_condition(image):
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image

control_image = make_canny_condition(init_image)

controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16)

pipe = AutoPipelineForInpainting.from_pretrained(
    "jayparmr/icbinp_v8_inpaint_v2", controlnet=controlnet, torch_dtype=torch.float16, enable_pag=True, requires_safety_checker=False, safety_checker=None,
)

pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

pag_scales =  [0.0, 3.0]
guidance_scales = [0.0, 2.0]

grid = []
for pag_scale in pag_scales:
    for guidance_scale in guidance_scales:
        generator = torch.Generator(device="cpu").manual_seed(0)
        # generate image
        image = pipe(
            "a man with ray-ban sunglasses",
            negative_prompt="low quality, bad quality",
            num_inference_steps=20,
            generator=generator,
            eta=1.0,
            image=init_image,
            mask_image=mask_image,
            control_image=control_image,
            pag_scale=pag_scale,
            strength=0.99,
            guidance_scale=guidance_scale,
            controlnet_conditioning_scale=0.2
        ).images
        grid.append(image[0])

from diffusers.utils import make_image_grid
make_image_grid(grid, rows=len(pag_scales), cols=len(guidance_scales))

… in the docs

…ipeline

…et==9

…nnels

…odel_input

HuggingFaceDocBuilderDev · 2024-07-17T23:07:07Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

a-r-r-o-w

Thanks for working on this! The PR looks absolutely perfect with all required PAG changes. I think you need to run make style for the CI to go green. @yiyixuxu WDYT?

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd_inpaint.py

a-r-r-o-w · 2024-07-19T20:13:05Z

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd_inpaint.py

+        control_image = control_images if isinstance(control_image, list) else control_images[0]
+        controlnet_prompt_embeds = prompt_embeds
+
+        # 7.2 Create tensor stating which controlnets to keep


The numbering here onwards seems incorrect

Oh you're right! Changing it...

a-r-r-o-w · 2024-07-19T20:14:21Z

src/diffusers/pipelines/pag/pipeline_pag_controlnet_sd_inpaint.py

+        mask_image,
+        height,
+        width,
+        callback_steps,


I think you can remove this parameter completely instead of passing None

Oh sure! I just removed it

…ality

juancopi81 · 2024-07-20T00:08:48Z

Hi @a-r-r-o-w ,

Thank you very much for your feedback 🚀 ! I just updated my PR trying to address all the issues.

Thanks again.

juancopi81 · 2024-07-20T15:02:41Z

Sorry I had a small mistake that made some tests failed. It should be solved now!

yiyixuxu

thanks! super cool PR! I left one comment,
also, can we make sure it work with from_pipe()?

yiyixuxu · 2024-07-23T03:14:35Z

src/diffusers/pipelines/auto_pipeline.py

@@ -944,7 +946,10 @@ def from_pretrained(cls, pretrained_model_or_path, **kwargs):
        if "enable_pag" in kwargs:
            enable_pag = kwargs.pop("enable_pag")
            if enable_pag:
-                orig_class_name = config["_class_name"].replace("Pipeline", "PAGPipeline")
+                if "controlnet" in kwargs:


can you provide an example where this code addition is needed? I think it should already be addressed in line 944 no?

Hi @yiyixuxu,

Thank you very much for your comments. Sorry if I am missing something here, but my idea is as follows:
There are 4 cases that need to be handled:

No ControlNet and no Pag -> Keep Pipeline

With ControlNet and no Pag -> Replace Pipeline with ControlNetPipeline

No ControlNet and with Pag -> Replace Pipeline with PAGPipeline

With ControlNet and with Pag -> Replace Pipeline with ControlNetPAGPipeline, which will override the first case.

That's why I added the nested conditions, but maybe I am missing something 🤔

i see! thank for explaining it, you were absolute ly right!

can we do this instead (to be consistent with AutoPipelineForText2Image and AutoPipelineForImage2Image)

if "controlnet" in kwargs: orig_class_name = config["_class_name"].replace("Pipeline", "ControlNetPipeline") if "enable_pag" in kwargs: enable_pag = kwargs.pop("enable_pag") if enable_pag: orig_class_name = orig_class_name.replace("Pipeline", "PAGPipeline")

diffusers/src/diffusers/pipelines/auto_pipeline.py

Line 363 in 73acebb

orig_class_name = orig_class_name.replace("Pipeline", "PAGPipeline")

Oh sure, done! This way is better!

…with 'from_pipe'

juancopi81 · 2024-07-24T00:21:45Z

thanks! super cool PR! I left one comment, also, can we make sure it work with from_pipe()?

Hi @yiyixuxu,

Thanks for your support and feedback. I updated my PR with a new test. I run it with:

tests/pipelines/test_pipelines_auto.py::AutoPipelineFastTest::test_from_pipe_pag_controlnet_inpaint

and it seems to be working fine. How do you like it? Would be enough for testing that it works with from_pipe()?

Please let me know 😃

yiyixuxu

very nice! I left some comments for the test, since all the from_pipe tests are based on SDXL, if you want to skip it, it is ok too! Just let us know!

Let's merge this soon

yiyixuxu · 2024-07-27T07:26:02Z

tests/pipelines/test_pipelines_auto.py

@@ -283,6 +283,26 @@ def test_from_pipe_pag_inpaint(self):
        pipe = AutoPipelineForInpainting.from_pipe(pipe_pag, enable_pag=False)
        assert pipe.__class__.__name__ == "StableDiffusionXLInpaintPipeline"

+    def test_from_pipe_pag_controlnet_inpaint(self):


nice! we can combine this test with test_from_pipe_pag_inpaint and make sure it covers all the scenarios similarly as in https://github.com/huggingface/diffusers/blob/main/tests/pipelines/test_pipelines_auto.py#L141

test_from_pipe_pag_inpaint test for the inpainting XL pipeline, we can update to test SD pipeline instead because we now have a controlnet + inpaint + PAG for SD but not for SDXL

make sure :

we test use from_pipe on inpainting pipeline, inpainting controlnet pipeline and pag inpainting controlnet

for each, make sure we test the combination of enable_pag and controlnet`

but if you want to skip the test, it's ok too! we can update the test when we have the sdxl pag inpainting pipeline as well

Thanks @yiyixuxu! I'm short on time right now, so I removed it. Would this work for now? I can come back later to finish it properly, but I wanted to get the PR merged sooner. Please let me know what you think! 😄

…ag are present based on yiyixuxu feedback

juancopi81 · 2024-07-29T22:20:29Z

Thanks a lot @yiyixuxu for your feedback. I addressed your comment in src/diffusers/pipelines/auto_pipeline.py, and remove for now the test I added so the PR can be merged. I can come back later to finish it properly or if someone want to take it, please let me know!

github-actions · 2024-09-29T15:04:11Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

yiyixuxu

thanks so much for the PR
and very sorry I forgot about it and only merge it now

juancopi81 · 2024-10-01T13:44:05Z

No worries @yiyixuxu! Thank you very much for the support. I am very happy that now is merged 🚀

…ngface#8875) * Add pag to controlnet inpainting pipeline --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

* Add pag to controlnet inpainting pipeline --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

juancopi81 added 13 commits July 16, 2024 20:19

Add pag to controlnet inpainting pipeline

5885d74

Fix minor but with timeste_cond

2c11062

Add code to dummy_torch_and_transformers_objects and use autopipeline…

6267102

… in the docs

Fix logic in auto_pipeline when pag and controlnet are present

d152e16

Add some debug prints

d4093e0

Change replacement logic for now so it can take ControlNetPAGInpaintP…

3f502f1

…ipeline

Add check of shapes when unet channels == 9

f6a98cd

Remove some print statements, add some debug prints to shapes when un…

7d1a260

…et==9

Expand mask and masked_image_latents if necessary when unet has 9 cha…

088f0c8

…nnels

Adjust repeat factor so mask, and masked_image_latents match latent_m…

bb01aba

…odel_input

Concatenate tensors when unet has 9 channels

102ce9d

Add tests to pag controlnet sd inpaint pipeline

f9e4543

Make style changes

512a252

a-r-r-o-w approved these changes Jul 19, 2024

View reviewed changes

Final changes based on @a-r-r-o-w feedback and run make style make qu…

ce28845

…ality

Fix incorrect comment for message 'Copied from ...'

00f8c7a

yiyixuxu reviewed Jul 23, 2024

View reviewed changes

Add test to ensure StableDiffusionControlNetPAGInpaintPipeline works …

f7a183e

…with 'from_pipe'

yiyixuxu reviewed Jul 27, 2024

View reviewed changes

Remove cn_inpainting test for now and change logic if cn and enable_p…

93fca2d

…ag are present based on yiyixuxu feedback

yiyixuxu added the PAG label Sep 4, 2024

Merge branch 'main' into pag_controlnet_inpaint_sd15

2fb4fd4

yiyixuxu mentioned this pull request Sep 7, 2024

Add PAG support to SD1.5 #8710

Closed

6 tasks

github-actions bot added the stale label Sep 29, 2024

yiyixuxu removed the stale label Sep 30, 2024

yiyixuxu added 3 commits September 30, 2024 19:12

Merge branch 'main' into pag_controlnet_inpaint_sd15

d5214db

style

7b2368b

loraloadermixin -> stablediffusionloraloadermixin

669052a

yiyixuxu approved these changes Oct 1, 2024

View reviewed changes

yiyixuxu merged commit 33fafe3 into huggingface:main Oct 1, 2024
15 checks passed

juancopi81 deleted the pag_controlnet_inpaint_sd15 branch October 1, 2024 13:42

leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline (huggi…

deba573

…ngface#8875) * Add pag to controlnet inpainting pipeline --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

sayakpaul pushed a commit that referenced this pull request Dec 23, 2024

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline (#8875)

91360de

* Add pag to controlnet inpainting pipeline --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875

juancopi81 commented Jul 17, 2024

HuggingFaceDocBuilderDev commented Jul 17, 2024

a-r-r-o-w left a comment

a-r-r-o-w Jul 19, 2024

juancopi81 Jul 19, 2024

a-r-r-o-w Jul 19, 2024

juancopi81 Jul 19, 2024

juancopi81 commented Jul 20, 2024

juancopi81 commented Jul 20, 2024

yiyixuxu left a comment •

edited

Loading

yiyixuxu Jul 23, 2024

juancopi81 Jul 23, 2024

yiyixuxu Jul 27, 2024

juancopi81 Jul 29, 2024

juancopi81 commented Jul 24, 2024

yiyixuxu left a comment

yiyixuxu Jul 27, 2024

juancopi81 Jul 29, 2024

juancopi81 commented Jul 29, 2024

github-actions bot commented Sep 29, 2024

yiyixuxu left a comment

juancopi81 commented Oct 1, 2024

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875

Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875

Conversation

juancopi81 commented Jul 17, 2024

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Jul 17, 2024

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juancopi81 commented Jul 20, 2024

juancopi81 commented Jul 20, 2024

yiyixuxu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juancopi81 commented Jul 24, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juancopi81 commented Jul 29, 2024

github-actions bot commented Sep 29, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

juancopi81 commented Oct 1, 2024

yiyixuxu left a comment •

edited

Loading