Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Added Support of Xlabs controlnet to FluxControlNetInpaintPipeline #9770

Merged
merged 6 commits into from
Oct 25, 2024

Conversation

SahilCarterr
Copy link
Contributor

What does this PR do?

Fixes #9733 Adds Support of Xlabs controlnet to FluxControlNetInpaintPipeline

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed.
@yiyixuxu


Example

import torch
from diffusers import FluxControlNetInpaintPipeline
from diffusers.models import FluxControlNetModel
from diffusers.utils import load_image

controlnet = FluxControlNetModel.from_pretrained(
  "Xlabs-AI/flux-controlnet-canny-diffusers",
  torch_dtype=torch.float16,
  use_safetensors=True,
)

pipe = FluxControlNetInpaintPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", controlnet=controlnet, torch_dtype=torch.float16
)

pipe.to("cuda")

prompt = "A woman wearing a red dress with the text 'Flux' on it and cat print"

image = pipe(
    prompt,
    image=init_image,
    mask_image=mask_image,
    control_image=canny_image,
    height=960,
    width=640,
    num_inference_steps=50,
    strength=0.8,
    guidance_scale=6.5,
    controlnet_conditioning_scale=0.7,
    control_guidance_start=0.2,
    control_guidance_end=0.8,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
Input Image Mask Image Canny Image Output Image
girl_8 maskofgirl_8 canny_example_girl8 girl_08_output

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@yiyixuxu yiyixuxu merged commit 298ab6e into huggingface:main Oct 25, 2024
13 of 15 checks passed
@SahilCarterr SahilCarterr deleted the flux_controlnet_inpaint branch October 26, 2024 03:12
@SahilCarterr SahilCarterr restored the flux_controlnet_inpaint branch October 26, 2024 03:12
@SahilCarterr SahilCarterr deleted the flux_controlnet_inpaint branch October 26, 2024 03:13
sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Xlabs controlnets do not work with FluxControlNetInpaintPipeline
3 participants