Skip to content

Add community class StableDiffusionXL_T5Pipeline #11626

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 7 commits into from
Jun 9, 2025

Conversation

ppbrown
Copy link
Contributor

@ppbrown ppbrown commented May 29, 2025

What does this PR do?

This adds a new community pipeline, named
StableDiffusionXL_T5Pipeline

It grafts the T5 xxl encoder onto SDXL, completely replacing TE1 and TE2.

It has been tested to produce output. However, the output isnt particularly valid.
The unet needs to be retrained.

Will initially be used with base model opendiffusionai/stablediffusionxl_t5
until a proper (re)fine-tune can be made.

Fixes # (issue)

Before submitting

Who can review?

Will be used with base model opendiffusionai/stablediffusionxl_t5
@ppbrown
Copy link
Contributor Author

ppbrown commented May 29, 2025

PS: This code was tested locally and ran, using the following test harness:

from diffusers import DiffusionPipeline
import torch.nn as nn, torch, types

SDXL_DIR = "/home/phil/git/models/t5-sdxl-model"

LOCAL_CODE = "/home/phil/git/diffusers.t5/examples/community/"

pipe = DiffusionPipeline.from_pretrained(
    SDXL_DIR, custom_pipeline=LOCAL_CODE, use_safetensors=True,
    torch_dtype=torch.bfloat16,
)

print("model initialized. Now moving to CUDA")
pipe.to("cuda")

print("Trying render now...")

images = pipe("a misty Tokyo alley at night",num_inference_steps=30).images
images[0].save("test.png")

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

@bot /style

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

thanks @ppbrown , can you add some simple information to the README so people will know what this pipeline is, just a description and maybe a link to your experiment?

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

also the style bot didn't work, can you run make style and make quality so the test passes

@ppbrown
Copy link
Contributor Author

ppbrown commented Jun 9, 2025

I committed the tweaks from "make style".

"make quality" didnt seem to do anything, I think?

I looked at the README_contrib..... file, but.... the existing formatting in that made my eyes bleed and my head hurt :(
So I didnt touch it.
I did at least add some more explanitory comments to the top of the code, though.

@asomoza
Copy link
Member

asomoza commented Jun 9, 2025

I looked at the README_contrib..... file, but.... the existing formatting in that made my eyes bleed and my head hurt :(

yeah, probably that file is too big right now and not easy to modify and read, maybe we should think a better way to maintain this @stevhliu

WDTY of just a index file that points to an individual README for each community pipeline? this way people can just read directly the readme for the pipeline they want to use.

@ppbrown
Copy link
Contributor Author

ppbrown commented Jun 9, 2025

There are already a LOT of files in that directory.
Doubling that number doesnt sound like a great idea to me.

IMO, a better approach might be:

redo the README_community as a SINGLE LINE summary of each file. eg:

|filename| 100-char max descrption|

And then come up with a docstring standard expected format for a comment inside each pipeline, for more details.

Alternatively, go with the seperate README file for each, but make them live in a "docs" subdir.

@asomoza
Copy link
Member

asomoza commented Jun 9, 2025

@bot /style

Copy link
Contributor

github-actions bot commented Jun 9, 2025

Style fixes have been applied. View the workflow run here.

@asomoza
Copy link
Member

asomoza commented Jun 9, 2025

yeah I don't have a strong opinion about how to maintain it more simple and clean so anything works for me, probably not the best idea to impose some conditions for it like a specific format for the docstrings or README since the idea for this section is to people to contribute without the need to maintain a strict guideline.

let's wait for @stevhliu opinion about this., this is not a merge blocker, just something for the future.

@asomoza
Copy link
Member

asomoza commented Jun 9, 2025

@ppbrown thanks a lot!

@asomoza asomoza merged commit 6c7fad7 into huggingface:main Jun 9, 2025
@stevhliu
Copy link
Member

stevhliu commented Jun 9, 2025

I agree something simple and clean works best. Maybe just keep the table short and entirely get rid of the ## Example usages which is making the doc huge, and instead, rely on link to the notebook example

Description Notebook example Authors
describe the pipeline example and add link to the code file add link to notebook example with inference code add authors

@ppbrown
Copy link
Contributor Author

ppbrown commented Jun 10, 2025

I for one have no idea how to use "notebooks", so that would be an annoying barrier to entry for me.
I dont see why it has to be a "notebook example" instead of just regular code usage.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants