Add community class StableDiffusionXL_T5Pipeline

[ppbrown]

What does this PR do?

This adds a new community pipeline, named
StableDiffusionXL_T5Pipeline

It grafts the T5 xxl encoder onto SDXL, completely replacing TE1 and TE2.

It has been tested to produce output. However, the output isnt particularly valid.
The unet needs to be retrained.

Will initially be used with base model opendiffusionai/stablediffusionxl_t5
until a proper (re)fine-tune can be made.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x ] Did you read the contributor guideline?
• [x ] Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

• Pipelines and pipeline callbacks: @yiyixuxu and @asomoza
• Training examples: @sayakpaul
• Docs: @stevhliu and @sayakpaul
• JAX and MPS: @pcuenca
• Audio: @sanchit-gandhi
• General functionalities: @sayakpaul @yiyixuxu @DN6

[ppbrown]

PS: This code was tested locally and ran, using the following test harness:

from diffusers import DiffusionPipeline
import torch.nn as nn, torch, types

SDXL_DIR = "/home/phil/git/models/t5-sdxl-model"

LOCAL_CODE = "/home/phil/git/diffusers.t5/examples/community/"

pipe = DiffusionPipeline.from_pretrained(
    SDXL_DIR, custom_pipeline=LOCAL_CODE, use_safetensors=True,
    torch_dtype=torch.bfloat16,
)

print("model initialized. Now moving to CUDA")
pipe.to("cuda")

print("Trying render now...")

images = pipe("a misty Tokyo alley at night",num_inference_steps=30).images
images[0].save("test.png")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

@bot /style

[asomoza]

thanks @ppbrown , can you add some simple information to the README so people will know what this pipeline is, just a description and maybe a link to your experiment?

[asomoza]

also the style bot didn't work, can you run make style and make quality so the test passes

[ppbrown]

I committed the tweaks from "make style".

"make quality" didnt seem to do anything, I think?

I looked at the README_contrib..... file, but.... the existing formatting in that made my eyes bleed and my head hurt :(
So I didnt touch it.
I did at least add some more explanitory comments to the top of the code, though.

[asomoza]

I looked at the README_contrib..... file, but.... the existing formatting in that made my eyes bleed and my head hurt :(

yeah, probably that file is too big right now and not easy to modify and read, maybe we should think a better way to maintain this @stevhliu

WDTY of just a index file that points to an individual README for each community pipeline? this way people can just read directly the readme for the pipeline they want to use.

[ppbrown]

There are already a LOT of files in that directory.
Doubling that number doesnt sound like a great idea to me.

IMO, a better approach might be:

redo the README_community as a SINGLE LINE summary of each file. eg:

|filename| 100-char max descrption|

And then come up with a docstring standard expected format for a comment inside each pipeline, for more details.

Alternatively, go with the seperate README file for each, but make them live in a "docs" subdir.

[asomoza]

@bot /style

[github-actions]

Style fixes have been applied. View the workflow run here.

[asomoza]

yeah I don't have a strong opinion about how to maintain it more simple and clean so anything works for me, probably not the best idea to impose some conditions for it like a specific format for the docstrings or README since the idea for this section is to people to contribute without the need to maintain a strict guideline.

let's wait for @stevhliu opinion about this., this is not a merge blocker, just something for the future.

[asomoza]

@ppbrown thanks a lot!

[stevhliu]

I agree something simple and clean works best. Maybe just keep the table short and entirely get rid of the ## Example usages which is making the doc huge, and instead, rely on link to the notebook example

Description	Notebook example	Authors
describe the pipeline example and add link to the code file	add link to notebook example with inference code	add authors

[ppbrown]

I for one have no idea how to use "notebooks", so that would be an annoying barrier to entry for me.
I dont see why it has to be a "notebook example" instead of just regular code usage.

[stevhliu]

A notebook just lets you execute and run code directly and you can share them so I think they're actually pretty accessible. It's still regular code usage.

I think having it all self-contained in a notebook is cleaner than having all the inference code from all the community examples on the page.

[ppbrown]

I know what it IS. but I never found it convenient to USE/write, personally. Please keep in mind that just because something is easy and well known to you personally, doesnt make it easy to use for everyone else. Effectively, it would be just another barrier to entry, if you require this. Which seems to be the opposite sentiment to "hey lets make this simpler for everyone". Speaking for myself, if it was it was a requirement, I wouldnt have bothered to put in the PR. edit: but i agree that the sample code doesnt all belong in the community README page either.