feat(ui): support FLUX Fill on Canvas #7811
Conversation
psychedelicious
requested review from
blessedcoolant,
maryhipp,
hipsterusername,
brandonrising and
jazzhaiku
as code owners
github-actions
bot
added
python
|
Likely should override silently for Guidance specifically. Can’t rely on obscure knowledge for things to work, and “non-standard” guidance there is likely a job for workflows Failing on T2I or I2I with a clear message works. — How did we land on solving the ergonomics of having to switch to fill? Are users manually selecting that as their model? |
Yes - FLUX Fill is a separate main model alongside FLUX dev/schnell and works like other main models. We can differentiate between FLUX Fill and other FLUX models with the variant field. It is "inpaint" for the Fill model and "normal" for the others. We check this when building the graph. Due to the partial feature overlap between the FLUX model variants, it doesn't make sense to automatically choose FLUX Fill or dev/schnell on behalf of the user based on the generation mode (t2i/i2i/inpaint/outpaint). For example, Control LoRA isn't compatible with FLUX Fill. There are some other gotchas. Also, we don't have confidence that all the normal FLUX auxiliary models work the same with Fill. So we offer too much flexibility & control to choose the base model for the user without causing a lot of headaches and possibly compromising outputs. |
|
Error message when attempting to t2i or i2i with FLUX Fill: Warnings when using Control LoRA with FLUX Fill: When a FLUX Fill model is selected, the Guidance widget is hidden and guidance is silently set to 30 - the user-defined value is ignored. |
It simply applies the mask to an image.
…iting needs Previously we used erode/dilate and a Gaussian blur to expand and fade the edges of Canvas masks. The implementation a number of problems: - Erode/dilate kernel sizes were not calculated correctly, and extra iterations were run to compensate. The result is the blur size, which should have been pixels, was very inaccurate and unreliable. - What we want is to add a "soft bleed" - like a drop shadow with no offset - starting from the edge of the mask, extending out by however many pixels. But Gaussian blur does not do this. The blurred area starts _inside_ the mask and extends outside it. So it kinda blurs inwards and outwards. We compensated for this by expanding the mask. - Using a Gaussian blur can cause banding artifacts. Gaussian blur doesn't have a "size" or "radius" parameter in the sense that you think it should. It's a convolution matrix and there are _no non-zero values in the result_. This means that, far away from the mask, once compositing completes, we have some values that are very close to zero but not quite zero. These values are quantized by HTML Canvas, resulting in banding artifacts where you'd expect the blur to have faded to 0% alpha. At least, that is my understanding of why the banding artifacts occur. The new node uses a better strategy to expand the mask and add the fade out effect: - Calculate the distance from each white pixel to the nearest black pixel. - Normalize this distance by dividing by the fade size in px, then clip the values to 0 - 1. The result represents the distance of each white pixel to its nearest black pixel as a percentage of the fade size. At this point, it is a linear distribution. - Create a polynomial to describe the fade's intensity so that we can have a smooth transition from the masked region (black) to unmasked (white). There are some magic numbers here, deterined experimentally. - Evaluate the polynomial over the normalized distances, so we now have a matrix representing the fade intensity for every pixel - Convert this matrix back to uint8 and apply it to the mask This works soooo much better than the previous method. Not only does it fix the banding issues, but when we enable "output only generated regions", we get a much smaller image. Will add images to the PR to clarify.
… model fetching utils
psychedelicious
force-pushed
the
psyche/feat/ui/flux-fill-canvas
branch
from
df69bfd to
0f1519e
Compare
Summary
Canvas Compositing
I'm using this canvas setup for testing:
On the current release, there are subtle banding artifacts on the generated outputs. Here's a video attempting to show them:
Screen.Recording.2025-03-20.at.12.39.13.pm.mov
I took a screenshot and fiddled with the levels to accentuate the artifacts:
Depending on the image, these artifacts can range from nearly invisible to hard-to-miss.
Compare to this PR - there is still some minor banding but it's vastly improved:
Screen.Recording.2025-03-20.at.12.44.57.pm.mov
Besides the banding artifacts, the new nodes fix the size of the mask and its fade-out. The old logic didn't really make sense and didn't work how we wanted it to. The result was masks were blurred too much, then had to be made larger to compensate. So the mask parameters weren't respected.
See b80b964 for technical details and my best guess as to why the banding occurred in the first place.
TODO
Related Issues / Discussions
There have been numerous discussions about the banding and mask size issues. Hopefully this resolves them all.
QA Instructions
Besides the new FLUX Fill support on Canvas, I revised the graphs for inpainting and outpainting to use the new nodes. Both scaled bbox and unscaled bbox pathways were changed.
So, testing this PR fully means trying:
Merge Plan
n/a
Checklist
What's Newcopy (if doing a release after this PR)