Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Support and automatically detect SDXL V-prediction models #16567

Merged
merged 1 commit into from
Oct 19, 2024
Merged

Support and automatically detect SDXL V-prediction models #16567

merged 1 commit into from
Oct 19, 2024
+106 −3

Conversation

catboxanon
Copy link
Collaborator

@catboxanon catboxanon commented Oct 19, 2024 •
edited
Loading

Description

Title. Based on the detection here, where the existence of a v_pred key in the state_dict is checked.

The motivation for this change is due to an emerging model being trained with V-pred (currently only useable with an additional config file next to the model file) and ztSNR (supported as of 1.8.0) that is on par with NovelAI's closed source V3 model. The hope is this will reduce friction and increase user adoption.

Example model: https://civitai.com/models/833294?modelVersionId=1046043

If you wish to modify existing models that do not have these keys added, it's straightforward to do so.

from safetensors.torch import load_file, save_file
import torch
state_dict = load_file('/path/to/model.safetensors')
state_dict['v_pred'] = torch.tensor([])
state_dict['ztsnr'] = torch.tensor([])  # Added for PR #16569
save_file(state_dict, '/path/to/new_model.safetensors')

This also fixes config loading when the same base model architecture is being utilized.

Checklist:

@catboxanon catboxanon mentioned this pull request Oct 19, 2024
Merged
@catboxanon catboxanon changed the title Support SDXL V-prediction models Support and automatically detect SDXL V-prediction models Oct 19, 2024
@AUTOMATIC1111 AUTOMATIC1111 merged commit 8b19b75 into dev Oct 19, 2024
6 checks passed
@AUTOMATIC1111 AUTOMATIC1111 deleted the feat/sdxl-vpred branch October 19, 2024 14:41
@catboxanon catboxanon mentioned this pull request Oct 19, 2024
Merged
4 tasks
kafffka-cat added a commit to kafffka-cat/stable-diffusion-webui that referenced this pull request Oct 30, 2024
@kafffka-cat
Squashed commit of the following:
d3cb8fd 
commit d712b4f40a05498bc80ae83c8ea7ffaad6c9ece1
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Wed Oct 30 20:15:56 2024 +0300

    Update sd_models.py

commit 02fca85de807ece7b8caeb379ba1c7b38b50339c
Merge: 3b3bc10b e76d0a2
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Wed Oct 30 18:55:01 2024 +0300

    Merge branch 'master' of https://github.com/Kittensx/stable-diffusion-webui into pr/16598

commit 3b3bc10bbb35ec3e3bd69a1548fc9bf9364a1afe
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Wed Oct 30 18:53:54 2024 +0300

    Squashed commit of the following:

    commit acf8a5b6651f1916df81e0010dae9428bc8e002a
    Merge: 0d874a6 7a91722
    Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
    Date:   Wed Oct 30 18:43:10 2024 +0300

        Merge branch 'pr/16594' into pr/16290

    commit 7a91722
    Author: Won-Kyu Park <wkpark@gmail.com>
    Date:   Sat Oct 26 23:32:29 2024 +0900

        support loading libtcmalloc_minimal.dll for windows

    commit 5865da2
    Merge: bb1f391 c2ce1d3
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 11:40:02 2024 +0900

        Merge pull request AUTOMATIC1111#16569 from AUTOMATIC1111/feat/ztsnr-auto

        Automatically enable ztSNR based on existence of key in `state_dict`

    commit bb1f391
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:58:53 2024 +0900

        clarify readme: weget ... chmod +x webui.sh (AUTOMATIC1111#16251)

    commit 6a59766
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:56:12 2024 +0900

        Add Skip Early CFG to XYZ (AUTOMATIC1111#16282)

        Co-authored-by: Yevhenii Hurin <evgeny.gurin@gmail.com>

    commit 65423d2
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:52:47 2024 +0900

        MIME type text/css (AUTOMATIC1111#16406)

    commit c2bc187
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:51:59 2024 +0900

        fix modalImageViewer preview/result flicker (AUTOMATIC1111#16426)

    commit d0b27dc
    Merge: bb4cbaf cbaaf0a
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:42:21 2024 +0900

        Merge pull request AUTOMATIC1111#16300 from hello2564/fix_NGMS_pr_typo

        fix NGMS pr typo

    commit c2ce1d3
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 19:58:13 2024 -0400

        Automatically enable ztSNR based on existence of key in state_dict

    commit bb4cbaf
    Merge: c462e5a 9677b09
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 08:27:12 2024 +0900

        Merge pull request AUTOMATIC1111#16341 from gutris1/devv

        add break-word for geninfo in pnginfo

    commit c462e5a
    Merge: 8b19b75 c9a06d1
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 10:51:16 2024 -0400

        Merge pull request AUTOMATIC1111#16460 from AUTOMATIC1111/sd-1.5-url

    commit 8b19b75
    Merge: 907bfb5 1ae073c
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Oct 19 17:40:56 2024 +0300

        Merge pull request AUTOMATIC1111#16567 from AUTOMATIC1111/feat/sdxl-vpred

        Support and automatically detect SDXL V-prediction models

    commit 907bfb5
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Oct 19 17:33:58 2024 +0300

        add w-e-w and catboxanon to codeowners file

    commit 1ae073c
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 06:53:19 2024 -0400

        Support SDXL v-pred models

    commit c9a06d1
    Author: missionfloyd <missionfloyd@users.noreply.github.com>
    Date:   Tue Oct 8 16:50:39 2024 -0600

        Use stable-diffusion-v1-5 repo instead

    commit f57ec2b
    Author: missionfloyd <missionfloyd@users.noreply.github.com>
    Date:   Tue Sep 3 19:58:29 2024 -0600

        Update stable diffusion 1.5 URL

    commit 9677b09
    Author: gutris1 <132797949+gutris1@users.noreply.github.com>
    Date:   Wed Aug 7 17:37:23 2024 +0700

        add break-word for geninfo in pnginfo

    commit cbaaf0a
    Author: hello2564 <ksncdos@163.com>
    Date:   Wed Jul 31 14:55:30 2024 +0800

        fix NGMS pr typo

    commit 0d874a6
    Author: Abhishek Vakharia <davakharia@gmail.com>
    Date:   Sun Jul 28 21:23:34 2024 -0700

        fix move2move tab not showing

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit e79134c3f7d3becce2b9ed5584faacc7179d4088
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Wed Oct 30 18:50:53 2024 +0300

    Squashed commit of the following:

    commit 920a3a4
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Wed Jul 31 02:43:13 2024 +0900

        SD3 Lora page filter - detection not implemented

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit 834ba48f88f0176725771dc9d65bea1cb4a7e8fa
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Wed Oct 30 18:50:32 2024 +0300

    Squashed commit of the following:

    commit 09771cb4669da07ca12972ffd1b1cb0be3af4461
    Merge: 6796147 77c0c16
    Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
    Date:   Wed Oct 30 18:44:20 2024 +0300

        Merge branch 'sd-1.5-url' into warning-WebUI-is-installed-under-a-dot-directory

    commit 77c0c16
    Merge: de44ee2 e76d0a2
    Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
    Date:   Wed Oct 30 17:32:27 2024 +0300

        Merge branch 'pr/16598' into sd-1.5-url

    commit de44ee2
    Merge: c9a06d1 4595f9a
    Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
    Date:   Wed Oct 30 17:32:07 2024 +0300

        Merge branch 'pr/16553' into sd-1.5-url

    commit 6796147
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Wed Oct 30 15:44:14 2024 +0900

        add version check

    commit e76d0a2
    Merge: 4ea5403 2571d78
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 12:07:45 2024 -0400

        Merge pull request AUTOMATIC1111#4 from Kittensx/Kittensx-patch-2

        Add files via upload

    commit 2571d78
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 12:06:08 2024 -0400

        Add files via upload

        forgot to include this in my lasst commit

    commit 4ea5403
    Merge: 6ffb728 da2e709
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 11:08:53 2024 -0400

        Merge pull request AUTOMATIC1111#3 from Kittensx/Kittensx-patch-Simple-KES

        Update simple_karras_exponential_scheduler.py

    commit da2e709
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 11:08:06 2024 -0400

        Update simple_karras_exponential_scheduler.py

    commit 6ffb728
    Merge: 82a973c 78514e0
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 10:15:35 2024 -0400

        Merge pull request #1 from Kittensx/Kittensx-Simple-Karras-Exponential-Scheduler-2

        Added New Scheduler

    commit 78514e0
    Author: Kittensx <photographer.ksu30@gmail.com>
    Date:   Sun Oct 27 10:08:52 2024 -0400

        Add files via upload

    commit e568ea5
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Fri Oct 25 00:07:15 2024 +0900

        warning WebUI is installed under a dot directory

    commit 984b952
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Thu Oct 24 22:05:51 2024 +0900

        Fix DAT models download (AUTOMATIC1111#16302)

    commit 5865da2
    Merge: bb1f391 c2ce1d3
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 11:40:02 2024 +0900

        Merge pull request AUTOMATIC1111#16569 from AUTOMATIC1111/feat/ztsnr-auto

        Automatically enable ztSNR based on existence of key in `state_dict`

    commit bb1f391
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:58:53 2024 +0900

        clarify readme: weget ... chmod +x webui.sh (AUTOMATIC1111#16251)

    commit 6a59766
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:56:12 2024 +0900

        Add Skip Early CFG to XYZ (AUTOMATIC1111#16282)

        Co-authored-by: Yevhenii Hurin <evgeny.gurin@gmail.com>

    commit 65423d2
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:52:47 2024 +0900

        MIME type text/css (AUTOMATIC1111#16406)

    commit c2bc187
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:51:59 2024 +0900

        fix modalImageViewer preview/result flicker (AUTOMATIC1111#16426)

    commit d0b27dc
    Merge: bb4cbaf cbaaf0a
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 09:42:21 2024 +0900

        Merge pull request AUTOMATIC1111#16300 from hello2564/fix_NGMS_pr_typo

        fix NGMS pr typo

    commit c2ce1d3
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 19:58:13 2024 -0400

        Automatically enable ztSNR based on existence of key in state_dict

    commit bb4cbaf
    Merge: c462e5a 9677b09
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sun Oct 20 08:27:12 2024 +0900

        Merge pull request AUTOMATIC1111#16341 from gutris1/devv

        add break-word for geninfo in pnginfo

    commit c462e5a
    Merge: 8b19b75 c9a06d1
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 10:51:16 2024 -0400

        Merge pull request AUTOMATIC1111#16460 from AUTOMATIC1111/sd-1.5-url

    commit 8b19b75
    Merge: 907bfb5 1ae073c
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Oct 19 17:40:56 2024 +0300

        Merge pull request AUTOMATIC1111#16567 from AUTOMATIC1111/feat/sdxl-vpred

        Support and automatically detect SDXL V-prediction models

    commit 907bfb5
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Oct 19 17:33:58 2024 +0300

        add w-e-w and catboxanon to codeowners file

    commit 1ae073c
    Author: catboxanon <122327233+catboxanon@users.noreply.github.com>
    Date:   Sat Oct 19 06:53:19 2024 -0400

        Support SDXL v-pred models

    commit c9a06d1
    Author: missionfloyd <missionfloyd@users.noreply.github.com>
    Date:   Tue Oct 8 16:50:39 2024 -0600

        Use stable-diffusion-v1-5 repo instead

    commit f57ec2b
    Author: missionfloyd <missionfloyd@users.noreply.github.com>
    Date:   Tue Sep 3 19:58:29 2024 -0600

        Update stable diffusion 1.5 URL

    commit 9677b09
    Author: gutris1 <132797949+gutris1@users.noreply.github.com>
    Date:   Wed Aug 7 17:37:23 2024 +0700

        add break-word for geninfo in pnginfo

    commit cbaaf0a
    Author: hello2564 <ksncdos@163.com>
    Date:   Wed Jul 31 14:55:30 2024 +0800

        fix NGMS pr typo

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <40751091+w-e-w@users.noreply.github.com>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <16777216c@gmail.com>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit e3e3d1bbdd2f8ddd33c361fce27ddc4f26e3a4a7
Author: Thomas Ward <teward@thomas-ward.net>
Date:   Sun Oct 13 17:36:45 2024 -0400

    Update requirements.txt

    Add wheel to requirements because 'packaging' module is referred to and is not available unless you install 'wheel' in the environment.

commit e76d0a2
Merge: 4ea5403 2571d78
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 12:07:45 2024 -0400

    Merge pull request AUTOMATIC1111#4 from Kittensx/Kittensx-patch-2

    Add files via upload

commit 2571d78
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 12:06:08 2024 -0400

    Add files via upload

    forgot to include this in my lasst commit

commit 4ea5403
Merge: 6ffb728 da2e709
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 11:08:53 2024 -0400

    Merge pull request AUTOMATIC1111#3 from Kittensx/Kittensx-patch-Simple-KES

    Update simple_karras_exponential_scheduler.py

commit da2e709
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 11:08:06 2024 -0400

    Update simple_karras_exponential_scheduler.py

commit 6ffb728
Merge: 82a973c 78514e0
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 10:15:35 2024 -0400

    Merge pull request #1 from Kittensx/Kittensx-Simple-Karras-Exponential-Scheduler-2

    Added New Scheduler

commit 78514e0
Author: Kittensx <photographer.ksu30@gmail.com>
Date:   Sun Oct 27 10:08:52 2024 -0400

    Add files via upload

commit f3bb7e8
Merge: 82a973c c9a06d1
Author: kafffka-cat <144359060+kafffka-cat@users.noreply.github.com>
Date:   Thu Oct 10 01:18:03 2024 +0300

    Merge pull request #1 from kafffka-cat/sd-1.5-url

    Sd 1.5 url

commit c9a06d1
Author: missionfloyd <missionfloyd@users.noreply.github.com>
Date:   Tue Oct 8 16:50:39 2024 -0600

    Use stable-diffusion-v1-5 repo instead

commit f57ec2b
Author: missionfloyd <missionfloyd@users.noreply.github.com>
Date:   Tue Sep 3 19:58:29 2024 -0600

    Update stable diffusion 1.5 URL
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@AUTOMATIC1111 AUTOMATIC1111 AUTOMATIC1111 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@catboxanon @AUTOMATIC1111