Use "hub" directory for cache instead of "diffusers" #2005

pcuenca · 2023-01-16T11:32:14Z

So far I've only changed the cache location. I think maybe we should migrate the existing cached files (so far living in diffusers) upon first launch. If so, how could we go about it? I remember there was a migration process in huggingface_hub when the cache layout was updated, so perhaps @Wauplin can recommend a sensible approach here? My initial idea would be to:

Detect whether cached files exist in diffusers.
If they do, attempt to move to hub. What happens if there's a directory conflict?
When successful, remember somewhere (where?) that a migration already took place so we don't attempt it again.
Do we need to deal with "old" versions of the cache layout?

HuggingFaceDocBuilderDev · 2023-01-16T11:36:43Z

The documentation is not available anymore as the PR was closed or merged.

Wauplin · 2023-01-16T11:54:52Z

Hi @pcuenca , thanks for making the change :)

About your questions, I see 2 things here:

Old-format vs new-format of the cache. I think this is a bit orthogonal in the sense that old style cached files are already not used by diffusers right? At least, moving the default cache location shouldn't be change the use of old vs new cache format (I expect).
Moving cache from old location to the new location would definitely be a good idea for users as it would avoid them to have to re-download everything. I remember that @sgugger did a script to convert cache from old-format to new-format when transformers made the change. If I remember correctly, the script was run once when loading transformers to convert all cached files. In your case (changing the location), you could have such a script as well to transfer the data from 1 folder to the other. The advantage is that once the script has run once, you don't have to care about checking the 2 locations each time you load a model. (maybe warn and ask the user to confirm before moving everything ?)

What happens if there's a directory conflict?

Good question. There shouldn't be conflicts in the snapshots/ and blobs/ directories but the refs/ might conflict. Since refs/ is a folder containing only references (e.g. files containing only 1 line -the commit oid-), I would advice to simply delete it. It will result in 1 additional HTTP call when the user loads a model for the first time but that's neglectable. snapshots/ is a folder containing only symlinks so I guess that if you have conflicts in it, it's best to just remove it (will cost extra HTTP calls afterwards but no re-download). Conflicts in the blobs/ folder would be much more problematic but I don't see how it could happen.

Remember somewhere (where?) that a migration already took place so we don't attempt it again

Easiest solution for that would be to delete the previous folder but that would cause issues for users that have both an old and a new version of diffusers on their machine. In general, I don't see how we could avoid any problem with such a change (either files are duplicated between old and new location -that's inefficient + when would you delete the old one ?-, either the user cannot run both a old and a new version of diffusers -would that be a problem ?-).
Maybe the safest way of handling that is to trigger a warning and suggest the user to manually delete the old folder ?

src/diffusers/utils/constants.py

I verified that the constants are available in huggingface_hub version 0.10.0, which is the minimum we require. Co-authored-by: Lucain Pouget <lucainp@gmail.com>

sgugger · 2023-01-16T14:11:48Z

For information, the whole logic to migrate to the new cache in Transformers is here.

patrickvonplaten · 2023-01-16T14:51:36Z

@pcuenca, it would be nice to automatically migrate the cache just like we do in transformers - think we can pretty much copy the whole logic :-)

pcuenca · 2023-01-16T14:54:03Z

Yes, I'll use @sgugger's logic. Thanks a lot for the pointer!

github-actions · 2023-02-25T15:03:43Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

keturn · 2023-02-26T05:45:22Z

Wait, this was never merged?

Well that explains some things.

julien-c · 2023-02-26T08:03:37Z

yeah i think we do want this merged

pcuenca

Simplifications with respect to the migration process in transformers:

Moving the whole cache folder (https://github.com/huggingface/transformers/blob/9ddf4f4f03608095224cd3354b62c6f7d0d4b009/src/transformers/utils/hub.py#L83-L98) is probably not going to happen as the new cache path will already exist, because diffusers uses transformers models that have possibly been migrated. That shortcut has been removed here.
The migration process consists of simply moving the model folders, as shown below. We don't need an update of the structure as far as I can tell.

src/diffusers/utils/hub_utils.py

pcuenca · 2023-02-26T17:10:56Z

One question I have, if we want to be extremely cautious, is whether to ignore the migration until the new version has been released (i.e., don't do anything when installing from main). This would prevent re-downloads if the user installs from main, the files are moved but then the user decides to go back to a PyPi release. I don't think this case was contemplated in transformers either.

src/diffusers/utils/hub_utils.py

keturn · 2023-02-27T15:28:48Z

One question I have, if we want to be extremely cautious, is whether to ignore the migration until the new version has been released (i.e., don't do anything when installing from main). This would prevent re-downloads if the user installs from main, the files are moved but then the user decides to go back to a PyPi release.

...or multiple venvs with different versions.

can you mitigate this by leaving symlinks at the old location to point to the new one?

Co-authored-by: Lucain <lucainp@gmail.com>

Especially important if we want to ensure that the user may want to invoke the process again later, if they are keeping multiple envs with different versions.

pcuenca · 2023-02-27T17:50:23Z

...or multiple venvs with different versions.
can you mitigate this by leaving symlinks at the old location to point to the new one?

Good idea. Note, however, that the migration process will only run once automatically. If you keep multiple versions, then you'll need to invoke diffusers.utils.hub_utils.move_cache() periodically to move models to the new location.

Wauplin

PR looks good to me, thanks for the changes 👍
Maybe another pair of eyes would be good before finally merging.

sayakpaul · 2023-02-28T13:20:19Z

src/diffusers/utils/hub_utils.py

+            "The cache for model files in Diffusers v0.14.0 has moved to a new location. Moving your "
+            "existing cached models. This is a one-time operation, you can interrupt it or run it "
+            "later by calling `diffusers.utils.hub_utils.move_cache()`."


Is there a need to also spit out the actual location in this message for ultra transparency?

Yes, perhaps we can include that too.

sayakpaul

Thanks a lot!

patrickvonplaten

Cool - thanks !

carson-katri · 2023-03-02T21:08:17Z

Only the blobs folder was moved when I ran the migration. Should the other subfolders within the models be moved, or are they not needed?

Wauplin · 2023-03-03T06:28:35Z

@carson-katri Yes it's normal. The blobs folders are the only ones that really take space on your disk. If the move_cache() script worked properly without warning, all the blob files should have been moved to the new location and a symlink should have been added in the old location. The goal is that both new and old cache should be functional if you have a transition period with multiple versions of diffusers on your machine.
(Note: the refs and snapshots folders will be re-created the first time you load a model (it's only symlinks so real quick). Don't use use_local_files=True on your first run after the moving script)

Hope this clarifies everything :)

carson-katri · 2023-03-03T13:34:10Z

@Wauplin Got it, thanks. My issue was I was reading the cache to see what models the user has available. But I think I should read the old cache and new cache now until those folders are recreated.

Wauplin · 2023-03-03T14:21:56Z

@carson-katri yes exactly. Also worth mentioning we have a scan-cache cli to list which models you have in your cache.

huggingface-cli scan-cache
# or
# huggingface-cli scan-cache --dir=path/to/cache

Disclaimer: I'm not really sure it will work on the old cache has the tool is not expecting symlinks in the blobs folder.

@damian0815

…accelerate 0.16 (#2865) Things to check for in this version: - `diffusers` cache location is now more consistent with other huggingface-hub using code (i.e. `transformers`) as of huggingface/diffusers#2005. I think ultimately this should make @damian0815 (and other folks with multiple diffusers-using projects) happier, but it's worth taking a look to make sure the way @lstein set things up to respect `HF_HOME` is still functioning as intended. - I've gone ahead and updated `transformers` to the current version (4.26), but I have a vague memory that we were holding it back at some point? Need to look that up and see if that's the case and why.

@Wauplin

* Use "hub" directory for cache instead of "diffusers" * Import cache locations from huggingface_hub I verified that the constants are available in huggingface_hub version 0.10.0, which is the minimum we require. Co-authored-by: Lucain Pouget <lucainp@gmail.com> * make style * Move cached directories to new location. * make style * Apply suggestions by @Wauplin Co-authored-by: Lucain <lucainp@gmail.com> * Fix is_file * Ignore symlinks. Especially important if we want to ensure that the user may want to invoke the process again later, if they are keeping multiple envs with different versions. * Style --------- Co-authored-by: Lucain Pouget <lucainp@gmail.com>

@Wauplin

* Use "hub" directory for cache instead of "diffusers" * Import cache locations from huggingface_hub I verified that the constants are available in huggingface_hub version 0.10.0, which is the minimum we require. Co-authored-by: Lucain Pouget <lucainp@gmail.com> * make style * Move cached directories to new location. * make style * Apply suggestions by @Wauplin Co-authored-by: Lucain <lucainp@gmail.com> * Fix is_file * Ignore symlinks. Especially important if we want to ensure that the user may want to invoke the process again later, if they are keeping multiple envs with different versions. * Style --------- Co-authored-by: Lucain Pouget <lucainp@gmail.com>

@Wauplin

* Use "hub" directory for cache instead of "diffusers" * Import cache locations from huggingface_hub I verified that the constants are available in huggingface_hub version 0.10.0, which is the minimum we require. Co-authored-by: Lucain Pouget <lucainp@gmail.com> * make style * Move cached directories to new location. * make style * Apply suggestions by @Wauplin Co-authored-by: Lucain <lucainp@gmail.com> * Fix is_file * Ignore symlinks. Especially important if we want to ensure that the user may want to invoke the process again later, if they are keeping multiple envs with different versions. * Style --------- Co-authored-by: Lucain Pouget <lucainp@gmail.com>

Use "hub" directory for cache instead of "diffusers"

3947c12

Wauplin reviewed Jan 16, 2023

View reviewed changes

src/diffusers/utils/constants.py Outdated Show resolved Hide resolved

pcuenca and others added 2 commits January 16, 2023 13:36

Import cache locations from huggingface_hub

c3f1e9f

I verified that the constants are available in huggingface_hub version 0.10.0, which is the minimum we require. Co-authored-by: Lucain Pouget <lucainp@gmail.com>

make style

e643da4

github-actions bot added the stale Issues that haven't received updates label Feb 25, 2023

pcuenca removed the stale Issues that haven't received updates label Feb 26, 2023

pcuenca added 3 commits February 26, 2023 15:09

Merge remote-tracking branch 'origin/main' into cache-location

babeed8

Move cached directories to new location.

0e3de0a

make style

ed72dbf

pcuenca commented Feb 26, 2023

View reviewed changes

src/diffusers/utils/hub_utils.py Outdated Show resolved Hide resolved

pcuenca requested review from patrickvonplaten and williamberman February 26, 2023 17:07

Wauplin reviewed Feb 27, 2023

View reviewed changes

src/diffusers/utils/hub_utils.py Outdated Show resolved Hide resolved

pcuenca and others added 3 commits February 27, 2023 17:49

Apply suggestions by @Wauplin

3b742cd

Co-authored-by: Lucain <lucainp@gmail.com>

Fix is_file

b0f0239

Ignore symlinks.

d8c6127

Especially important if we want to ensure that the user may want to invoke the process again later, if they are keeping multiple envs with different versions.

Style

49dd37f

Wauplin approved these changes Feb 28, 2023

View reviewed changes

sayakpaul reviewed Feb 28, 2023

View reviewed changes

sayakpaul approved these changes Feb 28, 2023

View reviewed changes

patrickvonplaten approved these changes Feb 28, 2023

View reviewed changes

pcuenca merged commit 477aaa9 into main Feb 28, 2023

pcuenca deleted the cache-location branch February 28, 2023 19:01

keturn mentioned this pull request Mar 5, 2023

deps: upgrade to diffusers 0.14, safetensors 0.3, transformers 4.26, accelerate 0.16 invoke-ai/InvokeAI#2865

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use "hub" directory for cache instead of "diffusers" #2005

Use "hub" directory for cache instead of "diffusers" #2005

pcuenca commented Jan 16, 2023

HuggingFaceDocBuilderDev commented Jan 16, 2023 •

edited

Loading

Wauplin commented Jan 16, 2023

sgugger commented Jan 16, 2023

patrickvonplaten commented Jan 16, 2023

pcuenca commented Jan 16, 2023

github-actions bot commented Feb 25, 2023

keturn commented Feb 26, 2023

julien-c commented Feb 26, 2023

pcuenca left a comment •

edited

Loading

pcuenca commented Feb 26, 2023

keturn commented Feb 27, 2023

pcuenca commented Feb 27, 2023

Wauplin left a comment •

edited

Loading

sayakpaul Feb 28, 2023

pcuenca Feb 28, 2023

sayakpaul left a comment

patrickvonplaten left a comment

carson-katri commented Mar 2, 2023 •

edited

Loading

Wauplin commented Mar 3, 2023

carson-katri commented Mar 3, 2023

Wauplin commented Mar 3, 2023

Use "hub" directory for cache instead of "diffusers" #2005

Use "hub" directory for cache instead of "diffusers" #2005

Conversation

pcuenca commented Jan 16, 2023

HuggingFaceDocBuilderDev commented Jan 16, 2023 • edited Loading

Wauplin commented Jan 16, 2023

sgugger commented Jan 16, 2023

patrickvonplaten commented Jan 16, 2023

pcuenca commented Jan 16, 2023

github-actions bot commented Feb 25, 2023

keturn commented Feb 26, 2023

julien-c commented Feb 26, 2023

pcuenca left a comment • edited Loading

Choose a reason for hiding this comment

pcuenca commented Feb 26, 2023

keturn commented Feb 27, 2023

pcuenca commented Feb 27, 2023

Wauplin left a comment • edited Loading

Choose a reason for hiding this comment

sayakpaul Feb 28, 2023

Choose a reason for hiding this comment

pcuenca Feb 28, 2023

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

patrickvonplaten left a comment

Choose a reason for hiding this comment

carson-katri commented Mar 2, 2023 • edited Loading

Wauplin commented Mar 3, 2023

carson-katri commented Mar 3, 2023

Wauplin commented Mar 3, 2023

HuggingFaceDocBuilderDev commented Jan 16, 2023 •

edited

Loading

pcuenca left a comment •

edited

Loading

Wauplin left a comment •

edited

Loading

carson-katri commented Mar 2, 2023 •

edited

Loading