hf_hub_download and cached_download should read the token by default

[osanseviero]

Describe the bug

Right now, hf_hub_download does not work for private repos unless you set use_auth_token=True. This is inconsistent with other methods in which the token is automatically retrieved if not specified. I think we should change

huggingface_hub/src/huggingface_hub/file_download.py

Lines 577 to 586 in 7e80140

	if isinstance(use_auth_token, str):
	headers["authorization"] = f"Bearer {use_auth_token}"
	elif use_auth_token:
	token = HfFolder.get_token()
	if token is None:
	raise EnvironmentError(
	"You specified use_auth_token=True, but a huggingface token was not"
	" found."
	)
	headers["authorization"] = f"Bearer {token}"

to always get the token if it's not explicitly passed.

WDYT @LysandreJik @julien-c?

Reproduction

No response

Logs

No response

System Info

-

[julien-c]

we've discussed it in the past and decided against it for privacy protection reasons (especially in the case when it's used from third party libraries – the hf.co server will "know" what models user A is downloading without explicitly opting-in)

But maybe it does make sense to revisit this

[osanseviero]

That's the thing, a user A that wants to download a private model will need to explicitly opt-in. I'm not sure how the current setup guarantees more privacy protection that the other approach. My suggestion is to change

if isinstance(use_auth_token, str): 
     headers["authorization"] = f"Bearer {use_auth_token}" 
 elif use_auth_token: 
     token = HfFolder.get_token() 
     if token is None: 
         raise EnvironmentError( 
             "You specified use_auth_token=True, but a huggingface token was not" 
             " found." 
         ) 

to

if isinstance(use_auth_token, str): 
     headers["authorization"] = f"Bearer {use_auth_token}" 

token = HfFolder.get_token() 
if token is None: 
     raise EnvironmentError( 
         "You specified use_auth_token=True, but a huggingface token was not" 
         " found." 
     ) 

Note that the upload methods already do this automatically (as in

huggingface_hub/src/huggingface_hub/hf_api.py

Line 1792 in 6617550

token, name = self._validate_or_retrieve_token(token)

), so not sure why the download should be different.

[julien-c]

not sure why the download should be different

Upload methods aren't expected by the user to even work w/o authentication, but download methods are (for public repos anyways)

I will try to find the past discussions of this and link them here, for completeness.

But note that I'm not opposed to changing this behavior, just need to be aware of privacy implications

[julien-c]

one example of internal convo but earlier discussions were probably more interesting

[julien-c]

@sgugger pointed me to this transformers PR for the original discussion huggingface/transformers#9141