Set usedforsecurity=False in hashlib methods (FIPS compliance)

[Wauplin]

Solves #27034 (cc @DueViktor).

This PR makes transformers FIPS-compliant regarding hashlib usage by setting usedforsecurity=False in every hashlib method (used for file checking, not cryptography purposes). It's based on utilities added in huggingface/huggingface_hub#1782 and released in huggingface_hub==v0.19.0.

Note: before merging this we need to release a new tokenizers version that would allow new huggingface_hub version (see huggingface/tokenizers#1385). Tests are currently failing because of this which is expected. cc @ArthurZucker what's the status on the next release?

[amyeroberts]

Thanks for updating!

Happy to merge once all dependencies have been handled. Would be good to have a quick review from @ydshieh who is the packages and CI master 👑

[ydshieh]

LGTM.

Usually we don't touch any file udner examples/research_projects, but it's fine here (for me).

(not sure the name insecure_hashlib is the best name, but that is not the point of this PR 😄 )

[ArthurZucker]

Thanks! Not 100% sure we need to rely on the utility as it's a pretty small piece of code for something pretty code + not very intuitive to me that we need to import from hf hub. If others are fine good for me as well

[Wauplin]

Thanks! Not 100% sure we need to rely on the utility as it's a pretty small piece of code for something pretty code + not very intuitive to me that we need to import from hf hub. If others are fine good for me as well

Agree it's not big piece of code. But having it from an explicitly named insecure_hashlib module makes it more easily distinguishable IMO => no need to check each hashlib call to verify that the parameter is correctly set. Also you would have to duplicate the logic that checks whether you are running on python 3.9 or not (2 lines, but still not best to copy-paste around).
Also the dependency on huggingface_hub already exists so no big deal IMO (and updating to latest version forces to get the latest fixes we shipped -that's more of a cool side-effect)

[ArthurZucker]

You are right! 🤗

[Wauplin]

Thanks for merging #27494 @ArthurZucker ! I merged main into this PR and once CI is green I think we'll be good to merge :)

[Wauplin]

Still some issue with the dependencies... 😕
I opened a PR in huggingface_hub and will make another patch release later.

[Wauplin]

I converted this PR to draft because of the dependency issues we're having. I'll push a new fix in huggingface_hub to definitely settle this (see slack thread -internal- for explanations).

Please do not merge even if CI is green.

[amyeroberts]

Thanks for all the investigative work on this!

[amyeroberts]

@Wauplin I can see the PR was marked as ready for review again. Does this mean it's now mergeable?

Would be good to get a final 👍 from @ydshieh to make sure all relevant files for our CI have been updated.

[ydshieh]

Look good!

[amyeroberts]

Confirmed with @Wauplin over slack that the PR is good to go - merging!