Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add rustls-tls feature #1732

Merged
merged 1 commit into from
Feb 11, 2025
Merged

Conversation

torymur
Copy link
Contributor

@torymur torymur commented Feb 10, 2025

Allow opting out of openssl in favor of rustls via features = ["http", "rustls-tls"].

Tokenizers with features = ["http"] depends on hf-hub:

[[package]]
name = "tokenizers"
version = "0.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ecededfed68a69bc657e486510089e255e53c3d38cc7d4d59c8742668ca2cae"
dependencies = [
 "aho-corasick",
 "derive_builder",
 "esaxx-rs",
 "getrandom 0.2.15",
 "hf-hub 0.3.2",
...
]

And default features of hf-hub bring openssl with it via native-tls:

[[package]]
name = "hf-hub"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b780635574b3d92f036890d8373433d6f9fc7abb320ee42a5c25897fc8ed732"
dependencies = [
 "dirs",
 "indicatif",
 "log",
 "native-tls",
 "rand",
 "serde",
 "serde_json",
 "thiserror 1.0.69",
 "ureq",
]

However, hf-hub provides a way out of openssl via rustls-tls feature: https://github.com/huggingface/hf-hub/blob/v0.4.1/Cargo.toml#L47

This change will allow to use tokenizers without openssl with features = ["http", "rustls-tls"].

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for both the bump and adding rustls-tls

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker ArthurZucker merged commit 4f1a810 into huggingface:main Feb 11, 2025
29 checks passed
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants