Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Remove deprecated HfFolder #6512

Merged
merged 3 commits into from
Dec 19, 2023
Merged

Remove deprecated HfFolder #6512

merged 3 commits into from
Dec 19, 2023

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Dec 19, 2023 •
edited
Loading

...and use huggingface_hub.get_token() instead

@lhoestq lhoestq requested a review from Wauplin December 19, 2023 14:40
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Wauplin
Wauplin reviewed Dec 19, 2023
View reviewed changes
Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening a PR. Actually I think you can even get rid of huggingface_hub.get_token if you want (see comment).

src/datasets/utils/file_utils.py Outdated
Comment on lines 262 to 265
if config.HF_HUB_VERSION > version.parse("0.20.0"):
token = huggingface_hub.get_token()
else:
token = huggingface_hub.HfFolder.get_token()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the whole get_authentication_headers_for_url can be replaced by huggingface_hub.utils.build_hf_headers which returns a dictionary with authorization + user_agent. You can then override some fields if needed.

albertvillanova
albertvillanova approved these changes Dec 19, 2023
View reviewed changes
Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@albertvillanova albertvillanova mentioned this pull request Dec 19, 2023
Closed
Wauplin
Wauplin reviewed Dec 19, 2023
View reviewed changes
src/datasets/utils/file_utils.py Outdated
Comment on lines 257 to 260
if config.HF_HUB_VERSION >= version.parse("0.20.0"):
return huggingface_hub.utils.build_hf_headers(
token=token, library_name="datasets", library_version=__version__
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_hf_headers is available since Sept 2022 (see huggingface/huggingface_hub#1064) so no need to check HF_HUB_VERSION here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed, therefore I got rid of HfFolder completely :)

Wauplin
Wauplin approved these changes Dec 19, 2023
View reviewed changes
Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for making the changes @lhoestq!

@lhoestq lhoestq merged commit a91582d into main Dec 19, 2023
12 checks passed
@lhoestq lhoestq deleted the remove-deprecarted-hffolder branch December 19, 2023 20:14
@github-actions GitHub Actions
Copy link

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric read_batch_formatted_as_numpy after write_array2d read_batch_formatted_as_numpy after write_flattened_sequence read_batch_formatted_as_numpy after write_nested_sequence read_batch_unformated after write_array2d read_batch_unformated after write_flattened_sequence read_batch_unformated after write_nested_sequence read_col_formatted_as_numpy after write_array2d read_col_formatted_as_numpy after write_flattened_sequence read_col_formatted_as_numpy after write_nested_sequence read_col_unformated after write_array2d read_col_unformated after write_flattened_sequence read_col_unformated after write_nested_sequence read_formatted_as_numpy after write_array2d read_formatted_as_numpy after write_flattened_sequence read_formatted_as_numpy after write_nested_sequence read_unformated after write_array2d read_unformated after write_flattened_sequence read_unformated after write_nested_sequence write_array2d write_flattened_sequence write_nested_sequence
new / old (diff) 0.005468 / 0.011353 (-0.005885) 0.003447 / 0.011008 (-0.007561) 0.062569 / 0.038508 (0.024061) 0.049427 / 0.023109 (0.026318) 0.238463 / 0.275898 (-0.037435) 0.268320 / 0.323480 (-0.055159) 0.002834 / 0.007986 (-0.005151) 0.002679 / 0.004328 (-0.001649) 0.048613 / 0.004250 (0.044363) 0.038793 / 0.037052 (0.001741) 0.247710 / 0.258489 (-0.010779) 0.277557 / 0.293841 (-0.016284) 0.027134 / 0.128546 (-0.101412) 0.010346 / 0.075646 (-0.065301) 0.205782 / 0.419271 (-0.213490) 0.035549 / 0.043533 (-0.007983) 0.241667 / 0.255139 (-0.013472) 0.268358 / 0.283200 (-0.014842) 0.017119 / 0.141683 (-0.124563) 1.108487 / 1.452155 (-0.343668) 1.177519 / 1.492716 (-0.315197)

Benchmark: benchmark_getitem_100B.json

metric get_batch_of_1024_random_rows get_batch_of_1024_rows get_first_row get_last_row
new / old (diff) 0.090925 / 0.018006 (0.072919) 0.310422 / 0.000490 (0.309932) 0.000212 / 0.000200 (0.000012) 0.000053 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric select shard shuffle sort train_test_split
new / old (diff) 0.018912 / 0.037411 (-0.018499) 0.061534 / 0.014526 (0.047008) 0.073608 / 0.176557 (-0.102949) 0.119278 / 0.737135 (-0.617858) 0.074698 / 0.296338 (-0.221640)

Benchmark: benchmark_iterating.json

metric read 5000 read 50000 read_batch 50000 10 read_batch 50000 100 read_batch 50000 1000 read_formatted numpy 5000 read_formatted pandas 5000 read_formatted tensorflow 5000 read_formatted torch 5000 read_formatted_batch numpy 5000 10 read_formatted_batch numpy 5000 1000 shuffled read 5000 shuffled read 50000 shuffled read_batch 50000 10 shuffled read_batch 50000 100 shuffled read_batch 50000 1000 shuffled read_formatted numpy 5000 shuffled read_formatted_batch numpy 5000 10 shuffled read_formatted_batch numpy 5000 1000
new / old (diff) 0.287224 / 0.215209 (0.072014) 2.792022 / 2.077655 (0.714367) 1.474605 / 1.504120 (-0.029515) 1.348714 / 1.541195 (-0.192481) 1.381339 / 1.468490 (-0.087151) 0.553033 / 4.584777 (-4.031744) 2.360745 / 3.745712 (-1.384967) 2.779281 / 5.269862 (-2.490580) 1.743922 / 4.565676 (-2.821754) 0.063817 / 0.424275 (-0.360458) 0.004954 / 0.007607 (-0.002653) 0.340039 / 0.226044 (0.113994) 3.336771 / 2.268929 (1.067843) 1.825573 / 55.444624 (-53.619051) 1.525362 / 6.876477 (-5.351115) 1.578793 / 2.142072 (-0.563280) 0.638432 / 4.805227 (-4.166795) 0.117601 / 6.500664 (-6.383063) 0.041890 / 0.075469 (-0.033579)

Benchmark: benchmark_map_filter.json

metric filter map fast-tokenizer batched map identity map identity batched map no-op batched map no-op batched numpy map no-op batched pandas map no-op batched pytorch map no-op batched tensorflow
new / old (diff) 0.936896 / 1.841788 (-0.904892) 11.426979 / 8.074308 (3.352671) 10.636043 / 10.191392 (0.444651) 0.136172 / 0.680424 (-0.544252) 0.014249 / 0.534201 (-0.519952) 0.287806 / 0.579283 (-0.291477) 0.266046 / 0.434364 (-0.168318) 0.326155 / 0.540337 (-0.214183) 0.455508 / 1.386936 (-0.931428)
PyArrow==latest
Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric read_batch_formatted_as_numpy after write_array2d read_batch_formatted_as_numpy after write_flattened_sequence read_batch_formatted_as_numpy after write_nested_sequence read_batch_unformated after write_array2d read_batch_unformated after write_flattened_sequence read_batch_unformated after write_nested_sequence read_col_formatted_as_numpy after write_array2d read_col_formatted_as_numpy after write_flattened_sequence read_col_formatted_as_numpy after write_nested_sequence read_col_unformated after write_array2d read_col_unformated after write_flattened_sequence read_col_unformated after write_nested_sequence read_formatted_as_numpy after write_array2d read_formatted_as_numpy after write_flattened_sequence read_formatted_as_numpy after write_nested_sequence read_unformated after write_array2d read_unformated after write_flattened_sequence read_unformated after write_nested_sequence write_array2d write_flattened_sequence write_nested_sequence
new / old (diff) 0.005199 / 0.011353 (-0.006154) 0.003476 / 0.011008 (-0.007532) 0.050519 / 0.038508 (0.012011) 0.050732 / 0.023109 (0.027623) 0.270140 / 0.275898 (-0.005758) 0.295539 / 0.323480 (-0.027941) 0.004057 / 0.007986 (-0.003928) 0.002771 / 0.004328 (-0.001558) 0.049157 / 0.004250 (0.044906) 0.039863 / 0.037052 (0.002811) 0.275934 / 0.258489 (0.017445) 0.306971 / 0.293841 (0.013130) 0.029405 / 0.128546 (-0.099141) 0.010746 / 0.075646 (-0.064900) 0.058427 / 0.419271 (-0.360845) 0.032448 / 0.043533 (-0.011085) 0.271851 / 0.255139 (0.016712) 0.290337 / 0.283200 (0.007138) 0.019145 / 0.141683 (-0.122538) 1.112232 / 1.452155 (-0.339922) 1.215153 / 1.492716 (-0.277564)

Benchmark: benchmark_getitem_100B.json

metric get_batch_of_1024_random_rows get_batch_of_1024_rows get_first_row get_last_row
new / old (diff) 0.088590 / 0.018006 (0.070584) 0.299047 / 0.000490 (0.298558) 0.000219 / 0.000200 (0.000019) 0.000050 / 0.000054 (-0.000005)

Benchmark: benchmark_indices_mapping.json

metric select shard shuffle sort train_test_split
new / old (diff) 0.022755 / 0.037411 (-0.014656) 0.078720 / 0.014526 (0.064194) 0.089051 / 0.176557 (-0.087505) 0.129330 / 0.737135 (-0.607805) 0.090645 / 0.296338 (-0.205693)

Benchmark: benchmark_iterating.json

metric read 5000 read 50000 read_batch 50000 10 read_batch 50000 100 read_batch 50000 1000 read_formatted numpy 5000 read_formatted pandas 5000 read_formatted tensorflow 5000 read_formatted torch 5000 read_formatted_batch numpy 5000 10 read_formatted_batch numpy 5000 1000 shuffled read 5000 shuffled read 50000 shuffled read_batch 50000 10 shuffled read_batch 50000 100 shuffled read_batch 50000 1000 shuffled read_formatted numpy 5000 shuffled read_formatted_batch numpy 5000 10 shuffled read_formatted_batch numpy 5000 1000
new / old (diff) 0.294083 / 0.215209 (0.078874) 2.907195 / 2.077655 (0.829540) 1.607392 / 1.504120 (0.103272) 1.481931 / 1.541195 (-0.059263) 1.486934 / 1.468490 (0.018444) 0.574093 / 4.584777 (-4.010684) 2.439775 / 3.745712 (-1.305937) 2.739818 / 5.269862 (-2.530044) 1.753922 / 4.565676 (-2.811755) 0.063738 / 0.424275 (-0.360537) 0.005219 / 0.007607 (-0.002388) 0.350342 / 0.226044 (0.124297) 3.463644 / 2.268929 (1.194716) 1.971598 / 55.444624 (-53.473026) 1.671752 / 6.876477 (-5.204724) 1.686504 / 2.142072 (-0.455569) 0.655870 / 4.805227 (-4.149357) 0.117580 / 6.500664 (-6.383084) 0.041210 / 0.075469 (-0.034259)

Benchmark: benchmark_map_filter.json

metric filter map fast-tokenizer batched map identity map identity batched map no-op batched map no-op batched numpy map no-op batched pandas map no-op batched pytorch map no-op batched tensorflow
new / old (diff) 0.996305 / 1.841788 (-0.845482) 12.426361 / 8.074308 (4.352053) 10.600309 / 10.191392 (0.408917) 0.129728 / 0.680424 (-0.550695) 0.015267 / 0.534201 (-0.518934) 0.285444 / 0.579283 (-0.293839) 0.272375 / 0.434364 (-0.161989) 0.323478 / 0.540337 (-0.216860) 0.547566 / 1.386936 (-0.839370)

# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@albertvillanova albertvillanova albertvillanova approved these changes

@Wauplin Wauplin Wauplin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lhoestq @HuggingFaceDocBuilderDev @albertvillanova @Wauplin