tiny tweak to allow BatchEncoding.token_to_char when token doesn't correspond to chars

[garyhlai]

Tagging @n1t0 @thomwolf @sgugger but this PR should be extremely quick to review for anyone.

Problem

BatchEncoding.token_to_char is supposed to return the char spans in the original string; however, right now, for tokens such as "<s>, </s>, <CLS>" that don't correspond to any chars in the original string, an error is raised TypeError: type object argument after * must be an iterable, not NoneType.

Run the following snippet replicate:

from transformers import AutoTokenizer, AutoModel

model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
text = "He is an absolutely amazing software developer"
tokenized_text = tokenizer(text)
tokenized_text.token_to_chars(0) # 0 corresponds to <CLS>

Fix

The solution is to return None instead of raising an error for tokens not corresponding to any chars in the original string.

P.S.

I am lost as to why run_tests_torch failed for if [ -f test_list.txt ]; then python -m pytest -n 3 --dist=loadfile -s --make-reports=tests_torch $(cat test_list.txt) | tee tests_output.txt fi. Some help would be appreciated.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[garyhlai]

Can I follow up on this? Would love to get some feedback -- even if it's about the specific reasons why this PR is not up to standard. Thanks!

[sgugger]

LGTM, thanks for the ping! Sorry I didn't pick this PR earlier. @LysandreJik do you want to have a look too?

[LysandreJik]

This looks good to me too!

[LysandreJik]

Sorry for taking so long to review, could you just rebase on main so that the test suite passes?

cc @SaulLu for knowledge

[garyhlai]

Done! @sgugger @LysandreJik

[sgugger]

Thanks again for your contribution!