Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Bug fix] Tokenization multiprocessing fix #845

Merged
merged 1 commit into from
May 31, 2024
Merged

[Bug fix] Tokenization multiprocessing fix #845

merged 1 commit into from
May 31, 2024

Conversation

wheresmyhair
Copy link
Collaborator

  1. Tokenization multiprocessing bug fix
    fix

  2. Disable passing a ConversationTemplate object directly through model.tokenize() method. For customizing conversation templates, please refer to doc.

research4pan
research4pan approved these changes May 31, 2024
View reviewed changes
Copy link
Contributor

@research4pan research4pan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 The fix moves the tokenization function outside, avoiding shared arguments passing through closure, which is highly probable the source of the multi-process bug.

@research4pan research4pan merged commit 365b5ef into main May 31, 2024
2 checks passed
This was referenced May 31, 2024
Open
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@research4pan research4pan research4pan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wheresmyhair @research4pan