Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 #1044

Merged
merged 4 commits into from
Oct 8, 2024

Conversation

pablomlago
Copy link
Collaborator

Fix for issue #1036. When fuse_sequences is False, and the seqlen is greater than 842 (maximum sequence length in WikiText2), the condition in line 131 of optimum/amd/brevitas/data_utils.py always evaluates to True, resulting in an infinite loop (see commit d0d06070225804dc6fecde00911ae38b53080123 in optimum-amd).

Consequently, we opted for enabling passing fuse_sequences as an argument and warning the user, when fuse_sequences=False, than an overly large seqlen can result in the infinite loop indicated in issue #1036.

@pablomlago
Copy link
Collaborator Author

pablomlago commented Oct 8, 2024

The distribution of the sequence lengths in WikiText2, when fuse_sequences is set to False, is shown in the following picture:
length_distribution_wikitext2_train
Note that, for the default value seqlen=2048, the infinite loop is entered, as the maximum sequence length is 842.

@pablomlago pablomlago changed the title [DO NOT MERGE] Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 Oct 8, 2024
@nickfraser nickfraser added the next release PRs which should be merged for the next release label Oct 8, 2024
@Giuseppe5
Copy link
Collaborator

Can you update the readme?
Generate the usage description with
python main.py -h
And copy and past it in the readme

@Giuseppe5 Giuseppe5 merged commit 4d8b153 into Xilinx:dev Oct 8, 2024
23 checks passed
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
next release PRs which should be merged for the next release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants