Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[INFER][LLM] Add the AutoPredictor for inference #9445

Merged
merged 9 commits into from
Dec 3, 2024

Conversation

zeroRains
Copy link
Contributor

@zeroRains zeroRains commented Nov 17, 2024

PR types

New features

PR changes

Others

Description

整合现有全部Predictor的选择条件,创建AutoPredicotr,可以处理static graph/dynamic graph,inference_mode/not inference_mode,block_attn/not block_attn的情况。

调用方式

predictor = AutoPredictor.create_predictor(predictor_args, config, model_args, tokenizer, model=model)

AutoPredictor的执行流程图如下:

Predictor流程图

当前LLM最简单的调用流程如下图:

8CD0A83C3720B34F3C11FC75FE048AFB

Copy link

paddle-bot bot commented Nov 17, 2024

Thanks for your contribution!

Copy link

codecov bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 53.08%. Comparing base (4b02477) to head (42b3e6a).
Report is 20 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/transformers/auto/modeling.py 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9445      +/-   ##
===========================================
+ Coverage    52.93%   53.08%   +0.14%     
===========================================
  Files          688      699      +11     
  Lines       109379   110476    +1097     
===========================================
+ Hits         57899    58641     +742     
- Misses       51480    51835     +355     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yuanlehome yuanlehome self-assigned this Nov 27, 2024
@yuanlehome yuanlehome self-requested a review November 27, 2024 03:27
@ZHUI ZHUI merged commit 2c1387f into PaddlePaddle:develop Dec 3, 2024
9 of 12 checks passed
@zeroRains zeroRains deleted the auto branch December 3, 2024 03:55
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants