Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merge gpugraph to develop #48507

Merged
merged 453 commits into from
Dec 21, 2022
Merged

Conversation

lxsbupt
Copy link
Contributor

@lxsbupt lxsbupt commented Nov 29, 2022

PR types

New features

PR changes

Others

Describe

gpugraph 二期合入develop,主要升级点

  1. 新增embedding和graph图属性的多级存储,通过ssd&内存,提升参数和图属性的存储规模。
  2. 新增采样、参数拉取和训练的三级流水线,最大化资源效率,提升框架性能。
  3. 新增graphsage&erniesage采样,扩展图采样算法,优化策略效果。
  4. 通过显存池、变长slot-feature存储等feature,优化显存利用,降低显存开销。
  5. 升级多处性能损耗点,比如topo_aware_、gpu table冲突统计和跨卡直接访问等。
  6. 升级cpu graph的的加载粒度和存储周期,解决大图场景的内存瓶颈。

@paddle-bot
Copy link

paddle-bot bot commented Nov 29, 2022

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot paddle-bot bot added contributor External developers status: proposed labels Nov 29, 2022
@paddle-bot
Copy link

paddle-bot bot commented Nov 29, 2022

✅ This PR's description meets the template requirements!
Please wait for other CI results.

@CLAassistant
Copy link

CLAassistant commented Nov 29, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 22 committers have signed the CLA.

✅ AndPuQing
✅ yuanlehome
✅ From00
✅ HydrogenSulfate
❌ zhengqiwen1997
❌ Ayuan2021
❌ zhaoyinglia
❌ wangxn12138
❌ sljlp
❌ mingxu1067
❌ haohongxiang
❌ jiweibo
❌ HermitSun
❌ yjjiang11
❌ Wangzheee
❌ Silv3S
❌ kk-2000
❌ LearningPawn
❌ lxsbupt
❌ laitingsheng
❌ feifei-111
❌ kangguangli
You have signed the CLA already but the status is still pending? Let us recheck it.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 9 committers have signed the CLA.

✅ danleifeng
✅ Thunderbrook
✅ DesmonDay
❌ chao9527
❌ miaoli06
❌ root
❌ huwei02
❌ qingshui
❌ lxsbupt


root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

lxsbupt and others added 24 commits December 6, 2022 14:31
)

* [CodeStyle][isort][Dy2St] sort imports in test_error

* update lineno
…ddle#47579)

* delete Bias and ResidualData in OpMaker of conv2d

* delete extra input of conv3d

* refactor pass of conv_bias_fusion

* fix mkldnn dependency

* fix mkldnn compile

* fix test_conv_bias_mkldnn_fuse_pass

* police some code

* remove useless log

* fix analyzer_vit_ocr_tester

* fix conv_activation_mkldnn_fuse_pass

* fix test_analyzer_ocr

* add fused_conv_sig

* fix performence regression

* fix performance regression
* make bilinear interpolate stable.

* fix code
* [Zero-Dim] Support 0D for paddle.diagflat
* modify slice infershape

* code style

* modify slice_unittest

* temp fix

* batch_norm api move

* code_style

* codestyle

* ci_static

* add __init__

* reset other change

* revert .cc

* add import batchnorm

* conflict and revert

* fix bug

* fix third conflict one day

* fix conflict

* fix conflict bug

* fix conflict bug

* modify api

* code_style

* modify doc

* add lost doc stable

* fix conflict bug

* ci lack of gpu
…nv RowConv (PaddlePaddle#48654)

* [remove fluid] PRelu BilinearTensorProduct

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv

* [remove fluid] PRelu BilinearTensorProduct Conv2DTranspose SequenceConv RowConv
@paddle-bot
Copy link

paddle-bot bot commented Dec 20, 2022

See the preview results: http://preview-paddle-pr-48507.paddle-docs-preview.paddlepaddle.org.cn/documentation/docs/en/api/index_en.html
You may need to wait a while for the preview task to complete.
For more information about the preview tool: PaddlePaddle document preview tool

@luotao1
Copy link
Contributor

luotao1 commented Dec 21, 2022

LGTM for PR-CI-Codestyle-Check, 分布式相关的C++代码格式问题 @danleifeng 后续会找其他同学修,本次先豁免

2022-12-21 08:21:30 paddle/fluid/distributed/ps/service/ps_local_client.h:186:  "virtual" is redundant since function is already declared as "override"  [readability/inheritance] [4]
2022-12-21 08:21:30 paddle/fluid/distributed/ps/service/ps_local_client.h:199:  "virtual" is redundant since function is already declared as "override"  [readability/inheritance] [4]
2022-12-21 08:21:30 Done processing paddle/fluid/distributed/ps/service/ps_local_client.cc
2022-12-21 08:21:30 Done processing paddle/fluid/distributed/ps/service/ps_local_client.h
2022-12-21 08:21:30 Total errors found: 30
2022-12-21 08:21:30 paddle/fluid/distributed/ps/table/ssd_sparse_table.cc:552:  Missing username in TODO; it should look like "// TODO(my_username): Stuff."  [readability/todo] [2]
2022-12-21 08:21:30 paddle/fluid/distributed/ps/table/ssd_sparse_table.cc:581:  Missing username in TODO; it should look like "// TODO(my_username): Stuff."  [readability/todo] [2]
2022-12-21 08:21:30 Done processing paddle/fluid/distributed/ps/table/ssd_sparse_table.cc
2022-12-21 08:21:30 Total errors found: 2
2022-12-21 08:21:30 paddle/fluid/framework/fleet/heter_ps/test_cpu_query.cu:31:  Static/global string variables are not permitted.  [runtime/string] [4]
2022-12-21 08:21:30 Done processing paddle/fluid/framework/fleet/heter_ps/test_cpu_query.cu
2022-12-21 08:21:30 Total errors found: 1

Copy link
Contributor

@yaoxuefeng6 yaoxuefeng6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@lanxianghit lanxianghit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for Flags

@phlrain phlrain self-requested a review December 21, 2022 07:53
Copy link
Contributor

@zhiqiu zhiqiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for const_cast

@yaoxuefeng6 yaoxuefeng6 merged commit 1acddc3 into PaddlePaddle:develop Dec 21, 2022
@qili93
Copy link
Contributor

qili93 commented Jan 3, 2023

@yaoxuefeng6 这个 PR 关于 rocksdb.cmake 的修改 broken 了 Paddle 在 aarch64 环境下的源码编译功能,辛苦修复下!

这里 rocksdb.cmake 里面的 sse 以及 -mpclmul 相关的编译选项都是 x86 环境下独有的,在 aarch64下会导致编译失败

image

rocksdb 本身是支持 aarch64 平台下的编译的,请根据 rocksdb 的 cmake 自动根据编译环境来选择编译选项,代码见 https://github.com/Thunderbrook/rocksdb/blob/main/CMakeLists.txt#L261

image

当前 work-around 办法是在 aarch64 编译时加上 WITH_PSCORE=OFF 来关闭对于 rocksdb 的编译和引用。

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
contributor External developers
Projects
None yet
Development

Successfully merging this pull request may close these issues.