-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
NCCL Library Integration #156
Conversation
657995b
to
94b52af
Compare
@@ -118,6 +118,7 @@ function(caffe_print_configuration_summary) | |||
caffe_status(" USE_LEVELDB : ${USE_LEVELDB}") | |||
caffe_status(" USE_LMDB : ${USE_LMDB}") | |||
caffe_status(" ALLOW_LMDB_NOLOCK : ${ALLOW_LMDB_NOLOCK}") | |||
caffe_status(" USE_NCCL : ${USE_NCCL}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add this down below with the other CUDA options?
https://github.com/NVIDIA/caffe/blob/v0.15.1/cmake/Summary.cmake#L144-L148
CMake didn't auto-detect and use the deb package installation of NCCL automatically. I really like that feature for cuDNN (one less flag to set). |
@lukeyeager NCCL does not come as a Deb package. This is old school |
@@ -424,7 +424,7 @@ RegisterBrewFunction(time); | |||
|
|||
int main(int argc, char** argv) { | |||
// Print output to stderr (while still logging). | |||
FLAGS_alsologtostderr = 1; | |||
FLAGS_alsologtostderr = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this causing a problem?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, just a purist habit... :) This flag is boolean.
https://github.com/NVIDIA/nccl/releases
It can find it, but it doesn't use it until I specify |
Agreed, if found it should be the default. If not found it should probably warn at that point that multi gpu will be disabled.
|
94b52af
to
6b4d102
Compare
Notes from Luke and Simon have been addressed, thank you. |
# ---[ NCCL | ||
if(USE_NCCL) | ||
add_definitions(-DUSE_NCCL) | ||
find_package(NCCL REQUIRED) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now I can't build Caffe with CUDA unless I have NCCL. I'd like to see this behavior (same as with USE_CUDNN):
USE_NCCL | found NCCL? | use? |
---|---|---|
On |
Yes | Yes |
On |
No | ERROR |
unset | Yes | Yes |
unset | No | No |
Off |
* | No |
6b4d102
to
f43e15e
Compare
|
@lukeyeager yes, I've seen it and fixing it now. |
bb8c03f
to
ad43d82
Compare
ad43d82
to
97d59ad
Compare
All redundant changes removed, CMake is green. |
No description provided.