Ucc integration #591

kaiyingshan · 2022-06-23T06:10:47Z

No description provided.

nirandaperera · 2022-06-25T15:41:10Z

@kaiyingshan I'm getting the following seg fault. I can't figure out where and why it is coming from. I'm using latest UCC master branch.

(cylon_dev) niranda@aurora-r10:~/git/cylon/build$ ./bin/ucc_allgather_example 
[1656170916.449172] [aurora-r10:49843:0]          ucc_cl.c:57   UCC  ERROR no TLs are selected for CL_BASIC
[1656170916.449190] [aurora-r10:49843:0]         ucc_lib.c:127  UCC  ERROR lib_init failed for component: basic
[aurora-r10:49843:0:49843] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x164)
==== backtrace (tid:  49843) ====
 0  /home/niranda/miniconda3/envs/cylon_dev/lib/libucs.so.0(ucs_handle_error+0x2fd) [0x7f3eff20778d]
 1  /home/niranda/miniconda3/envs/cylon_dev/lib/libucs.so.0(+0x2b994) [0x7f3eff207994]
 2  /home/niranda/miniconda3/envs/cylon_dev/lib/libucs.so.0(+0x2bb5a) [0x7f3eff207b5a]
 3  /lib/x86_64-linux-gnu/libc.so.6(+0x43090) [0x7f3f00071090]
 4  /home/niranda/git/ucc/install/lib/libucc.so.1(ucc_collective_init+0x1e0) [0x7f3eff09bf30]
 5  /home/niranda/git/cylon/install/lib/libcylon.so.0.5.0(_ZNK5cylon3ucc21UccTableAllgatherImpl20AllgatherBufferSizesEPKiiPi+0x8e) [0x7f3f005f2b9e]
 6  /home/niranda/git/cylon/install/lib/libcylon.so.0.5.0(_ZN5cylon3net18TableAllgatherImpl7ExecuteERKSt10shared_ptrINS_15TableSerializerEERKS2_INS_9AllocatorEEiPSt6vectorIiSaIiEEPSB_IS2_INS_6BufferEESaISG_EEPSB_ISD_SaISD_EE+0xfc) [0x7f3f008d44dc]
 7  /home/niranda/git/cylon/install/lib/libcylon.so.0.5.0(_ZN5cylon3net18TableAllgatherImpl7ExecuteERKSt10shared_ptrINS_5TableEEPSt6vectorIS4_SaIS4_EE+0x1d1) [0x7f3f008d4ea1]
 8  /home/niranda/git/cylon/install/lib/libcylon.so.0.5.0(_ZNK5cylon3net15UCXCommunicator9AllGatherERKSt10shared_ptrINS_5TableEEPSt6vectorIS4_SaIS4_EE+0x5c) [0x7f3f005f192c]
 9  ./bin/ucc_allgather_example(+0x5f39) [0x55fbb51b8f39]
10  ./bin/ucc_allgather_example(+0x5c69) [0x55fbb51b8c69]
11  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f3f00052083]
12  ./bin/ucc_allgather_example(+0x5e1e) [0x55fbb51b8e1e]
=================================
Segmentation fault (core dumped)

kaiyingshan · 2022-06-29T04:36:02Z

I don't know if this is the same issue that I experienced, which is due to the ucc team size. When I run the example this way, it gives a warning "ucc_team.c:114 UCC WARN minimal size of UCC team is 2, provided 1", and it is caused by ucc_collective_init; when I run with mpirun -n it won't result in segfault.

Sergei-Lebedev · 2022-07-01T06:55:12Z

cpp/src/examples/ucc_allgather_vector_example.cpp

+  std::cout<<std::endl;
+
+  /* Cleanup UCC */
+  UCC_CHECK(ucc_team_destroy(team));


team destroy is nonblocking operation, it might return UCC_INPROGRESS, should be something like this

ucc_status_t status; while (UCC_INPROGRESS == (status = ucc_team_destroy(team.team))) { if (UCC_OK != status) { std::cerr << "ucc_team_destroy failed\n"; break; } }

Sergei-Lebedev · 2022-07-01T07:04:32Z

cpp/src/cylon/net/ucx/ucx_communicator.cpp

+
+  RETURN_CYLON_STATUS_IF_UCC_FAILED(ucc_context_config_read(lib, nullptr, &ctx_config));
+  RETURN_CYLON_STATUS_IF_UCC_FAILED(ucc_context_create(lib, &ctx_params, ctx_config, &uccContext));
+  while (UCC_OK != ucc_context_progress(uccContext)) {


ucc_context_create is blocking, no need to call ucc_context_progress

Sergei-Lebedev · 2022-07-01T07:10:07Z

I don't know if this is the same issue that I experienced, which is due to the ucc team size. When I run the example this way, it gives a warning "ucc_team.c:114 UCC WARN minimal size of UCC team is 2, provided 1", and it is caused by ucc_collective_init; when I run with mpirun -n it won't result in segfault.

recently we added support for team size 1 (openucx/ucc#511)

…into ucc-integration � Conflicts: � cpp/src/cylon/net/ucx/ucx_communicator.cpp

…into ucc-integration

…ation # Conflicts: # cpp/src/cylon/net/ucx/ucx_communicator.cpp

nirandaperera · 2022-07-12T16:29:57Z

I don't know if this is the same issue that I experienced, which is due to the ucc team size. When I run the example this way, it gives a warning "ucc_team.c:114 UCC WARN minimal size of UCC team is 2, provided 1", and it is caused by ucc_collective_init; when I run with mpirun -n it won't result in segfault.

recently we added support for team size 1 (openucx/ucc#511)

@Sergei-Lebedev I'm still getting the following error with team size 1.
I opened an issue in ucx regarding this openucx/ucc#567

(cylon_dev) niranda@aurora-r10:~/git/cylon$ ./build/bin/ucc_example 
[1657643132.702496] [aurora-r10:188948:0]   cl_basic_team.c:131  CL_BASIC ERROR no tl teams were created
[1657643132.702508] [aurora-r10:188948:0]        ucc_team.c:294  UCC  ERROR No CL teams were created
failed to create ucc team
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -6.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
(cylon_dev) niranda@aurora-r10:~/git/cylon$ mpirun -n 1 ./build/bin/ucc_example 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -6.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[1657643157.590587] [aurora-r10:189016:0]   cl_basic_team.c:131  CL_BASIC ERROR no tl teams were created
[1657643157.590601] [aurora-r10:189016:0]        ucc_team.c:294  UCC  ERROR No CL teams were created
failed to create ucc team

…into ucc-integration

nirandaperera · 2022-07-13T20:38:08Z

@kaiyingshan I reviewed your code and made some changes myself in this commit 9a7e9aa

Could you please check that?

nirandaperera · 2022-07-14T12:10:07Z

@kaiyingshan thank you for doing this

kaiyingshan added 4 commits June 19, 2022 21:11

initial commit

a54c345

allreduce

5de0358

start on allgather

d9af4fd

use wait logic in ucc tests

797c529

kaiyingshan requested a review from nirandaperera June 23, 2022 06:11

kaiyingshan added 2 commits June 30, 2022 17:09

add allgather vector example

c70eda1

improve ucc ctx/team initialization

c480440

Sergei-Lebedev reviewed Jul 1, 2022

View reviewed changes

kaiyingshan and others added 19 commits July 1, 2022 02:29

fixed missing data issue in allgatherv

bb5d9d0

bcast

26a526f

allgather

570886b

add a test for implemented operators

6020b59

make ucc team with one process works

6f088cc

addressed blocking/non-blocking in ucc functions

af4ac43

removing depricated method

7e62805

enable tests for ucx/ucc

2f37d4d

adding Make method to communicator API

80ee656

Merge branch 'ucc-integration' of https://github.com/cylondata/cylon …

9482238

…into ucc-integration � Conflicts: � cpp/src/cylon/net/ucx/ucx_communicator.cpp

minor cmake change

450b114

avoid copy by placing count vector outside loop

813b184

Merge branch 'ucc-integration' of https://github.com/cylondata/cylon …

942a5f4

…into ucc-integration

fix double free issue

8b4baea

using BUILD_CYLON_UCC def

01562e2

Merge remote-tracking branch 'origin/ucc-integration' into ucc-integr…

d78ad01

…ation # Conflicts: # cpp/src/cylon/net/ucx/ucx_communicator.cpp

fixing openmpi at 4.1.2

1d10796

fixing openmpi at 4.1.3 ha1ae619_105 build

cfb47e8

fixing openmpi at 4.1.3 ha1ae619_105 build

caefb81

fixing double free error

eedf09b

nirandaperera and others added 9 commits July 12, 2022 19:17

fixing ucx tests

72da6af

fixing ucx tests

3b34219

fixing gcylon

747735a

fixing java build

71bbe55

minor fix

fcdbe14

fix warnings

76229d6

Merge branch 'ucc-integration' of https://github.com/cylondata/cylon …

65eb44f

…into ucc-integration

fixing ucx world size 1

d2c2cce

Merge branch 'ucc-integration' of https://github.com/cylondata/cylon …

830853c

…into ucc-integration

nirandaperera marked this pull request as ready for review July 13, 2022 15:23

nirandaperera added 2 commits July 13, 2022 13:42

changing branch

610585b

review changes

9a7e9aa

nirandaperera closed this Jul 13, 2022

nirandaperera reopened this Jul 13, 2022

resolve conflict

eb73788

nirandaperera approved these changes Jul 14, 2022

View reviewed changes

nirandaperera merged commit 4dd359f into main Jul 14, 2022

nirandaperera mentioned this pull request Jul 15, 2022

build: UCC integration #575

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ucc integration #591

Ucc integration #591

kaiyingshan commented Jun 23, 2022

nirandaperera commented Jun 25, 2022

kaiyingshan commented Jun 29, 2022

Sergei-Lebedev Jul 1, 2022

Sergei-Lebedev Jul 1, 2022

Sergei-Lebedev commented Jul 1, 2022

nirandaperera commented Jul 12, 2022 •

edited

Loading

nirandaperera commented Jul 13, 2022

nirandaperera commented Jul 14, 2022

Ucc integration #591

Ucc integration #591

Conversation

kaiyingshan commented Jun 23, 2022

nirandaperera commented Jun 25, 2022

kaiyingshan commented Jun 29, 2022

Sergei-Lebedev Jul 1, 2022

Choose a reason for hiding this comment

Sergei-Lebedev Jul 1, 2022

Choose a reason for hiding this comment

Sergei-Lebedev commented Jul 1, 2022

nirandaperera commented Jul 12, 2022 • edited Loading

nirandaperera commented Jul 13, 2022

nirandaperera commented Jul 14, 2022

nirandaperera commented Jul 12, 2022 •

edited

Loading