Process isAssertion failed in file ../../src/mpid/ch4/shm/posix/eager/include/intel_transport_recv.h at line 1160: cma_read_nbytes == size

[JunxiChhen]

Issue occured:

When running llama2-7b, input4096, output2048, BS16, Beam1, on SPR-HBM flat mode SNC4.
Benchmarking CMD:

[pujiang2018]

Looks something wrong with oneCCL. @JunxiChhen Do we still encounter such issue?

[JunxiChhen]

Looks something wrong with oneCCL. @JunxiChhen Do we still encounter such issue?

Yes. But it only occurred on SPR-HBM snc4 flat mode now. I didn't see any issue on SPR Quad mode.

[pujiang2018]

@shanzhou2186 Have you ever encountered such issue? for such issue, need an environment to debug.

[shanzhou2186]

Have you checked the memory usage on each sub-numa? Is it possible that one of sub numa OOM？

[shanzhou2186]

@shanzhou2186 Have you ever encountered such issue? for such issue, need an environment to debug.

No. I haven't tried such large input/output before.

[zongy17]

@JunxiChhen Hi, I also got this error when using Intel OneAPI (version 2023.2.0). Did you finally know how to avoid this bug?