Presence of chaotic clusters in longitudinal phase space with MPI run

[Litong-thu]

When I run the puffin with mpi method, the particle dump seems to have chaotic cluster areas, and the number of the clusters is equal to my parallel computing number.
And I notice that the chaotic cluster distribution is very similar to the buffer area of Puffin MPI. I tried to change the bufferlength but nothing different.

The case in simulating is the FEL process of electron bunch trains, in which the micro-bunches are spacing at the resonant wavelength.

[Litong-thu]

here is my slurm input code

#!/bin/bash
#SBATCH -J np-4
#SBATCH -p cnall
#SBATCH -N 1
#SBATCH -o puffin_stdout.%j.out
#SBATCH -e puffin_stderr.%j.err
#SBATCH --ntasks-per-node=4
#SBATCH --no-requeue

module load libs/hdf/hdf-1.10.2-intel2019
module load compilers/intel/2019/v2019-all
module load libs/fftw/v3.3.8-intel2019
mpirun -np 4 puffin thz_undulator.in

[Litong-thu]

&MDATA
!
! Flags:
!
qScaled = .false.
qOneD = .false.
qFieldEvolve = .true.
qElectronsEvolve = .true.
qElectronFieldCoupling = .true.
qFocussing = .false.
qDiffraction = .true.
qUndEnds = .true.
qFilter = .true.
q_Noise = .false.
qsdds = .false.
qhdf5 = .true.
!
! File pointers:
!
beam_file = 'beamfile_h5.in'
lattFile = 'thz_undulator.latt'
!
! Mesh setup:
!
meshType = 0
sFModelLengthX = 7E-1
sFModelLengthY = 7E-1
sFModelLengthZ2 = 4e-11
nodesPerLambdar = 20
iNumNodesX = 151
iNumNodesY = 151
iRedNodesX = 21
iRedNodesY = 21
nspinDX = 41
nspinDY = 41
!
! Scaling frame:
!
srho = 0.005
!
! Electron beam:
!
sElectronThreshold = 0
sgamma_r = 48.923679
!
! Lattice:
!
saw = 2.847235
lambda_w = 0.1
nPeriods = 50
zundType = 'planepole'
!
! Numerical options:
!
stepsPerPeriod = 15
sFiltFrac = 0.3
sDiffFrac = 2.0
sRedistLen = 64
iRedistStp = 10
!
! Writing options:
!
iWriteNthSteps = 50
iWriteIntNthSteps = 50
ioutinfo = 3
qDumpEnd = .false.
/

[mightylorenzo]

Hmm interesting...so sometimes with certain MPI wrappers on certain machines, I'd have similar problems with the shot-noise in the beam - so the randomness was not globally random, but repeated in each MPI process. Of course, this shouldn't happen, but in some dodgy compilers/MPI versions, it seemed to arise.

From what I can see here, you're using a beam input file, so the beam has already been generated. You could try doing a simple 1D run with the version of Puffin you have compiled, just for a few periods, and seeing if you can notice identical radiation power slices, which look random within the MPI process, but are actually repeated from process to process. That would indicate you should try to rebuild Puffin with a different compiler/MPI lib version. Do you know if you notice the same deviation in behaviour if you switch off diffraction - i.e. check that the single process run behaviour is the same as the multi-process run. If they don't diverge then that at least narrows it down to the diffraction simulation in some way...in which case it may be problems with the FFTW lib, or something deeper in Puffin that's emerging for your use case (I wonder if tuning the high-pass filter, or increasing the transverse mesh size etc would help)...

[mightylorenzo]

If you're able to get the beam file and lattice file to me then I can try to see if I can reproduce it...

Are you able to also post plots of the radiation power?

[Litong-thu]

Hmm interesting...so sometimes with certain MPI wrappers on certain machines, I'd have similar problems with the shot-noise in the beam - so the randomness was not globally random, but repeated in each MPI process. Of course, this shouldn't happen, but in some dodgy compilers/MPI versions, it seemed to arise.

From what I can see here, you're using a beam input file, so the beam has already been generated. You could try doing a simple 1D run with the version of Puffin you have compiled, just for a few periods, and seeing if you can notice identical radiation power slices, which look random within the MPI process, but are actually repeated from process to process. That would indicate you should try to rebuild Puffin with a different compiler/MPI lib version. Do you know if you notice the same deviation in behaviour if you switch off diffraction - i.e. check that the single process run behaviour is the same as the multi-process run. If they don't diverge then that at least narrows it down to the diffraction simulation in some way...in which case it may be problems with the FFTW lib, or something deeper in Puffin that's emerging for your use case (I wonder if tuning the high-pass filter, or increasing the transverse mesh size etc would help)...

Thank you so much for your suggestions! I've tried all the simulation variables you mentioned above, but the chaotic clusters still remained.

But inspired by you, I ran the same puffin code with same input particle file on another machine, and I got the simulation results without those randomness.

So I think I shall first recompile the puffin with other compilers or MPI wrappers on my sever, and I'll post my new results here as fast as possible.