Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Prevent GPU particle communication/removal bug when all particles are in the guard cells #333

Merged
merged 1 commit into from
Mar 5, 2019

Conversation

RemiLehe
Copy link
Member

@RemiLehe RemiLehe commented Mar 5, 2019

There is currently a bug in the particle communication/removal routines on GPU when all particles of a given species are on the right-hand guard cells: in this case the particles are not removed from the local domain.

Fundamentally, this is because the code computes the indices i_min and i_max between which particles should be kept. When all particles are in the right hand guard cells, then i_min=0 and i_max=0. But then a special-case portion of the code (removed in this PR) erroneously sets i_max=Ntot.

This special part of the code used to here because the last cells of prefix_sum used to be filled with 0. However this is no longer the case since we introduced prefill_prefix_sum in #110.

Therefore that erroneous portion can be safely removed.

@RemiLehe
Copy link
Member Author

RemiLehe commented Mar 5, 2019

Here is a reproducer for the bug. When using dev, the particle output shows the unremoved particles at all the iterations. With this PR, there are no particles after the first iteration.

# -------
# Imports
# -------
import numpy as np
from scipy.constants import c
# Import the relevant structures in FBPIC
from fbpic.main import Simulation
from fbpic.lpa_utils.bunch import add_elec_bunch_gaussian
from fbpic.openpmd_diag import ParticleDiagnostic

# ----------
# Parameters
# ----------
# Whether to use the GPU
use_cuda = True

# The simulation box
Nz = 400         # Number of gridpoints along z
zmax = 10.e-6    # Right end of the simulation box (meters)
zmin = -10.e-6   # Left end of the simulation box (meters)
Nr = 50          # Number of gridpoints along r
rmax = 20.e-6    # Length of the box along r (meters)
Nm = 1           # Number of modes used

# The simulation timestep
dt = (zmax-zmin)/Nz/c   # Timestep (seconds)

# The diagnostics
diag_period = 1         # Period of the diagnostics in number of timesteps

# ---------------------------
# Carrying out the simulation
# ---------------------------

# NB: The code below is only executed when running the script,
# (`python lwfa_script.py`), but not when importing it (`import lwfa_script`).
if __name__ == '__main__':

    # Initialize the simulation object
    sim = Simulation( Nz, zmax, Nr, rmax, Nm, dt, boundaries='open',
            zmin=zmin, use_cuda=use_cuda, exchange_period=1 )

    add_elec_bunch_gaussian(sim, sig_r=0.1e-6, sig_z=0.1e-6, n_emit=0.,
                            gamma0=1000., sig_gamma=0., Q=1.e-12, N=100,
                            zf=2.e-6)

    # Shift the electron so that it is in the guard cells
    sim.ptcl[1].z += 15.e-6
    
    # Add diagnostics
    sim.diags = [ ParticleDiagnostic( diag_period, {"bunch" : sim.ptcl[1]}, comm=sim.comm) ]

    ### Run the simulation
    sim.step( 3 )

@MKirchen
Copy link
Contributor

MKirchen commented Mar 5, 2019

Thanks for catching this :)

@MKirchen MKirchen merged commit 2e3c93a into fbpic:dev Mar 5, 2019
@RemiLehe RemiLehe deleted the fix_part_comm branch May 30, 2020 20:17
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants