Alma Mater Studiorum · Università di Bologna
FACOLTÀ DI SCIENZE MATEMATICHE, FISICHE E NATURALI Corso di Laurea in Fisica
Development of algorithms
for an electromagnetic particle in cell code and implementation on a hybrid
architecture (CPU+GPU)
Tesi di Laurea Magistrale in Fisica
Relatore:
Chiar.mo Prof.
Giorgio Turchetti
Presentata da:
Francesco Rossi
Sessione II
Anno Accademico 2010-2011
Indice
Acknowledgements 1
Introduction 2
Summary 5
1 The Physics of plasma based laser accelerators 6
1.1 Basic plasma physics . . . 6
1.1.1 Debye Length and plasma parameter . . . 7
1.1.2 Kinetic and fluid plasma descriptions . . . 9
1.1.3 Collective motion: plasma electron waves . . . 11
1.1.4 The collision frequency . . . 12
1.1.5 Collisionless limit and the plasma parameter Λ . . . 12
1.1.6 Electromagnetic waves propagation in a cold plasma . . . 14
1.2 Laser plasma acceleration physical concepts and regimes . . . 16
1.2.1 Electron acceleration: LWFA . . . 16
1.2.2 Electron acceleration: bubble regime . . . 18
1.2.3 Ion acceleration: TNSA and RPA . . . 21
2 The numerical modelization 27 2.1 The particle-in-cell method . . . 27
2.1.1 Phase space representation . . . 27
2.1.2 Passes of an electromagnetic PIC code and numerical parameters of a laser plasma interaction simulation . . . 30
2.1.3 Interpolation and deposition using shape functions . . . 31
2.1.3.1 Force interpolation . . . 31
2.1.3.2 Charge and current deposition . . . 32
2.1.3.3 Common shapefunctions and shapefactors . . . 32
2.2 The “standard” second-order PIC: leapfrog and FDTD . . . 38 2.2.1 Solving the Maxwell equations numerically using the Yee Lattice . 38
ii
2.2.2 Boris pusher . . . 42
2.2.3 Bringing all togheter . . . 44
2.3 Charge conservation . . . 45
2.3.1 Spectral Poisson corrector . . . 45
2.3.2 Esirkepov Shape functions . . . 47
2.4 Envelope model . . . 49
2.4.1 Laser envelope equation derivation . . . 50
2.4.2 Plasma motion and wakefield equations in cylindrical comoving coordinates . . . 51
3 INF&RNO 54 3.1 Motivation . . . 54
3.2 Numerical Scheme . . . 56
3.2.1 Plasma modelization . . . 56
3.2.1.1 Fluid plasma model . . . 56
3.2.1.2 PIC plasma model . . . 56
3.2.1.3 Numerical choices . . . 56
3.3 Parallelization . . . 57
3.3.1 Shared memory (OpenMP) . . . 60
3.3.2 Message-Passing (MPI) . . . 61
3.3.2.1 1D domain decomposition . . . 62
3.3.2.2 2D domain decomposition . . . 64
3.3.3 GPU parallelization . . . 66
3.4 3D Visualization . . . 66
4 Jasmine: a flexible, hybrid (CPU+GPU), PIC framework 70 4.1 Structure of the framework . . . 71
4.2 GPU Parallelization . . . 72
4.2.1 GPU architecture and CUDA programming model . . . 72
4.2.1.1 Throughput . . . 72
4.2.1.2 Parallelism and memory model (hierarchy) . . . 74
4.2.1.3 Programming GPUs: CUDA libraries and our multi ar- chitecture approach . . . 76
4.2.2 Our approach for writing cross architecture code . . . 78
4.2.3 Current and density deposition algorithm . . . 79
4.3 Hybrid parallelization . . . 80
4.3.1 Inter-node & intra-node parallelization . . . 80
Indice
4.3.2 Inter-node communication . . . 83 4.4 Performance benchmarks . . . 83
Conclusions 85
iv
For giving me the opportunity of performing the work in this thesis, I would like to thank my supervisor Prof. Giorgio Turchetti, a person of reference during all the period of my studies.
Particular thanks are for Dr. Carlo Benedetti, who has introduced and guided me, with contagious enthusiasm, through this field of research and gave me the possibility to collaborate, for a while, with the LOASIS group at Lawrence Berkeley National Lab, one of the excellences in the field. I also thank him personally for the great time he made me spend in Berkeley.
During this period, it has always been a pleasure to work with both the groups in Bologna and in Berkeley, and I want to give a special thank to all the people I have been working with, whose essential advices and support have been always fundamental and encouraging. It was a pleasure to meet my supervisor in Berkeley Carl Schroeder and to collaborate with Andrea Sgattoni and Pasquale Londrillo, who helped me with incredible altruism.
It has been an honor and incredibly motivating to work for experts of the likes of Wim P.
Leemans, Eric Esarey and Cameron Geddes. It has been a pleasure to be in contact with Professor Graziano Servizi, Nicholas Matlis, Daniel Bakker, Lule Yu and Joshua Renner, who gave me the chance for very fruitful discussions, though not always centered on theme of this thesis.
I thank CINECA, and in particular Carlo Cavazzoni and Riccardo Brunino, for showing great interest in my work and giving me the possibility to benchmark my codes on their new machine PLX.
Sofia, Emma and Rolando have been always next to me and the support they have always been giving me is invaluable.
I acknowledge support by the INFN, the “Collegio Superiore dell’Università di Bologna”
and by the Department of Energy under the Office of Science contract No. DE-AC02- 05CH11231.
Introduction
The development of laser-plasma accelerators was inspired by the work of Tajima and Dawson [32, 1979] and it was boosted by the rapid upgrowth of high intensity laser systems, which allow to explore interactions that drive electrons up to ultra-relativistic velocities. The particle beams generated in such accelerators are expected to find appli- cation in a wide range of contexts, including high energy physics, proton therapy for the treatment of cancers and generation of intense X-ray radiation.
Plasma-based particle acceleration is of great interest because of their capability to sus- tain extremely high acceleration gradients. The accelerating electrical field in a con- ventional, radio frequency, linear accelerator (linac) is limited, for structure breakdown reasons, to approximately∼ 100MV/m. A ionized plasma (having electron number den- sity n0), instead, can sustain electron oscillations that generate electric wakefields that can exceed the non-relativistic wave-breaking limit:
E0(V /m) ≃ 96p
n0(cm−3) (0.1)
This limit can be various orders of magnitudes greater than the limit in a conventional linac, for example n0 = 1018cm−3 yields E0 = 100GV /m. Wakefields of this order, generated by the plasma electron waves, are able to accelerate electrons and static fields can be used to accelerate protons and ions.
New laser technologies are able to provide high power (P W scale) and ultra-short (∼
10f s) pulses. These features make lasers ideal acceleration drivers, as they can be used for exciting these plasma electron waves in quasi-resonance conditions.
Experiments have demonstrated, so far, the acceleration of ultra-short, monochromatic electron bunches up to 1GeV ([1]) and ion bunches up to ∼ 100MeV . Electron acceler- ation up to 10GeV in ∼ 1m of plasma, is currently under development at LBNL by the LOASIS group ([8],[11]).
2
The theoretical study of laser plasma interaction phenomena requires to solve systems of nonlinear partial differential equations which, in general, can not be solved analytically.
Numerical modeling and computer simulation codes are therefore fundamental tools for understanding the Physics of laser-plasma acceleration and for supporting the design of the experiments. The use of simulations, in particular the so called particle-in-cell codes, has made possible to discover new physical regimes (as the bubble one) and has had, in general, a very significant impact in designing, optimizing and understanding laser-plasma accelerators.
These simulations are usually very demanding from the point of view of the required CPU time and memory. In fact, full dimensionality (3D) models are required for describing the dynamics correctly and it is required to resolve many physical length scales, which can differ by various orders of magnitude.
Given the crucial importance of simulations in this field, the effort for making them more accurate and faster (which means, in most of the cases, possible) is definitely worth.
Those objectives can be achieved via efficient code parallelization and numerical scheme improvement.
Since 2007, our group in Bologna has been developing an MPI-parallel particle in cell code, ALaDyn ([5], acronym for Acceleration by Laser and Dynamics of charged parti- cles), featuring high order integration (Runge-Kutta 4) and derivative schemes, which allowed to reduce computational requirements. It has been used for modeling the INFN laser plasma acceleration experiments LILIA (proton acceleration) and PLASMON-X (electrons acceleration).
Furthermore, customized codes for specific acceleration regimes have been developed.
The INF&RNO code framework, developed by Carlo Benedetti at Lawrence Berkeley National Laboratory, is designed to simulate laser plasma electron accelerators that work in underdense plasma regime, which, in certain cases, allows the use of the laser envelope approximation and the boosted Lorentz frame.
In this thesis I will discuss the parallelization of the INF&RNO code - which required to find some original and physically guided solutions for solving some recursive dependencies in the numerical scheme - and my implementation (in modern C++) of a generic particle in cell codes framework, named jasmine, in which I have implemented some of the schemes of ALaDyn and INF&RNO, capable to run full 3D PIC simulations on (general-
Indice purpose-)graphics processing unit (GP)GPU clusters.
The GPGPU architecture, developed in the recent years, represents a very good opportu- nity for efficient parallelization. It implements the parallelism at the very chip level and it provides a parallel-efficient memory hierarchy, hundreds of lightweight cores (providing more floating point computational power) and more memory bandwidth compared to a conventional processor (CPU).
Many among the world’s largest high performance computing clusters (4 in the worldwide top ten while writing this thesis) have, already, most of their computational power in their GPU nodes, as the new machine PLX at CINECA, which was used for jasmine benchmarking.
Furthermore, in the last years, the development of manycore, shared memory, architec- tures has been the only way to increase processors’ computational power, and therefore, in the future, they are likely to increase in importance. Parallelizing for these archi- tectures is not trivial: for a particle-in-cell code it is required to write some algorithms (such as densities deposition) using original and completely different approaches respect to their serial implementation.
My work for developing a GPU version of an electromagnetic particle in cell code is giving us the possibility to run the simulations exploiting the efficiency and power of such clusters, which are providing us great speedups: the GPU code, on a single GPU, is up to 50x faster than the CPU version on a single CPU core. Furthermore it makes our simulations ready for the (likely) upcoming manycore era, in which the CPUs and GPUs architectures will continue to converge.
At the moment of writing this thesis, no published article has claimed the development of an electromagnetic PIC code that can run on hybrid clusters using the main CPU memory efficiently, or that can run multi-gpu full-PIC 3D simulations, or that is easily adaptable for various geometries and numerical schemes as jasmine.
4
In this thesis I will discuss:
Chapter1 The physical background of laser plasma interactions, from the analytical theory, which derives the fluid model and explains linear oscillation modes and plasma parameters, to the nonlinear laser-plasma acceleration regimes, explained with some simulations run with my code jasmine (chapter 4).
Chapter2 The numerical solutions for the integration of the Maxwell Vlasov equations:
the particle-in-cell method, the numerical integration of the Maxwell equations in time, the standard second order PIC scheme, the charge conservation issues and the laser envelope approximation.
Chapter3 My work parallelizing and improving with 3D visualization of INF&RNO, an efficient code, developed at LBNL, designed for modeling underdense-plasmas laser interactions.
It assumes cylindrical symmetry, allows to run the simulations in a reference frame that balances the length scales (the so called Boosted Lorentz frame) and uses the laser envelope model, integrated implicitly. Given these unique features of the code, its parallelization required to find some original and physically guided solutions for solving some recursive dependencies in the numerical scheme (which make parallelization impossible).
Chapter4 Jasmine, my particle-in-cell code and framework, designed to be as flexible and reusable as possible, in order to reduce code rewrites for developing different schemes to the minimum.
It is parallelized for running 1D/2D/3D simulations on graphics processing units CUDA and hybrid (CPU+GPU nodes) HPC clusters, achieving very promising performance results: the order of the (1 GPU)/(CPU core) speedup can be of 50x. A standard, second order, PIC scheme and the INF&RNO scheme were implemented in the framework. At the moment of writing this thesis, no published article has claimed the development of a full electromagnetic PIC code with such features.
1 The Physics of plasma based laser accelerators
1.1 Basic plasma physics
A plasma is basically a fully ionized gas. The free charges in a plasma make it very reactive to electromagnetic solicitations. Plasmas exist in various forms, which can be different, in density, temperature or dimensions, of various orders of magnitude. However, their properties and behaviors are extremely scalable across these differences.
Dynamically, a plasma is a statistically relevant number of charged particles, interacting with and generating electromagnetic fields. In principle, the dynamics of a plasma is fully determined considering that: the force acting on each relativistic particle is the Lorentz force and the electromagnetic fields evolution is governed by the Maxwell equations. In CGS ( xi , pi= miγivi are the position and momentum of the i-th particle):
˙
xi = vi
˙pi = qi
E(xi) +vi×B(xc i) (1.1)
∇ B = 0
∇ E = 4πρ
∇ × B − 1c∂E∂t = 4pic j
∇ × E + 1c∂B∂t = 0
(1.2)
In the latter ones, the sources ρ and j are calculated starting from the particles’ phase space distribution without doing any spatial average operation: this guarantees to include in the model the binary collisions.
This approach is very impractical, both for what concerns numerical simulations (the number of the particles involved is not realistically computable) and analytical theory
6
(the nonlinear partial differential equations describing the physics are not solvable ana- lytically).
A cascade of physical considerations and approximations lead to a cascade of simplified models.
The first step can be to neglect the collisional effects in the kinetic, phase space density plasma description. The validity of the collisionless model is evaluated considering the Debye length and its relations with other plasma parameters.
The collisionless model of the plasma can be further simplified to a fluid model, if the phase space distribution can be considered as single valued for each point in space:
for each position in space the velocity is defined univocally. The fluid model can be used to describe various plasma oscillation modes, linear and nonlinear, but it can not include wavebreaking phenomena, where different particles in the same point in the the configuration space have different velocities.
1.1.1 Debye Length and plasma parameter
We can consider an hydrogen-like fully ionized plasma, in which we label electron and ion densities as ne and ni respectively.
The plasma is near to thermal equilibrium (at temperature T) and, in the unperturbed state, we consider ne= ni= n0 everywhere.
If we perturb the system by adding a discrete point charge Q(> 0), we see that it will attract the electrons and repel the ions, making a cloud of net negative charge that shields the point charge’s electric field and potential. The electrons are prevented from collapsing onto the point charge (to fully neutralize it) by their thermal energy.
We can estimate the amplitude of this shielding effect. The electrostatic potential affects the distribution function for the electrons and other species and the Boltzmann factor e−E/kT becomes:
fj(r, v) = exp
−mv2 2KT
exp
qjΦ kTj
(1.3)
We can then write the electron density as:
nj(r) = ˆ
fj(r, v)d3v = n0exp
−qjΦ kTj
(1.4)
1 The Physics of plasma based laser accelerators We assume that:
• qjΦ << kTj for each species j, so we can expand nj as:
nj = n0j
1 − qjΦ kTj
(1.5) In CGS, Poisson equation for electrostatic potential Φ reads:
∇2Φ = −4πρ = −4πX
j
qjnj− 4πQδ(r) (1.6)
Substituting njwe get:
−∇2Φ = 4πX
j
qjn0j
1 − qj
kTj
Φ + 4πQδ(r) (1.7)
Defining the Debye length as
λD = vu
ut 1
4πP
j
h qjn0j
1 −kTqjj
i (1.8)
and exploiting the spherical symmetry
−1 r2
∂
∂r
r2∂Φ
∂r
= Φ
λ2D + 4πQδ(r) (1.9)
we can solve the equation for Φ, giving:
Φ(r) = Q rexp
− r λD
(1.10)
Being mi ≫ me it is reasonable to consider the ions as immobile in most of the cases, especially on short time scales; this allows to rewrite the Debye length as:
λD =
r kT
4πn0ee2 (1.11)
We can interpret the Debye length as the space scale at which the plasma shields the electrostatic potentials generated by single point charges.
The number of particles in a Debye sphere is called the plasma parameter Λ, whose significance is explained in1.1.5:
8
Λ = 4
3nλ3D = 4 3
kT 4πe2
3/2
n−1/2
!
(1.12)
1.1.2 Kinetic and fluid plasma descriptions
The kinetic model of a plasma describes the state of the particles of the system by means of a distribution function fj(x, p, t), which is just the density of particles (of species j) in phase space (x, p = γmv):
n(particles in dxdp) = fj(x, p, t)dxdp (1.13) Macroscopical (observable, fluid) quantities are obtained averaging over momenta:
nj(x) =´ fj(x, p, t)dp particle density njuj(x) =´ vfj(x, p, t)dp mean velocity
[Pkl(x)]j = mj´ vkvlf dp mean pressure
(1.14)
ρj(x) = qj´ fj(x, p, t)dp charge density
jj(x) = qj´ vfj(x, p, t)dp current density (1.15) On a time scale much smaller than collision frequencies, the Boltzmann equation for fj
reduces to the phase space continuity equation:
∂fj
∂t + ∂
∂x ( ˙xfj) + ∂
∂p ( ˙pfj) = 0 (1.16) , which, once coupled with equations of motion of particles (Lorentz force), is named Vlasov equation:
˙ x = v
˙p = qj
E(x) + v×B(x)c (1.17)
∂fj
∂t + v∂fj
∂x + qj
E+v× B c
∂fj
∂p = 0 (1.18)
The electromagnetic fields are given by Maxwell equations, which close the system and make it self-consistent:
1 The Physics of plasma based laser accelerators
∇ B = 0
∇ E = 4πρ
∇ × B − 1c∂E∂t = 4pic j
∇ × E + 1c∂B∂t = 0
(1.19)
In the non-relativistic limit, we can obtain the n-fluid model (each species j is treated as a separate fluid interacting with the others by means of EM fields) of a plasma by considering the momenta of the Vlasov equation. The first two are:
ˆ dp
∂fj
∂t + v∂fj
∂x + qj
E+v× B c
∂fj
∂p
= 0 (1.20)
ˆ dpp
∂fj
∂t + v∂fj
∂x + qj
E+v× B c
∂fj
∂p
= 0 (1.21)
We obtain the spatial continuity equation for the particle density and a fluid equation that describes the motion of the charged fluids.
∂nj
∂t + ∂
∂x (njuj) = 0 (1.22)
nj
∂uj
∂t + uj∂uj
∂x
= njqj
mj
E+uj× B c
− 1 mj
∂
∂x [Pkl]j m
We have therefore reduced the 6-dimensional Vlasov equation to a 3 dimensional fluid equation.
This integration performed considering the momentum of next order (second) would lead to the equation for a pressure tensor P . It is reasonable to restrict to these two equation if further assumptions concerning heat flux and obtaining a thermodynamical state equation.
10
Being the k and ω respectively the typical wavelength and frequency of the system and vj the thermal velocity of the particles of the j − th species, two simplifying assumptions can be:
• if ω/k ≪ vj,thermal, then the heat transfer is so fast that the fluid can be considered isothermal: pj = njθj
• if ω/k ≫ vj,thermal, then the heat flux is negligible, and we can use an adiabatic equation for pressure pj/nγj = constant
1.1.3 Collective motion: plasma electron waves
Using fluid equations we can study a common form of collective motions in plasmas: the charge and electrostatic field oscillation associated with the motion of the electrons.
We consider an 1D, initially uniform, non-relativistic, neutral plasma (ions with Z=1), with massive ions that can be considered as fixed in our process’ timescale. We consider only electrostatic fields and neglect thermal effects.
Under these conditions the fluid equations are simplified to:
∂ne
∂t + ∂
∂x(neue) = 0 (1.23)
∂
∂t(neue) + ∂
∂x(neu2e) = −neeE
me (1.24)
The electrostatic field can be obtained directly using Poisson equation:
∂E
∂x = −4πe (ne− n0ions) (1.25)
Considering small variations (denoted by tilde) in particle density n, the average velocity u and the electrostatic field E and thus linearizing equations, we get:
ne= n0+ ˜n, ue= ˜u (1.26)
∂ ˜E
∂x = −4πe˜n
0 = ∂∂t2˜n2 −nm0e∂ ˜∂xE (1.27)
1 The Physics of plasma based laser accelerators Substituting in the fluid equation we obtain a stationary wave equation:
∂2
∂t2 + ω2pe
˜
n = 0 (1.28)
ωpe= r
4πe2 n0
me = 5.64 × 104 ne
cm−31/2
(1.29) The last equation describes the oscillation of charge density with angular frequency ωpe, the plasma electron frequency.
1.1.4 The collision frequency
In fully-ionized plasma binary particle interactions are mostly due to the Coulomb force.
We can take the order of magnitude of the range of this force as the order of magnitude of the Debye length.
We consider a charged particle (mass m, charge q0, velocity v0) approaching a target particle, at rest, with mass M ≫ m and charge q0.
The particle will not be able to get closer than the distance of minimum approach δ to the target:
mv02 2 = qq0
δ → δ = 2qq0
mv02 (1.30)
so that, when a particles “tries” to overlap the δ region, is deflected at high angles.
Having a population of particles of density n, mean velocity v0, heading for our target particle, the high angle scattering rate can be evaluated computing the flux of particles passing within a radius δ from the target:
νc = (πδ2)nv0 = π 4e2
m2v02nv0=
= e4n
m2v03 ∝ T−3/2
(1.31)
1.1.5 Collisionless limit and the plasma parameter Λ Recalling (CGS units):
12
vc = 4πe4n
m2v03 (1.32)
λD =
r kT
4πn0e2 (1.33)
ωpe = s
4πn0e2
me (1.34)
and
vth= rkT
me (1.35)
then:
ωpe= vth λD = 2π
Tpe (1.36)
A thermal electron travels about a Debye length in a plasma oscillation period. Just as Debye length represents the electrostatic correlation length, so the plasma period plays the role of the electrostatic correlation time.
We consider the ratio between the two frequencies which characterize the problem ωp and νc:
ωp
νc =
4πne2
m
1/2 m2v03 4πe4n
(1.37)
ωpe
νc = 4πΛ (1.38)
and it is apparent how Λ (the number of electrons in a Debye sphere) plays a key role for evaluating the dominance of collisional regimes, connecting the collective motion and the collision timescales.
The collisionless limit, in which the collision timescale is slow compared to the collective phenomena one, can be expressed as (using the plasma parameter definition 1.12):
ωpe νc
= τc
τpe = 4πΛ ≫ 1 (1.39)
1 The Physics of plasma based laser accelerators 1.1.6 Electromagnetic waves propagation in a cold plasma
From the non-relativistic fluid equations, neglecting pressure terms in a cold plasma, one gets:
∂n1
∂t + n0∇ · u1= 0 men0∂v∂t1 = −en0E1
∇ × B1 = −4πec n0u1+1c∂E∂t1
∇ × E1 = −1c∂B∂t1
(1.40)
in which, the suffix 0 indicates the equilibrium quantities and 1 the small fluctuations.
ne= n0+ n1
Expressing E in its plane plane waves Fourier decomposition,
E=X
k
Ekexp(k · r − ωt) (1.41)
considering the first order component in the series, and substituting in the second of the previous equations, one gets u1 = iωme eE1. Substituting the latter in the third equation in 1.40 one obtains:
∇ × B1 = −iω c
1 −ω2pe ω2
E1 (1.42)
Furthermore, taking a time derivative and using the fourth equation (∇ × E1 = ... ) one has:
ω2 c
1 −ω2pe ω2
E1= c∇ × (∇ × E1) (1.43)
, or, using the plane wave expression for E1 on the right hand side:
k× (k × E1) = −ω2 c
1 −ω2pe ω2
E1 (1.44)
Taking, for symmetry reasons, k = kz, the equation can be recast to the form:
14
ω2− ω2pe− k2c2 0 0 0 ω2− ωpe2 − k2c2 0
0 0 ω2− ωpe2
E1x E1y E1z
=
0 0 0
(1.45)
The solutions of this system are the dispersion relations for the electric field linear oscil- lation modes.
• A first solution (previously found) is for longitudinal electric field oscillations (plasma electron waves):
E1x = E1y = 0
ω2 = ωpe2 (1.46)
• Another one describes some transversal waves:
E1z = 0
ω2 = ω2pe+ k2c2 (1.47)
Substituting k from the dispersion equation in the plane wave expression one gets:
E= Ekexp(i(kz − ωt)) = Ekexp
i
q
ω2− ωpe2
c
z
exp (iωt) (1.48)
If ω2− ωpe2 < 0, the spatial term becomes a damping term, with a characteristic length λ
λ = c
qωpe2 − ω2 (1.49)
The plasma skin depth, defined in the ω ≪ ωpelimit as λskin depth= ωcpe =
5.31 × 105n−1/2e
cm represents the length scale at which the plasma damps electromagnetic waves of angular
frequency ω.
This electromagnetic waves non-propagation condition, ω2− ωpe2 < 0, can be expressed also introducing the plasma critical density for EM waves of frequency ω:
nc(ω) = me 4πe2
ω2 (1.50)
If ne > nc the plasma is called “overdense” or “overcritic” and it becomes opaque to the radiation having frequency smaller than ω.
1 The Physics of plasma based laser accelerators
1.2 Laser plasma acceleration physical concepts and regimes
In this section are presented some basic laser-plasma acceleration techniques.
1.2.1 Electron acceleration: LWFA
In the regime named LWFA (Laser WakeField Acceleration), an electron bunch is ac- celerated while it travels in phase with a plasma wakefield, generated by a laser pulse driver.
Qualitatively, when a ultra short laser pulse begins to propagate in a uniform underdense plasma, the ponderomotive force, pushing electrons, creates a longitudinal charge sepa- ration that results in a longitudinal electric field, which pulls back the electrons again.
This results in the formation of a plasma electron wave and the corresponding wakefield.
The quasi-resonance condition:
Llaser ∼= λplasma(nplasma)
2 (1.1)
makes the formation of the wake very efficient. In fact, in these conditions, the pondero- motive force and the wake electric field change sign at the same frequency.
For the laser pulse to propagate and create the wakefield, the plasma must be underdense.
The laser intensity, parametrized by the adimensional parameter a = mceA2 determines the shape of the wakefield: for a ≪ 1 the wakefield is linear, for a ∼ 1 it becomes nonlinear (and the electrons quiver motion becomes relativistic), and for higher a a bubble (see 1.2.2 for further conditions), or blow-out regime, can be achieved.
The electron bunch can be injected externally or generated from the wakefield itself, by wavebreaking phenomena occurring in relativistic conditions.
Analytical models manage to describe the wakefield generation phenomenon, for the linear regime, only if the driver (the laser beam) is assumed nonevolving: it is function of the comoving coordinate ξ = z − vpt vp ∼= vg ≤ c only (vp is the plasma wave phase velocity and vg is the laser group velocity).
On the other hand, nonlinear wakefield generation can be treated analytically only in both the nonevolving driver and one-dimensional approximations: the driver has to be
16
assumed to be broad, kpr⊥≫ 1, where r⊥ is characteristic radial drive beam dimension.
In this case, starting from the relativistic cold fluid equations and the Poisson equation:
∂n
c∂t+ ∇ · (nu/γ) = 0
∂(u − a)/∂ξ = ∇(a − Φ)
∂2Φ
∂ξ2 = k2p
n
n0 − 1, (1.2)
one can get the evolution equation for the electrostatic wakefield potential [12] (γp =
1 − (vp/c)2
−1/2) :
1 k2p
∂2φ
∂ξ2 = γp2
vp
c
"
1 − 1 + a2 γp2(1 + Φ)2
#−1/2
− 1
≃(γp2≫1)
1 + a2 2(1 + Φ)2 −1
2 (1.3)
In general, wakefield generation by an evolving laser pulse can be studied only with numerical simulations. The following simulation results will help illustrating some accel- eration regimes.
The units in all the plots are in CGS and the spatial scale of the density plots is the cell grid index. The longitudinal phase space plots units are cm and MeV per electron or proton. The physical parameters are given in terms of the adimensional laser potential a0= eA/mec2and the density is normalized as n/nc, being ncas in 1.50. All the jasmine simulations were run on one or multiple GPUs.
A first sample of simulation is a 2D one, run with my code jasmine (chapter 4), which illustrates the generation of a linear wakefield . The parameters are:
a0 n/nc λlaser Llaser w0,laser c∆t ∆x = ∆y Grid P.P.C 0.01 0.001 1µm λp/2 λp/6 0.035µm λlaser/20 4096x1024 16 and the longitudinal electrical current plot, showing how the generated linear wakefields looks like (the laser pulse longitudinal component is on the right):
1 The Physics of plasma based laser accelerators
For higher laser intensities, such as a0 = 1.75, nonlinear wakefields are created and it is observable a wavebreaking effect which results in electrons auto-injection. The parameters of a jasmine 2D simulation in this regime follow and the results are in figure 1.2.1.
a0 n/nc λlaser Llaser w0,laser c∆t ∆x ∆y Grid P.P.C.
1.75 0.004 1µm λp/2 2λp 0.029µm λlaser/30 2∆x 1280x1024 16
The parameters of a similar 3D simulation follows and the results are plot in figure 1.2.2.
a0 n λlaser Llaser w0,laser ∆t ∆x ∆z = ∆y Grid P.P.C.
4.0 1.38e19cm−3 0.8µm 15f s 8.2µm 0.1f s 0.08µm 0.32µm 256x224x224 4
1.2.2 Electron acceleration: bubble regime
The bubble mechanism is a very robust electron acceleration scheme that can produce very short self-injected monochromatic beams. Given its deep nonlinearity and com- plexity, the only method available for modeling accurately this regime is to run full 3D simulations (or cylindrical symmetry reduced ones, as in INF&RNO, chapter 3).
Qualitatively, an intense laser pulse, shorter than a plasma wavelength both in longitu- dinal and transversal directions, propagates in an underdense plasma and pushes away
18
Figure 1.2.1: Jasmine nonlinear wakefield simulation results. Wakefield generated in relativistic nonlinear conditions, density and longitudinal phase space [x(cm), E(M eV )] plots showing wavebreaking and acceleration. The first plot is at time ct = 51µm and the others at ct = 553µm.
1 The Physics of plasma based laser accelerators
Figure 1.2.2: Jasmine 3D nonlinear wakefield generation simulation results. Evolution, at times multiple of 12.8 fs, of a 3D wakefield slice , simulated with jasmine.
20
all the electrons that invests, creating a robust, low density cavity (the bubble) just be- hind. The electrons, accumulating on its borders, slip to the back of the cavity, where an injection mechanism happens. The injected electrons are trapped inside the cavity and are accelerated by the electrostatic field resulting from the charge separation. As the electrons inside the cavity get accelerated, they move towards inside it, allowing more electrons to be injected (beam loading).
The conditions required for the bubble regime are ([14]):
• Relativistic or ultra-relativistic electron motion a > 1.
• Underdense plasma n n
c(λlaser) ≪ 1 .
• Pulse length shorter than its transversal waist w0, and kpw0≃√ a.
I have prepared some simulations for explaining this regime further:
• A 2D simulation run with my code jasmine (chapter 4). The physical and numeri- cal parameters are:
a0 n/nc λlaser Llaser w0,laser c∆t ∆x ∆y Grid P.P.C.
7.5 0.008 0.8µm w0,laser/√
2 λp√
a/2π 0.023µm λlaser/30 2∆x 2048x1024 20 and the simulation results are plot in figure 1.2.3.
• A similar 3D simulation, whose results are plot in figure 1.2.4:
a0 n/nc λlaser Llaser w0,laser c∆t ∆x Grid P.P.C.
10 0.005 1µm 8µcm 3µcm 0.018µm λlaser/32 768x256x256 8
• A 2D, 3D in cylindrical symmetry, simulation run with INF&RNO (chapter 3), plotted in figure 1.2.5.
a0 n/nc λlaser Llaser w0,laser ∆z Grid P.P.C.
4 0.0025 = 2πkp
pn/nc 2/kp 4/kp 1/30λp 6/∆z 16
1.2.3 Ion acceleration: TNSA and RPA
In the regime named TNSA (Target Normal Sheath Acceleration), qualitatively, an intense (the electrons become relativistic very soon), linearly polarized laser pulse is partially
1 The Physics of plasma based laser accelerators
Figure 1.2.3: 2D bubble simulation. Electron charge density at ct = 294.4µm. Cavity for- mation (ct = 23µm,ct = 117µm) and particle phase space [x(cm), E(MeV )]
at ct = 491µm. Electric field inside the cavity (bubble) at ct = 117µm.
22
Figure 1.2.4:
3D bubble simulation. Electrons density 3D plot (with injected bunch in green) and sliced electron density at t = 0.004cm/c.
Figure 1.2.5:
INF&RNO bubble simulation. Electrons density and longitudinal electric field ωpt = 480
1 The Physics of plasma based laser accelerators
absorbed and reflected by an overdense plasma (initially solid and ionized by the laser), whose electrons are heated and slightly pushed forward, overcoming the rear side of the target. This charge displacement creates a static longitudinal electric field, that can pull plasma ions out of the target. The resulting accelerated ion bunch has an exponential distribution of energy, and the maximum energy reached is of the order of several MeV s ([24]).
To better illustrate this physical regime, I have run a 2D simulation. It was run in 2D with my PIC code jasmine. The simulation setup is the following:
a0 n/nc λlaser Llaser w0,laser c∆t ∆x ∆y Grid P.P.C.
10 80 1µm 25fs 3µm 5 ∗ 10−7cm 1/15λp 2∆x 5120x4096 64 , and the results are in figure 1.2.6.
In the regime named RPA (Radiation Pressure Acceleration) regime, particle acceleration is dominated by radiation pressure. The results of a simulation in this regime are in figure 1.2.7 , the setup is similar to the one in 1.2.3, but here a circularly polarized laser was taken.
a0 n/nc λlaser Llaser w0,laser c∆t ∆x ∆y Grid P.P.C.
20 50 1µm 8µm 8µm 6.3 ∗ 10−7cm 1/15λp 2∆x 5120x4096 64
24
Figure 1.2.6:
TNSA simulation results. Electron density displacement, corresponding static longitudinal electric field accelerating protons, whose phase space [x(cm), E(M eV )] at the end of simulation is plot rightmost. The time of the density plots is ct = 0.002048cm.
1 The Physics of plasma based laser accelerators
Figure 1.2.7:
RPA simulation results. Protons density displacement, static longi- tudinal electric field and protons phase space [x(cm), E(MeV )] at ct = 0.0036288cm
26
2.1 The particle-in-cell method
2.1.1 Phase space representation
The most straightforward and complete approach to model numerically a system de- scribed by Maxwell-Vlasov equations consists in computing, for each time, the phase space distribution fj(x, p, t), discretized on a grid. In a full three-dimensional model, the plasma phase space is six-dimensional. Thus, the number of grid points (the memory required for the execution of the simulation), scales as n6, n being the linear dimension of the discrete grid.
This memory requirements are far beyond the actual technology limits, for example taking a meaningful grid size, let’s say n = 1024, the memory required would be ∼ 1018Gb.
It is therefore necessary to use a “compressed” representation of the discretized fj(x, p, t).
A method that use a very sparse phase space representation is the so-called particle-in-cell method.
It decomposes the fj distribution into the sum of contributions coming from a finite Npj
set of computational macro-particles, or quasiparticles. Their trajectories are followed in the phase space in a lagrangian manner, while the electromagnetic fields are discretized on a spatial grid, with grid spacing ∆x.
The macro particles are not point like charges, they are represented by a density function which is extended in space so that they can be considered as a smooth cloud of charge , in order to smooth out the numerical noise. The support of these function has a size of the order of the grid cell size. Whereas in the configuration space the numerical particles are defined by a finite extension, in the momentum space they are point-like (they have definite momentum).
The interaction of the particles with the field grids, which complete the description of the dynamics, is achieved by processes of interpolation and deposition. The interpolation and
2 The numerical modelization
deposition processes, being the support of the quasi-particle density function compact involve only a small number of grid cells, the ones overlapping with the particle’s finite shape.
The Vlasov equation and the equations of motion read:
(∂t+ ˙x·∂x+ ˙p · ∂p) fj(x, p, t) = 0 (2.1)
˙x = p
γm, ˙p = F(x, p, t) (2.2)
The PIC approach consists in discretizing the phase space density function using a finite, approximated, sum:
fj(x, p, t) = f0j
Npj−1
X
n=0
g (x − xn(t)) δ (p − pn(t)) , (2.3)
in which, f0 is a normalization factor, xn(t) is the trajectory of the n-th macro-particle and pn(t) is its momentum.
The function g(x) is the macro-particle shape function.
The shape function is used as a convolution kernel and it is assumed to have δ-like properties (from which follows f0 = N1
p):
´ g(x − xn)dx = 1
´ ∂xg(x − xn)dx = 0
(2.4)
g(x) describes the macro-particle spatial extension in space and it is useful for reducing the numerical noise arising from interpolation and deposition processes, which would arise if a δ-function was used instead. The meaning of g(x) is evident considering the expression for the charge density, which becomes:
ρ(x, t) =P
jQj´ fj(x, p, t)dp ρ(x, t) =P
j,nqjg(x − xn)
28
, whereas the electrical current can be defined as:
j(x, t) =X
j,n
vnqjg(x − xn) (2.5)
Rewriting the Vlasov equation 2.1, using this discretized phase space discretization and equation 2.2, one gets:
∂tf = −f0PNp
n {[∂xg (x − xn(t)) · ˙xn(t)] δ (p − pn(t)) +g (x − xn(t)) [∂pδ (p − pn(t)) · ˙pn(t)]}
˙
x· ∂xf = f0PNp
n
npn(t)
γm · [∂xg (x − xn(t))] δ (p − pn(t))o
˙
p· ∂pf = f0PNp
n {F (z, pn(t), t) · g (x − xn(t)) [∂pδ (p − pn(t))]}
f0PNp
n
− ˙xn· gn′δn− ˙pngn· δ′n+γmpn · g′nδn+ F (z, pn(t), t) gn· δn′
= 0
(2.6)
Integrating in the momentum space and using the delta function properties, one has:
Np
X
n
− ˙xn+ pn γm
∂xg (x − xn(t)) = 0, ∀x → ˙xn= pn
γm (2.7)
Being Fnthe spatial average of the external force field acting on the n−th macroparticle F(x, pn, t) evaluated over the shape function g(x):
Fn(x, p, t) = ˆ
g(x − xn)F(x, pn, t)dx (2.8) , integrating on dz, and using the delta-like properties 2.4 of g(x), one gets:
Np
X
n
− ˙pn+ Fn
∂pδ(p − pn) = 0, ∀p = 0 → ˙pn= Fn (2.9)
The particle-in-cell method, therefore, reduces the computational complexity required for the evolution of a six-dimensional phase space grid to a system of 2Np (for each species) equations of motion, coupled with the proper equations (in our case for the e.m.
fields) that close the system giving an expression for the external force field F.
2 The numerical modelization
2.1.2 Passes of an electromagnetic PIC code and numerical parameters of a laser plasma interaction simulation
Dealing with charged particles, the physical description of the problem is closed by the Maxwell equations for the electromagnetic fields, which are coupled with the particle motion in a bidirectional way (by the Lorentz force and by the evaluation of charge and current densities).
The passes of an integration cycle of an electromagnetic PIC code are the following:
1. Time advancement of macro-particles momentum and position p, x, using the ob- tained equations of motion and the Lorentz force. The fields are interpolated from the E,B grids.
2. Deposition (spatial average on a discrete grid) of the external field quantities needed in Maxwell equations, ρ and j .
3. Time advancement of electromagnetic fields E,B, discretized on spatial grids, (see subsection 2.2.1), using Maxwell equations and the quantities computed in step 2 as external sources.
The critical parameter of a simulation is the grid cell size ∆x. The integration timestep
∆t is related to ∆x by the Courant condition ([6]). It is a condition required for the stability of the explicit integration schemes for the Maxwell PDEs, reading ∆t ≤ c∆x, where the constant c depends on the set of algorithms used.
Furthermore, the size ∆x must be small enough to resolve with enough grid points the typical lengths of the considered system.
30
In the case of a system of electromagnetic waves interacting with a plasma, these are:
• λem, the wavelength of the electromagnetic waves
• λsd = c/ωpe, the plasma skin depth
The smallest of the two length scales must be resolved with enough grid points. The two length scales correspond to two mutually-exclusive regimes:
• Sovracritical regime: ω ≪ ωpe → λsd ≪ λem, the λsd must be resolved, having the other one resolved as well.
• Underdense plasma regime: λem ≪ λsd , the λem must be resolved, having the other one resolved as well.
The laser envelope approximation (see section ) may come to help in this case, requiring only the much larger scale λsd to be resolved, allowing the use much smaller grid sizes.
Another critical parameter of a PIC simulation the number of macro-particles per cell, sampling the local phase space. Approximating the phase space distribution as a finite de- composition of a too small number of spatially extended macro-particles, can cause some regions of the phase space to be represented with not enough detail and the introduction of a statistical noise effect. The amplitude of the latter effect scales approximately with pNpart per cell ([6]).
2.1.3 Interpolation and deposition using shape functions
The spatial averaging needed for interpolation and deposition processes is defined using the particles’ shape function g.
2.1.3.1 Force interpolation
The average (interpolated) force acting on a particle is defined as, being F(x, pn, t) = q
E(x) +pnmγc×B(x) :
Fn(xn, pn, t) = ˆ
g(x − xn)F(x, pn, t)dx (2.10) Indexing the grid cells, with characteristic function χı, with the multidimensional index ı = (i, , j, k) it is possible to decompose the above integral average into a finite sum of single cell averages:
2 The numerical modelization
Fn(xn, pn, t) =X
ι∈G
ˆ
Xι
g(x − xn)F(x, pn, t)dx (2.11)
The force is given by the fields which are discretized in such a way that they take a single, constant, value per cell Ei, Bi. It is therefore possible to write the cell-integrals as function of the particle position only:
Fn(xn, pn, t) =X
ι∈G
Fı(pn, t) ˆ
Xι
g(x − xn)dx (2.12)
, or, introducing the shape factors (for a particle whose position is xn) Sı(xn) =´
Xιg(x−
xn)dx:
Fn(xn, pn, t) =X
ι∈G
Fı(pn, t)Sı(xn) (2.13)
For the shape factors the propertyP
ıSı(x) = 1 hold true.
2.1.3.2 Charge and current deposition
In order to evaluate the current and electrical charge density, discretized on a grid, it is necessary to “deposit” the macro-particle charge on the grid nodes. Being ρ(x) defined as ρ(x) =P
nq g(x − xn), then:
ρı =
´
χıρ(x)dx
´ dxχı = Vı = ˆ
χı
"
X
n
qg(x − xn)
# dx/Vı
=X
n
q
ˆ
χı
g(x − xn)dx
/Vı = 1 Vı
X
n
qSı(xn)
2.1.3.3 Common shapefunctions and shapefactors
For a regular 3D cartesian grid, with grid cells sized ∆x = (∆x, ∆y, ∆z) centered in the point xı=(i,j,k)= (xi, yj, zk), xi = x0+ ∆x · i, it is useful to introduce the centered and normalized shapefunctions and shapefactors ˜g(˜x) and ˜S(˜xı), defined starting from the cell-centered coordinate system (denoted by ~):
32
˜
xı= ((x − xi) /∆x, (y − yj) /∆y, (z − zk) /∆z) , (2.14) , in which, the shape factors becomes:
S(˜xı) = Sı(x) = ˜S ((x − xi) /∆x, (y − yj) /∆y, (z − zk) /∆z) (2.15) The equation Sı(xn) =´
Xιg(x−xn)dx, can be recast using the linear change of variables y= ∆y ∗ ˜yı+ yı:
S(˜˜ xı) = Sı(x) = ˆ
χi
g (y − (˜xı∗ d∆x + xı)) dy (2.16) S(˜˜ xı) = V
ˆ
χg(∆x ∗ (˜yı− ˜xı))d˜yı = ˆ
χ
˜
g(˜yı− ˜xı)d˜yı (2.17) where χ is the volume of the box defined by |ex| < 12, |ey| < 12, |ez| < 12 and ˜g(˜yı) ≡ V g(∆x ∗ ˜yı) .
It is natural for the shapefunction to be separable in one dimensional components, i.e.
g(x) = g(x)g(y)g(z). By simple integration properties, one has also:
S(x) = Sx(x)Sy(y)Sz(z) (2.18)
This last relation and equation 2.17 allow to compute easily the shape factors for any separable shape function. Some examples of normalized, one-dimensional shape func- tions/factors are (dropping the ~ in the figures) are:
2 The numerical modelization
34
These classical shape functions are defined piecewise on intervals of length ∆x (1 in the normalized coordinates system). By definition, the shape factor functions S(x) have the same properties. The intervals α ∈ Å of piecewise definition of these S(exi) are always of the kind α = [aα, aα+ 1], and can be identified by their parameter a ∈ A, integral or
2 The numerical modelization half-integral.
a ≤ exi≤ a + 1 (2.19)
Replacing some definitions in the relation above, and applying the floor and ceil function (⌊x⌋and ⌈x⌉) properties, one obtains directly the cell index corresponding to a given piece of function definition (for performance reasons, it is useful to know it in advance):
a∆x ≤ x − x0− ia∆x ≤ (a + 1)∆x
⌈(x − x0)/∆x − a − 1⌉ ≤ ia≤ ⌊(x − x0)/∆x − a⌋
∀a : ia= ⌊(x − x0)/∆x − a⌋ =
⌊˜x0⌋ − a ≡ i0− a; a integer
x˜0+12
− a′≡ i′0− a′; a = a′−12halfinteger (2.20) The optimized chain of computation reads (’ for the case in which a are half integer):
∀a ∈ A
↓ ia = i(0′)− a(′)
↓
xei0−a= exi0 − a(′)
↓
S(exi0− a(′)) ≡ Sa(exi0) optimizedSa
Defining b ∈ B and c ∈ C as the analogous, for the y and z directions, of the intervals a ∈ A, one can finally recompose the full 3D interpolation algorithm for a particle in position x = (x, y, z) (dropping ’):
36
(i0, j0, k0) = (⌊˜x0⌋ , ⌊˜y0⌋ , ⌊˜z0⌋) =
(x − x0)/∆x
, ..., ...
F =P
(abc)Sax(exi0) · Sby(eyj0) · Scz(ezk0) · Fi0+a, j0+b, k0+c
(2.21)
and the deposition algorithm (of the single particle quantityF ) :
(i0, j0, k0) = (⌊˜x0⌋ , ⌊˜y0⌋ , ⌊˜z0⌋) =
(x − x0)/∆x
, ..., ...
∀a, b, c : = Sax(exi0) · Sby(eyj0) · Scz(ezk0) · F → ⊕ → Fi0+a, j0+b, k0+c
(2.22)
More generally, considering symmetric shape factors S(˜x) with support supp(S) in the interval [−l, l = ˜b+ ∆x/2], the interpolation is computed only on the grid cells for which Si(x) 6= 0, or, equivalently, S(˜xı) 6= 0 holds true:
supp(S) = {−l ≤ ˜xi ≤ l}
−l ≤ (x − xi) /∆x ≤ l
−l∆x ≤ x − x0− i · ∆x
≤ l∆x (x − x0)/∆x − l ≤ i ≤ (x − x0)/∆x + l
⌈˜x0− l⌉ ≤ i ≤ ⌊˜x0+ l⌋ (2.23)
So, the cells interacting with our particle, are the one with i-index in the set I:
i ∈ I = {⌊˜x0− l⌋ + 1, ⌊˜x0− l⌋ + 2, ⌊˜x0− l⌋ + 3, ..., ⌊˜x0+ l⌋}
#(I) = 2l;
Furthermore,
• if l is an integer: i ∈ I = {⌊˜x0⌋ − l + 1, ..., ⌊˜x0⌋ − l + (l − 1), ⌊˜x0⌋ , ..., ⌊˜x0⌋ + l}, e.g. l = 1 : i ∈ I = {⌊˜x0⌋ , ⌊˜x0⌋ + 1}
• if l = m−12 is an half-integer: i ∈ I = {
˜ x0+12
−m+1, ...,
˜ x0+12
, ...,
˜ x0+12
+ m − 1} e.g. l = 32 : m = 2, i ∈ I = {
˜ x0+ 12
− 1,
˜ x0+12
,
˜ x0+12
+ 1}
