Plasma Simulation with Parallel Kinetic Particle Codes

P. Gibbon, R. Speck, B. Berberich, A. Karmakar, L. Arnold, M. Ma ek s

published in

NIC Symposium 2010 G. Munster, D. Wolf, M. Kremer (Editors) Forschungszentrum Julich GmbH, Julich Supercomputing Centre (JSC), John von Neumann Institute for Computing (NIC), Schriften des Forschungszentrums Julich, IAS Series, Vol. 3, ISBN 978-3-89336-606-4, pp. 383-390.

c 2010 by Forschungszentrum Julich Permission to make digital or hard copies of portions of this work for personal or classroom use is granted provided that the copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. To copy otherwise requires prior specic permission by the publisher mentioned above.

http://hdl.handle.net/2128/3697

Plasma Simulation with Parallel Kinetic Particle Codes

Paul Gibbon1 , Robert Speck1 , Benjamin Berberich1 , Anupam Karmakar1,2, Lukas Arnold1 , and Martin Maek3 s
Institute for Advanced Simulation, J lich Supercomputing Centre u Forschungszentrum J lich, 52425 J lich, Germany u u E-mail: {p.gibbon, r.speck, b.berberich, a.karmakar, l.arnold}@fz-juelich.de
2 1

ExtreMe Matter Institute EMMI, GSI Helmholtzzentrum f r Schwerionenforschung u Planckstrae 1, 64291 Darmstadt, Germany

Institute of Physics, Academy of Sciences of the Czech Republic, Prague, Czech Republic E-mail: masekm@fzu.cz
Recent work on plasma simulation with kinetic particle codes at JSC is described. Special attention is given to developments in mesh-free plasma modelling using parallel tree codes, covering algorithmic and performance issues, simulations of ion acceleration by high intensity lasers, and new applications of this technique to vortex-uids and edge-plasma modelling in tokamaks.

1 Introduction
Kinetic simulation methods are an indispensible tool for plasma modelling over large ranges of density and temperature. These techniques, which treat the plasma as a 6dimensional (3 space, 3 momenta) phase-uid, are more complex and expensive than hydrodynamic or magnetohydrodynamic models. In the project JZAM04, several complementary kinetic codes are being applied to model a variety of plasma physical problems. These include BOPS, a versatile, 1-dimensional particle-in-cell (PIC) code capable of modelling laser-solid interaction experiments; PSC, an open-source 3D PIC code from LMU Munich; ILLUMINATION, a similar 3D code (with different wave-solvers) developed by Queens University Belfast; and PEPC, a parallel, mesh-free tree code developed by the authors. The development and performance optimisation work with these codes forms a major activity of the recently established Plasma Physics Simulation Laboratory at JSC. In the present article we review recent applications and development work involving these codes, which have played a central role in two separate 3rd-party projects, and also form the basis of several Diploma and Ph. D. theses.

2 Hard x-Ray Sources with Thin Foils

Recent experiments by the U. Duisburg-Essen group to optimize hard x-ray line emission from thin coating targets were analysed with the help of the 1D PIC code BOPS. Hard x-rays are generated when a short-pulse, high-intensity laser is used to ionize a solid target, generating high-velocity electrons which proceed to create inner-shell holes in nearby atoms. The resulting hard x-ray pulse can be used for real-time imaging of rapid structural changes in solids or large molecules. It is found that the x-ray pulse efciency for

383

poor-contrast laser systems in which a large preplasma is suspected can be enhanced by using a near-normal incidence geometry even at high laser intensities. It can be shown that independently of the laser contrast ratio, i.e. for a steep density gradient or even in the presence of an extended density shelf, it is possible to control and optimise the K emission from a femtosecond laser plasma by adjusting the prepulse delay and/or angle of incidence appropriately. Laser systems with poor contrast can be better utilised by using target congurations with small incidence angles to optimise (resonant) absorption. The use of large incidence angles such as the standard 45-degree geometry may be contributing to poor K conversion efciencies recorded in many recent experiments. A high-contrast system can be optimised either by controlled prepulse or by going to larger incidence angles, exploiting the tendency for the density prole to be remoulded by the main pulse, allowing better absorption1, 2.

3 Heating and Ion Acceleration in Nanostructured Foils

Nanostructure surfaces are especially promising as highly absorbing targets for high-peakpower sub-picosecond laser-matter interaction. Efcient hot electron, fast ion, and thermonuclear neutron production with moderate laser intensity have already been reported, but theoretical investigations on the use of porous targets for these purposes are still scarce. In a recent study using PEPC, a new phenomenon of Coulomb implosion has been identied3 . The implosion effect is caused by hot electrons circulating inside the shells, drawing ions inwards, where they eventually meet in the centre Figure 1.

a)

b)

Figure 1. Coulomb implosion of thin shells initially arranged in a bcc foam matrix with ionized electrons conned to the inner shell surfaces a). Acceleration of ions off the inner surfaces leads to uniform convergence and peaked ion temperatures (hot spots) at the shell centres b).

Under the same irradiation conditions, a single shell simply blows apart, and does not exhibit the symmetric collapse observed in the foam lattice simulation here. In this case, some hot electrons circulate inside the shell, but most are dragged outside, leading to a net force on the ions directed radially outwards. (A fully stripped ion shell will, by Gauss Law, have zero electric eld inside). This implies that the laser heating is strongly modied by

384

the regular lattice structure, which in turn radically alters the ion dynamics. These ndings have recently been corroborated by 2D PIC simulations in which the electron heating and laser propagation is treated self-consistently with a fully electromagnetic eld solver Figure 2.

a)

b)
Figure 2. 2D PIC simulation of foam array irradiated by high intensity pulse (incident from the bottom) with similar parameters to those in the tree-code simulations. a) Electron density at a few fs, b) ion density after 100 fs.

Should this nano-implosion phenomenon scale to higher, relativistic intensities, it might also have potential as a compact neutron source. A recent comparison between atomic clusters and aerogels4 suggests that the latter are capable of yielding 10 times as many neutrons for the same laser energy because of the higher kinetic energy imparted to deuterons contained within the aerogel lattice. The present study indicates that density enhancements created by heating regular porous lattice structures should result in even higher neutron yields.

385

4 Mass Limited Ion Acceleration

One of the hot topics of laser-matter interactions is ion acceleration. Current experiments have made dramatic progress in producing multi-MeV beams of light ions and protons which may eventually be used in tumor therapy. Before this goal can be realised however, a number of challenges have to be overcome regarding the beam quality. Simulations are essential here for exploring novel target congurations in order to provide experimental guidance. Microstructured targets have been proposed to reduce the ion beam energy bandwidth and emittance. Our simulations show that a pure proton microdot target does not by itself result in a quasimonoenergetic proton beam: in fact, such a beam can only be produced with a very lightly doped target, in qualitative agreement with one-dimensional theory5 Figure 3. The simulations suggest that beam quality in current experiments6

a)

b)

Figure 3. Laser acceleration of proton microdot (central feature). Energy spread of the protons can be reduced by decreasing the relative proton density here 50% (a) and 5% (b) respectively.

could be dramatically improved by choosing microdot compositions with a 510 times lower proton fraction. Further investigations have further quantied these ndings, setting lower limits on the useful proton fraction, beyond which an energy ltering scheme becomes more effective7.

5 Mesh-Free Modelling of Tokamak Edge Physics

The group plasma edge simulations for fusion plasmas in IEF-4 (Prof. Reiter) develops and applies 2- and 3-dimensional computer simulation codes for interpretative and predictive studies of physics close to the plasma container wall (vacuum chamber). This domain is characterised by a complex interaction of plasma-chemical and turbulent processes. The models contain both empirical and rst-principles based modules. The long term goal of model development is step by step replacement of the ad hoc by ab initio models, aided by increasing HPC resources based on the parallel computing paradigm. The self-consistent electrical eld, obtained from the position of charged test particles, or,

386

in uid approximations, of charged uid parcels, is one such component. The mesh-free method offered by the tree code PEPC-B developed at JSC is a promising candidate either to provide self-consistent elds in tandem with existing modelling tools, or as a stand-alone (radiation-free) code for fusion plasma simulations. Currently this is being jointly developed by JSC and IEF-4 to model plasma edge phenomena such as the potentially catastrophic edge-localised modes (ELMs).

Figure 4. Example of a triangular mesh tted in the vessel of the MAST-tokamak.

In a rst step interfaces have been developed to provide PEPC-B with fusion relevant magnetic elds. For this purpose equilibrium elds similar to those used in the transport code EIRENE8 have been introduced. To dene the external magnetic eld PEPC-B assigns vector data according to a 2d triangular mesh (5), assuming toroidal symmetry in the tokamak. This feature in principle enables PEPC-B to run full tokamak core simulations. Next, the eld data provided from external sources have to be smoothed and adjusted to guarantee div(B) = 0. A further addition is the inclusion of a collision module based on the Monte-Carlo model of Takizuka and Abe9 . This permits collisions between the injected tracer particles and the plasma backround to be taken into account. An example application is the modelling of impurities such as C + ions occurring in fusion plasmas after sputtering or gas puff scenarios in a realistic plasma environment. A current priority is to simulate gas puff experiments recently performed on TEXTOR and to see how these ndings scale to larger machines such as ITER.

6 Darwin Model
A strategic goal with this project is to extend the fast summation algorithm deployed in PEPC to include self-generated magnetic elds. This Darwin or magnetoinductive approach has been pursued within the particle-in-cell paradigm for some time10 . A mesh-free Darwin model would open up a large range of new applications of mesh-free plasma simulation, not least in the tokamak modelling scenarios described above, and also in particle beam transport in dense plasmas. The model incorporates slowly varying magnetic elds

387

by neglecting the transversal part of displacement current in Amp` res law. This transforms e the hyperbolic equation system into a set of elliptic equations, which can again by solved by a fast summation algorithm. qj i (r i ) = (1) |r i rj |
j=i

E l (r i ) = i

qj r ij 3 rij 1 2c 1 qj v j + rij 2c 1 c (qj v j r ij )r ij 3 rij

(2)

Ai (r i ) =

(3)

B i (r i ) = Ai =

qj v j rij 3 rij 1 qj v j 2 rij 2c (qj v j r ij )r ij 3 rij

(4)

E t (r i ) = i

1 Ai 1 = 2 c t 2c

(5)

On the other hand, numerical difculties arising from this approximation have already been reported by previous authors who implemented the Darwin model within PIC or Vlasov codes. The standard scheme known from the fully electromagnetic codes used for the calculation of time derivative of the vector potential causes a violent numerical instability destroying the whole run in a few time-steps. One of the possible solutions to this problem is to express the quantities in terms of Hamiltonian generalized variables, which avoids the time derivative of the vector potential in the the equation of motion. Our magnetoinductive model employs a multipole expansion of the Darwin eld equation, modied to account for nite-sized particles and evaluated within the PEPC tree algorithm framework. First tests of this model have been made on charged particle beam evolution in vacuum.

7 Tree Code Scalability/Performance

The PEPC code is written in a generic fashion without the usage of external libraries. This results in excellent portability. In the PRACE benchmarking framework PEPC was run on four different computer architectures, namely: IBM Blue Gene/P (jugene), IBM Power6 (huygens), Cray XT5 (louhi), Intel Nehalem (juropa). As a test case we used 5 107 particles and a cubic, homogeneous initial distribution. The scaling behaviour is shown in Figure 5, where the time needed for one inner loop step is shown as a function of the peak performance of the used partition. As shown, PEPC is able to utilize the given partition performance independently of the architecture. Only the Intel Nehalem architecture shows a signicantly better performance. PEPCs overall scaling behaviour is also impressive for a tree code, although the current version is only able to use up to approximately 8192 cores. But with these the code is capable of simulating more than 108 particles.

388

Three different areas have been identied as challenges for further development of the code: The inevitable need for global communication in Barnes-Hut tree codes, a strong memory dependency of the number of cores and a reliable load balancing scheme. These problems have to be well analysed in order to make resonable use of PEPC on larger partitions. While the second topic is a question of data structure and organisation the two others focus on communication patterns and their impact on the performance. It is clear that although the tree code is an intrinsically adaptive algorithm which performs very well even with very inhomogeneuous particle distributions, its performance relies strongly on a sophisticated load balancing scheme, which is closely linked to the domain decomposition method. But even with a wisely chosen load balancing approach the need for non-local communication is an inevitable consequence of the elliptic nature of the Poisson equation. On the other hand, recent analyses of the communcation pattern reveal that nearly every communication instance is actually required at some point in the force calculation. For covering the span of necessary non-local communication and demanding memory requirements a physical separation of communication and computation process may provide a way to scale beyond 104 cores.

Figure 5. Performance of PEPC on various HPC architectures, normalised to the theoretical peak performance of a given multi-core partition.

8 Concluding Remarks
To summarize, we have reported on various activities concerning kinetic plasma modelling at JSC. Most of this work involves application and further development of the mesh-free

389

code PEPC, but recently we have also started using fully electromagnetic PIC codes to model laser-plasma interaction processes. Future work will include rigorous comparisons between the PIC and mesh-free approaches, in order to more clearly identify the advantages of the latter. The mesh-free Darwin or magnetoinductive approach offers completely new possibilities in many areas of plama physics from magnetic fusion to space physics. A major challenge for the usage of the mesh-free technique on contemporary HPC systems is to improve its parallel scalability. Efforts are underway to improve the current limit of 8192 cores on BG/P to at least 64k cores.

Acknowledgements
This work was supported by the Alliance Program of the Helmholtz Association (HA216/EMMI). Simulations were performed with computing resources granted by the VSR of the Research Centre J lich unter project JZAM04. u

References
1. P. Gibbon, M. Maek, U. Teubner, W. Lu, M. Nicoul, U. Shymanovich, A. Tarases vitch, P. Zhou, K. Sokolowski-Tinten, and D. von der Linde, Modelling and optimisation of fs laser-produced K sources, Appl. Phys. A, 2009. 2. W. Lu, M. Nicoul, U. Shymanovich, A. Tarasevitsch, P. Zhou, K. Sokolowski-Tinten, D. von der Linde, M. Maek, P. Gibbon, and U. Teubner, Optimized K x-ray ashes s from femtosecond-laser irradiated foils, Phys. Rev. E, 80, 026404, 2009. 3. P. Gibbon and O. N. Rosmej, Stability of nanostructure targets irradiated by high intensity laser pulses, Plasma Phys. Contr. Fus., 49, 18731883, 2007. 4. V. P. Krainov and M. B. Smirnov, Nuclear fusion induced by a super-intense ultrashort laser pulse in a deuterated glass aerogel, JETP, 93, 485490, 2001. 5. A. P. L Robinson and P. Gibbon, Production of proton beams with narrow-band energy spectra from laser-irradiated ultrathin foils, Phys. Rev. E, 75, no. 1, 015401, 2007. 6. S. M. Pfotenhauer, O. J ckel, A. Sachtleben, J. Polz, W. Ziegler, H.-P. Schlenvoigt, a K.-U. Amthor, M. C. Kaluza, K. W. D. Ledingham, R. Sauerbrey, P. Gibbon, A. P. L. Robinson, and H. Schwoerer, Spectral shaping of laser generated proton beams, New J. Phys., 10, 033034, 2008. 7. A. P. L. Robinson, P. Gibbon, M. Zepf, S. Kar, R. G. Evans, and C. Bellei, Relativistically correct hole-boring and ion acceleration by circularly polarized laser pulses, Plasma Phys. and Cont. Fus., 51, 024004, 2009. 8. D. Reiter, M. Baelsmans, and P. B rner, The Eirene and B2-Eirene Codes, Fusion o Science and Technology, 47, 172, 2005. 9. T. Takizuka and H. Abe, A binary collision model for plasma simulation with a particle code, J. Comp. Phys., 1977. 10. D. W. Hewett, Elimination of electromagnetic radiation in plasma simulation: the Darwin or magnetoinductive approximation, Space. Sci. Rev., 42, 2940, 1985.

390