Skip to content

Commit dea63a7

Browse files
clarksepre-commit-ci[bot]roelof-groenewald
authored
Add feature projection div cleaner (#4944)
* Commit of initial projection based divergence cleaner to be called from Python callbacks. * Mvinf Div Cleaner into Initialization directory. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjusting div cleaner tolerance for single precision runs to allow convergence. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reducing single precision default tolerance further. * Disabling div cleaner in single precision for now. Changed number of ghost cells. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Using existing ComputeDivB in WarpX. Updated to pull BCs from WarpX. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removing dirichlet flag from RZ div cleaner. * Changing kernel invocation for RZ. Does not work with multiple modes currently. * Adding hook to call when loading B field from external file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed a couple RZ compile issues. * Adding defines for RZ. Removing hardcoded number of ghost cells in ProjectionDivCleaner. * Cleaning up a few items for clang-tidy CI tests for div cleaner. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Source/Initialization/WarpXInitData.cpp Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Rafactoring helper execution function in pyWarpX and placing ProjectionDivCleaner into warpx::initialization namespace. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removing extraneous semicolon * Updating a call to be static for clang-tidy passage. * Adding DivCleaner into GNUmake build system. * Switching to include guards from pragma once in ProjectionDiveCleaner.H * Lowering warning priority to low for launching div cleaner after loading external B field from file. ALso removing extra include. * Disabling div_cleaner in RZ mode for now until further testing is done. * Update Source/Initialization/DivCleaner/ProjectionDivCleaner.cpp Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Moving logic into WarpX class dic cleaner launching function so it will check when called via Python as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adding CI tests and fixing a few bugs for external field loading. Exposing Bfield_dp_external to Python for loading external fields. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added support for single precision and RZ div cleaners. Exposed external fields to python. Updated benchmarks and adding RZ CI test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adding CI test for laoding fields from Python. Updated documentation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixing CI test. * Updating flags in PICMI to conform to current standards. Fixing some RZ compilation issues. * Fixing clang-tidy const issue. * Fixing previous merge error for multifab allocations of external fields. * Update Source/WarpX.H Fix typo in comment. Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Adjusting bc mapping in projection div cleaner to use std::map for assignment instead of if statements. Signed-off-by: S. Eric Clark <25495882+clarkse@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Changing default behavior for externally loaded fields to automatically scrub b field divergence. * Update Python/pywarpx/picmi.py fixing typo in comment. Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Update Python/pywarpx/picmi.py Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Update Python/pywarpx/picmi.py Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Update Python/pywarpx/picmi.py Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Adding more requirements to enable divergence cleaner by default. * Changed location of initialization logic for external divb cleaning to be after grid_type is initialized. * Disabling projection div cleaner when diffusion div cleaner is enabled for Yee algorithm. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Source/Python/WarpX.cpp Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Throw low priority warning when cleaning MR saying that only level 0 was cleaned. * Update Regression/WarpX-tests.ini Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Update Regression/WarpX-tests.ini Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Update Regression/WarpX-tests.ini Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Updating picmi to use new callback loaction for loading external fields. * Update WarpXInitData.cpp changing to execute callback at finest level when loading external fields so all levels are accessible * Loosened default tolerance slightly on rtol for projection div cleaner. Updated failing test benchmarks. Adding input parameters to control tolerance on command line or through python. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixing clang-tidy initialization issue. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleaning up style errors. Signed-off-by: S. Eric Clark <25495882+clarkse@users.noreply.github.com> --------- Signed-off-by: S. Eric Clark <25495882+clarkse@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com>
1 parent a23e100 commit dea63a7

31 files changed

+1253
-50
lines changed

.zenodo.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939
"name": "Blelly, Aurore"
4040
},
4141
{
42-
"affiliation": "Lawrence Livermore National Laboratory",
42+
"affiliation": "Helion Energy, Inc.",
4343
"name": "Clark, Stephen Eric",
4444
"orcid": "0000-0002-0117-776X"
4545
},

Docs/source/usage/parameters.rst

+4
Original file line numberDiff line numberDiff line change
@@ -2502,6 +2502,10 @@ Additional parameters
25022502
to propagate (at the speed of light) to the boundaries of the simulation
25032503
domain, where it can be absorbed.
25042504

2505+
* ``warpx.do_divb_cleaning_external`` (`0` or `1` ; default: 0)
2506+
Whether to use projection method to scrub B field divergence in externally
2507+
loaded fields. This is automatically turned on if external B fields are loaded.
2508+
25052509
* ``warpx.do_subcycling`` (`0` or `1`; default: 0)
25062510
Whether or not to use sub-cycling. Different refinement levels have a
25072511
different cell size, which results in different Courant–Friedrichs–Lewy
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
#!/usr/bin/env python3
2+
#
3+
# --- Input file for loading initial field from Python callback
4+
5+
import numpy as np
6+
import scipy.constants as con
7+
from mpi4py import MPI as mpi
8+
from scipy.special import ellipe, ellipk
9+
10+
from pywarpx import fields, picmi
11+
12+
constants = picmi.constants
13+
14+
comm = mpi.COMM_WORLD
15+
16+
simulation = picmi.Simulation(verbose=True)
17+
18+
19+
def augment_vector(v):
20+
d = v[1] - v[0]
21+
vp = (v[1:] + v[:-1]) / 2.0
22+
vp = np.insert(vp, 0, vp[0] - d)
23+
vp = np.append(vp, vp[-1] + d)
24+
return vp
25+
26+
27+
class CurrentLoop(object):
28+
def __init__(self, **kw):
29+
self.radius = kw.pop("radius") # (m)
30+
self.z0 = kw.pop("z0", 0.0) # (m) Defines the middle of the current loop
31+
self.B0 = kw.pop("B0", 1.0) # (T) Strength of field in middle of loop
32+
33+
# Compute loop current to normalize the B field of the center of the current loop
34+
# from the following on axis equation for Bz
35+
36+
self.I = 2.0 * self.radius * self.B0 / con.mu_0
37+
38+
def psi(self, r, z):
39+
# Convert to spherical coordinates
40+
rho = np.sqrt(r**2 + (z - self.z0) ** 2)
41+
theta = np.arctan2(r, z - self.z0)
42+
43+
coeff = con.mu_0 * self.I / (4.0 * np.pi)
44+
denom = self.radius**2 + rho**2 + 2.0 * self.radius * rho * np.sin(theta)
45+
46+
k2 = 4.0 * self.radius * rho * np.sin(theta) / denom + 1e-12
47+
48+
term1 = 4.0 * self.radius / np.sqrt(denom)
49+
term2 = ((2.0 - k2) * ellipk(k2) - 2.0 * ellipe(k2)) / k2
50+
51+
return coeff * term1 * term2 * r
52+
53+
def __call__(self, xv, yv, zv, coord="x"):
54+
# Generate B-field mesh
55+
XMB, YMB, ZMB = np.meshgrid(xv, yv, zv, indexing="ij")
56+
RMB = np.sqrt(XMB**2 + YMB**2)
57+
58+
dx = xv[1] - xv[0]
59+
dy = yv[1] - yv[0]
60+
dz = zv[1] - zv[0]
61+
62+
if coord == "x":
63+
# Gradient is along z direction
64+
zvp = augment_vector(zv)
65+
66+
# A mesh, which will be reduced later after gradients taken
67+
XMP, YMP, ZMP = np.meshgrid(xv, yv, zvp, indexing="ij")
68+
RMP = np.sqrt(XMP**2 + YMP**2)
69+
psi = self.psi(RMP, ZMP)
70+
grad_psi_z = (psi[:, :, 1:] - psi[:, :, :-1]) / dz
71+
72+
return -XMB * grad_psi_z / RMB**2
73+
elif coord == "y":
74+
# Gradient is along z direction
75+
zvp = augment_vector(zv)
76+
77+
# Psi mesh, which will be reduced later after gradients taken
78+
XMP, YMP, ZMP = np.meshgrid(xv, yv, zvp, indexing="ij")
79+
RMP = np.sqrt(XMP**2 + YMP**2)
80+
psi = self.psi(RMP, ZMP)
81+
grad_psi_z = (psi[:, :, 1:] - psi[:, :, :-1]) / dz
82+
83+
return -YMB * grad_psi_z / RMB**2
84+
elif coord == "z":
85+
# Gradient is along x,y directions
86+
xvp = augment_vector(xv)
87+
88+
# Psi mesh, which will be reduced later after gradients taken
89+
XMP, YMP, ZMP = np.meshgrid(xvp, yv, zv, indexing="ij")
90+
RMP = np.sqrt(XMP**2 + YMP**2)
91+
psi = self.psi(RMP, ZMP)
92+
grad_psi_x = (psi[1:, :, :] - psi[:-1, :, :]) / dx
93+
94+
yvp = augment_vector(yv)
95+
96+
# Psi mesh, which will be reduced later after gradients taken
97+
XMP, YMP, ZMP = np.meshgrid(xv, yvp, zv, indexing="ij")
98+
RMP = np.sqrt(XMP**2 + YMP**2)
99+
psi = self.psi(RMP, ZMP)
100+
grad_psi_y = (psi[:, 1:, :] - psi[:, :-1, :]) / dy
101+
102+
return (XMB * grad_psi_x + YMB * grad_psi_y) / RMB**2
103+
else:
104+
error("coord must be x/y/z")
105+
return None
106+
107+
108+
class ProjectionDivCleanerTest(object):
109+
# Spatial domain
110+
Nx = 40 # number of cells in x direction
111+
Ny = 40 # number of cells in y direction
112+
Nz = 80 # number of cells in z direction
113+
114+
MAX_GRID = 40
115+
BLOCKING_FACTOR = 8
116+
117+
# Numerical parameters
118+
DT = 1e-9 # Time step
119+
120+
def __init__(self):
121+
"""Get input parameters for the specific case desired."""
122+
123+
# output diagnostics 5 times per cyclotron period
124+
self.diag_steps = 1
125+
self.total_steps = 1
126+
127+
self.Lx = 1.0
128+
self.Ly = 1.0
129+
self.Lz = 2.0
130+
131+
self.DX = self.Lx / self.Nx
132+
self.DY = self.Ly / self.Ny
133+
self.DZ = self.Lz / self.Nz
134+
135+
self.dt = self.DT
136+
137+
"""Setup simulation components."""
138+
139+
#######################################################################
140+
# Set geometry and boundary conditions #
141+
#######################################################################
142+
143+
self.grid = picmi.Cartesian3DGrid(
144+
number_of_cells=[self.Nx, self.Ny, self.Nz],
145+
warpx_max_grid_size_x=self.MAX_GRID,
146+
warpx_max_grid_size_y=self.MAX_GRID,
147+
warpx_max_grid_size_z=self.MAX_GRID,
148+
warpx_blocking_factor=self.BLOCKING_FACTOR,
149+
lower_bound=[-self.Lx / 2.0, -self.Ly / 2.0, -self.Lz / 2],
150+
upper_bound=[self.Lx / 2.0, self.Ly / 2.0, self.Lz / 2.0],
151+
lower_boundary_conditions=["periodic", "periodic", "neumann"],
152+
upper_boundary_conditions=["periodic", "periodic", "neumann"],
153+
lower_boundary_conditions_particles=["periodic", "periodic", "absorbing"],
154+
upper_boundary_conditions_particles=["periodic", "periodic", "absorbing"],
155+
)
156+
simulation.time_step_size = self.dt
157+
simulation.max_steps = self.total_steps
158+
159+
#######################################################################
160+
# Field solver and external field #
161+
#######################################################################
162+
163+
self.solver = picmi.ElectrostaticSolver(grid=self.grid)
164+
simulation.solver = self.solver
165+
166+
#######################################################################
167+
# Install Callbacks #
168+
#######################################################################
169+
init_field = picmi.LoadInitialFieldFromPython(
170+
load_from_python=load_current_ring,
171+
warpx_do_divb_cleaning_external=True,
172+
load_E=False,
173+
)
174+
simulation.add_applied_field(init_field)
175+
176+
#######################################################################
177+
# Add diagnostics #
178+
#######################################################################
179+
180+
field_diag = picmi.FieldDiagnostic(
181+
name="field_diag",
182+
grid=self.grid,
183+
period=self.diag_steps,
184+
data_list=["B"],
185+
write_dir=".",
186+
warpx_file_prefix="Python_projection_divb_cleaner_callback_3d_plt",
187+
warpx_format="plotfile",
188+
)
189+
simulation.add_diagnostic(field_diag)
190+
191+
comm.Barrier()
192+
193+
# Initialize inputs and WarpX instance
194+
simulation.initialize_inputs()
195+
simulation.initialize_warpx()
196+
197+
198+
def load_current_ring():
199+
curr_loop = CurrentLoop(radius=0.75)
200+
201+
Bx = fields.BxFPExternalWrapper(include_ghosts=True)
202+
By = fields.ByFPExternalWrapper(include_ghosts=True)
203+
Bz = fields.BzFPExternalWrapper(include_ghosts=True)
204+
205+
Bx[:, :, :] = curr_loop(Bx.mesh("x"), Bx.mesh("y"), Bx.mesh("z"), coord="x")
206+
207+
By[:, :, :] = curr_loop(By.mesh("x"), By.mesh("y"), By.mesh("z"), coord="y")
208+
209+
Bz[:, :, :] = curr_loop(Bz.mesh("x"), Bz.mesh("y"), Bz.mesh("z"), coord="z")
210+
211+
comm.Barrier()
212+
213+
214+
run = ProjectionDivCleanerTest()
215+
simulation.step()
216+
217+
##############################################
218+
# Post load image generation and error check #
219+
##############################################
220+
Bxg = fields.BxWrapper(include_ghosts=True)
221+
Byg = fields.ByWrapper(include_ghosts=True)
222+
Bzg = fields.BzWrapper(include_ghosts=True)
223+
224+
Bx_local = Bxg[:, :, :]
225+
By_local = Byg[:, :, :]
226+
Bz_local = Bzg[:, :, :]
227+
228+
dBxdx = (Bx_local[1:, :, :] - Bx_local[:-1, :, :]) / run.DX
229+
dBydy = (By_local[:, 1:, :] - By_local[:, :-1, :]) / run.DY
230+
dBzdz = (Bz_local[:, :, 1:] - Bz_local[:, :, :-1]) / run.DZ
231+
232+
divB = dBxdx + dBydy + dBzdz
233+
234+
import matplotlib.pyplot as plt
235+
236+
plt.imshow(np.log10(np.abs(divB[:, 24, :])))
237+
plt.title("log10(|div(B)|)")
238+
plt.colorbar()
239+
plt.savefig("divb.png")
240+
241+
error = np.sqrt((divB[2:-2, 2:-2, 2:-2] ** 2).sum())
242+
tolerance = 1e-12
243+
244+
print("error = ", error)
245+
print("tolerance = ", tolerance)
246+
assert error < tolerance

0 commit comments

Comments
 (0)