P.H. Lauritzen1, M.A. Taylor2, S. Goldhaber1, P.A. Ullrich3, J

CAM-SE-CSLAM: Consistent finite-volume transport !
with spectral-element dynamics!
1
Lauritzen ,
2
Taylor ,
1
Goldhaber ,
3,
Ullrich
2
Overfelt ,
1
Nair ,
1
Kelly
P.H.
M.A.
S.
P.A.
J.
R.D.
R.
1National Center for Atmospheric Research, Boulder
2
2
! ! Sandia National Laboratories, New Mexico !!!! Univeristy of California, Davis ! !
NCAR
is sponsored
by the
Foundation (NSF)!
302
Ramachandran
D. Nair,National
Michael N. LevyScience
and Peter H. Lauritzen
Introduction!
Problem: Coupling!
Results!
Setup: Transport inert and passive tracers in Jablonowski &
Williamson (2006) baroclinic wave
Gaussian “ball”!
sigma-pressure floating Lagrangian vertical coordinate.
Zonally symmetric (smooth)!
/4
(-1,+1)
(+1,+1)
Slotted cylinder (rough)!
e
x
1
Q
(-1,-1)
/4
Physical Domain
x
2
(+1,-1)
+ /4
Computational Domain
The continuity equation for air mass (2) is solved with
CAM-SE and the continuity for tracers (3) is solved with
CSLAM. We need the CSLAM solution (3) to reduce to the
CAM-SE solution for air mass (2) when q=1:
S
e
Fig. 9.22 A schematic diagram showing the mapping between each spherical tile (element)
of the physical domain (cubed-sphere) S onto a planar element e on the computational domain
C (cube). For a DG discretization each element on the cube is further mapped onto a unique
reference element Q, which is dened by the Gauss-Lobatto-Legendre (GLL) quadrature points.
The horizontal discretization of the HOMME dynamical cores relies on this grid system.
! 
! 
! 
! 
! 
!  Computational throughput for many-tracer applications
The cubed-sphere has the attractive feature that the domain S is naturally deDiscretization
is mimetic
=> mass-conservation
and total
is
composed
into non-overlapping
quadrilateral
elements (tiles) eS . This topology
well-suited
high-order element-based
methods such as spectral element or DG
energyfor
conservation
on element
methods, and amenable to efcient parallel implementation. Each face of the cubedConserves
angular momentum
very well
2
sphere
has Ne × Naxial
e elements, thus Nelm = 6 Ne elements span the entire spherical
Nelm
S
(Lauritzen
et =al.,
2014)
domain
such that S
∪ e=1
e ; in Fig. 9.22 Ne is 4. There exists a one-to-one corS and
the planar
element
respondence
between
spherical element eS on
e
Supports
staticthemesh-refinement
and
retains
formal
order
on C as depicted in Fig. 9.22. The element-wise continuous mapping allows us to
of accuracy!
perform
integrations on the sphere in a mapped (local) Cartesian geometry rather
Hon
ighly
scalable
than
the surface
of the sphere. The High-Order Method Modeling Environment
(HOMME)
developed atsimilar
NCAR relies
on this grid
system (Dennis et al, 2005).
AMIP-climate
to current
model
CSLAM (Conservative semi-Lagrangian Multi-tracer
scheme) is based on a cell-integrated semi-Lagrangian
approach that, unlike CAM-SE, allows for long time-steps,
is locally and globally conservative, has a linear correlation
shape-preserving limiter/filter and is geometrically very
flexible meaning that the method can accommodate any
spherical grid constructed from great-circle arcs. Since
geometric information computed for the transport of one
trace species can be re-used for each additional tracer, the
scheme (CSLAM) is termed multi-tracer efficient.
CAM-SE-CSLAM
CAM-SE
!
!
!
!
!
Solution: flux-form
!
!
!
!
!
!
!
!!
(a)
CAM-SE-CSLAM CAM-SEreference
(b)
-  Compute air mass-flux through CSLAM control volume
sides using method developed by Taylor et al.
-  Find swept fluxes (using Newton iteration) so that
CSLAM swept flux matches CAM-SE flux to round-off:
a-b: Find perpendicular (x and y) CSLAM fluxes that
match CAM-SE fluxes
c: flux areas define departure points
d: add extra point to swept side and iterate so that 2D
CSLAM flux match CAM-SE
day 15!
day 15!
day 17!
day 17!
day 15!
Throughput data produced on NCAR’s Yellowstone
supercomputer.
day 17!
1 degree configuration (NE30NP4NC3), 40 tracers
ntask 256, 1 degree (NE30NP4NC3), Yellowstone computer
120
100
SE: Total tracers
CSLAM: Total tracers
CSLAM: fill halo
CSLAM: reconstruction
CSLAM: remap
CSLAM: high-order weights
CSLAM: iterate
100
80
60
40
20
0
0
20
40
60
80
100
120
140
Number of tracers
Day 7.5!
Day 7.5!
Day 7.5!
Day 9!
Day 9!
day 5!
day 5!
day 5!
day 10!
day 10!
day 10!
Day 9!
CAM-SE
!
!
!
!
!
Performance!
Surface pressure evolution for CAM-SE and CAM-SECSLAM match to round-off every time-step (not shown)
GLL Quadrature Grid
Properties:
Simulations are performed with 30 vertical levels with 1
degree CAM-SE (left column), CAM-SE-CSLAM (middle
column) and reference solution is 0.25 degree CAM-SE.
Time in seconds
e
S
160
180
200
10
1
CSLAM
SE
One element
per processor!
1000
number of processors
Summary!
!
Acknowledgments!
•  CSLAM has been consistently coupled with spectral
Weelement
thank I. Güor
laboratory
assistance,
Mary Juana
for
(SE) for
dynamical
core,
i.e. CSLAM
conserves
seeds,
Herbmass,
Isside for
care, and(monotone),
M.I. Menter for
tracer
is greenhouse
shape-preserving
and
questionable
advice.
Funding
this project was
preserves statistical
a constant
mixing
ratiofordistribution
(freeprovided
the Swarthmore College Department of
streamby
preserving).
Biology,
a Merck
stipend,
and my
that
•  CSLAM
takes summer
a 3x longer
time-step
for mom.
tracers[Note
than SE
people’s
titles
omitted.]
•  CSLAM
hasare
been
shown to be more accurate for tracers
with steep gradients compared to SE and equally accurate
for smooth tracer distributions
•  CSLAM preserves linear correlation even under forced
conditions in `toy’ terminator chemistry test whereas SE
does not
•  CSLAM is faster than SE when transporting more than
approximately 18 tracers
•  Even though CSLAM needs a halo of 3 cell width, it
scales to one element (3x3 CSLAM control volumes) per
processor
More information: contact pel@ucar.edu ; Home page: http://www.cgd.ucar.edu/cms/pel !
The terminator ‘toy’-chemistry test: A simple tool
to assess errors in transport schemes
(Lauritzen et al., 2015, GMD)!
Transport 2 reactive species (Cl and Cl2) in Jablonowski and
Williamson (2006) flow. The sources and sinks for Cl and
Cl2 are given by a simple, but non-linear, "toy" chemistry
that mimics photolysis-driven conditions near the solar
terminator, where strong gradients in the spatial distribution
of the species develop near its edge. Despite the large spatial
variations in each species, the weighted sum Cly=2CL+Cl2
should always be preserved
Total tracers time in seconds
terms of local orthogonal Cartesian coordinates x 1 , x2 ∈ [− /4, /4], as shown in
Fig. 9.22 . Thus C is essentially a union of six non-overlapping sub-domains (faces)
and any point on C can be uniquely represented by the ordered triple (x 1 , x2 , )
where = 1, . . . , 6, is the cube-face or panel index. The projections and the logical
orientation of the cube panels are described in Nair et al (2005b) and Lauritzen et al
(2010).
CAM-SE
(Community
Atmosphere
– Spectral
The equiangular
central projection
results in a Model
uniform element
width ( x 1 =
x2 ) on C , which
is an advantage
for practical implementation.
Figure 9.22
proElements)
is based
on a continuous
Galerkin spectral
finite
vides a schematic
diagram
of thehorizontal
mapping between
the physical
domain
S (cubedelement
method
in the
directions
and
a hybrid
sphere) and the computational domain C (cube).