Projects

Extending PETSc’s Composable Solvers

Funding: DOE ASCR

PETSc is a widely-used library of parallel algebraic solvers, time integrators, and optimizers. Black-box methods, such as sparse direct solvers or basic iterative methods are not scalable or efficient for large problems and highly parallel computers; efficient methods must exploit problem-specific structure. PETSc exposes a compositional algebra and rich diagnostic tools that enable researchers and production users to design solvers that exploit application-specific structure to efficiently and reliably solve problems needed by that application. It is also widely used, by maintainers and core developers as well as occasional contributors, for experiments and dissemination of research on new methods.

CRIKit: Constitutive Relation Inference Toolkit

Funding: NSF CISE

Overview

Physically-based models consist of equations that describe structural properties such as conservation of mass, momentum, and energy, combined with equations that relate stress with strain, density and internal energy to pressure, etc. While the former enjoy widespread consensus, the latter, called constitutive relations, are hotly disputed in nearly every discipline of science and engineering. The reason why is clear: there are many nonlinear materials in nature, accurate experiments are often difficult and expensive to conduct, and there are many functional forms capable of expressing constitutive relations that are admissible with respect to reference frame invariance, entropy, and the like. Although constitutive modeling is an active research field, using a constitutive relation from the published literature is usually labor-intensive due to the need to implement all the equations and coefficients, track down missing information, discover and correct for all-too-common misprints, and debug and validate the result. Meanwhile, creating new constitutive relations requires designing experiments and developing (usually) in-house software to iteratively adjust model complexity and solve the resulting nonlinear regression problems in search of robustness, accuracy, and efficiency.

The proposed work will streamline these processes for both constitutive modelers and users by creating a new open source package: CRIKit, the Constitutive Relation Inference Toolkit. CRIKit will connect PDE solvers to machine learning libraries to facilitate a streamlined modeling workflow capable of incorporating indirect or ``impure’’ measurements. CRIKit will track model and data provenance and incorporate a packaging tool whereby inferred constitutive relations can be published for others to use. In addition to the inference tools, CRIKit will also provide interfaces for simulation-based scientists and engineers to call these constitutive relations from popular simulation environments including MOOSE, FEniCS, and Abaqus.

Intellectual Merit

CRIKit will serve constitutive modelers by providing expressive tools for representing and optimizing constitutive models and for publishing/distributing the results to practitioners. It will also enable inference from indirect measurements, enabling less expensive experiments, use of data collection techniques such as multispectral imagery, and constitutive modeling of volatile materials in-situ. Since high-fidelity constitutive relations require many parameters, this approach is only computationally feasible when gradients are available. CRIKit’s approach will integrate machine learning software like TensorFlow with parallel PDE solvers like FEniCS/dolfin-adjoint that provide compatible discretizations with efficient gradient computation, yielding a system with parallelism in terms of both training data and within the PDE solves for each training example or batch of examples. CRIKit will facilitate research in optimization, model reduction, and regression using representations that are efficient to compute on modern hardware by capturing the subjective decisions such as loss functions and admissibility criteria/constraints to offer rigorous quantitative comparisons. This also enables models to be readily updated with new experimental data as well as reproducibility and validation studies. CRIKit’s models will improve simulation capability for scientists and engineers by providing ready access to the cutting edge of constitutive modeling.

Broader Impacts

CRIKit is part of a larger convergence of data and simulation-based science that is a priority of the National Strategic Computing Initiative. The development of CRIKit will involve extension of data elements towards simulation (e.g., multi-modal parallelism) and of simulation software elements towards data (such as a pathway for inserting data products into FEniCS). These extensions will be positive externalities that benefit other NSCI projects. The indirect inference capabilities offered by CRIKit will enable new experimental techniques to gain foundational understanding of multi-physics and non-equilibrium processes in a broad range of applications including geophysical rheology and biomechanics. The modular structure of CRIKit will facilitate inference at scales of data and model complexity beyond what is currently feasible, and the dissemination of the results to wide communities of domain scientists.

CEED: Center for Efficient Exascale Discretization

Funding: DOE Exascale Computing Project

CEED is a co-design center focused on high-order discretizations, which offer to greatly reduce memory (bandwidth) requirements in PDE solvers and thereby improve performance. Among other components, we develop libCEED, which provides efficient and performance portable interfaces for representing linear and nonlinear operators, as well as constructing preconditioning ingredients.

Micromorph

Funding: DOE NNSA PSAAP

The PSAAP Multi-disciplinary Simulation Center (MSC) for Micromorphic Multiphysics Porous and Particulate Materials Simulations Within Exascale Computing Workflows seeks to advance the state of multiscale predictive science for porous composite inelastic materials through grain-resolving direct numerical simulation, calibration and validation through unique experiments with advanced imaging, and numerical upscaling to create fast-running analysis and design tools.

RDycore

Compound flooding (CF) poses significant risks to human and natural systems. It can impair basic infrastructure and threaten lives and livelihood. The urgent need for an improved predictive understanding of CF and its impact on sediment dynamics (SD) and riverine saltwater intrusion (rSWI) at global scales under a future climate is well recognized.

However, several shortcomings in the current version of the Department of Energy’s (DOE’s) Energy Exascale Earth System Model (E3SM), including those related to model structure, lack of mechanistic representation of key processes, and coarse spatial resolution, pose limits. Efforts towards improved predictions using E3SM are also restricted by the lack of portability of the current E3SM’s river model on DOE’s exascale-class supercomputers, which have heterogeneous computing architectures.

The project objectives of this interdisciplinary research are to develop a rigorously verified and validated river dynamical core (RDycore) for E3SM, which includes efficient and scalable solvers for DOE’s exascale-class supercomputers, for the study of CF and its impacts on SD and rSWI in a changing climate.

Funding: DOE SciDAC BER/ASCR

SWQU: Forecasting Small-Scale Plasma Structures in Earth’s Ionosphere-Thermosphere System

Funding: NSF AGS

Space weather caused by structures and irregularities in Earth?s ionosphere can disrupt transmission of Global Navigation Satellite System signals critical for precise positioning, navigation and timing, and can influence propagation of trans-ionospheric radio waves used in land-satellite communication. Despite decades of work, forecasting this space weather phenomenon remains a challenge. This project advances fundamental research underpinning such forecasts by establishing integrated models of ionosphere-thermosphere conditions that lead to ionospheric plasma irregularities. Novel data-assimilation techniques and uncertainty quantification methods are applied to estimate uncertainties in model predictions. Spatial and temporal variations of simulated ionosphere-thermosphere parameters are validated with ground- and satellite-based observations. The project team includes both senior and early-career scientists with expertise in space physics as well as software engineers and a graduate student. Collaborations with the UK, Japan, and Taiwan expand the availability and dissemination of the models and code. The improved models will be adopted into the operational version at NOAA Space Weather Prediction Center. This project directly addresses objectives in the National Space Weather Strategy and Action Plan and the National Strategic Computing Initiative Update.

The project improves the coupled whole atmosphere model and ionosphere-plasmasphere electrodynamics model (WAM-IPE) with high resolution capability (10s of km) using the Cornell ionospheric dynamics model. The data assimilation scheme address uncertainty in external forcing especially solar EUV irradiance and high-latitude heating associated with geomagnetic activity, and constrain the large-scale ionosphere-thermosphere dynamics while preserving small-scale perturbations generated by WAM-IPE without requiring unrealistic local adjustments. Highly scalable uncertainty quantification strategies based on low rank approximations are adopted and extended to estimate the model prediction uncertainties in the presence of high-dimensional input uncertainty. Key science questions investigated here include: can the high-resolution global model generate the range of scales driving plasma irregularities at low- and mid-latitudes; which parts of the spectrum of waves and background ionosphere-thermosphere conditions lead to the formation of plasma irregularities; and what are the key drivers controlling model parameters and their uncertainty?

Completed projects

TDycore: Terrestrial Dynamical Cores for E3SM

Funding: DOE SciDAC BER/ASCR

Climate modeling components are split into “dynamical cores”, which solve the partial differential equations representing conservation laws, and the “physics”, which parametrizes underresolved dynamical and chemical processes. The dynamical cores of conventional land models have only one dimensional models for subsurface hydrology: water goes below the surface in one grid cell and it stays there. The TDycore projects seeks to modernize in several ways including support for fully 3D subsurface flow. This is a discerning test for spatial discretizations and algebraic solvers due to extreme aspect ratio and heterogeneity; our work focuses on improving the accuracy of such spatial discretizations, the efficiency of solvers, and the ability to dynamically compose and extend the dynamical core.