A global research team, including researchers from the University of Queensland, is developing methods to 3D image the subsurface of the Earth from observational data.
Associate Professor Lutz Gross, from UQ's School of Earth and Environmental Sciences, is leading the software infrastructure development for the project, called Esys-escript, with his team largely creating appropriate computational techniques and algorithms.
Esys-escript is a programming tool for novices and experts to implement mathematical models in Python using the finite element method (FEM) in a wide range of geophysical modelling problems. It has applications in earth mantle convection, plate subduction, environmental surveying, exploration of mineral and carbonous materials, transport modelling and the monitoring of groundwater, erosion, tsunamis and earthquakes.
Esys-escript has been developed over several projects with various partners, including the University of Western Australia, University of Melbourne, and France’s Université Savoie Mont-Blanc.
The project’s goal is to include all available geophysical data sets — such as field, electro-magnetic and seismic data — for a region or province in a single inversion model. (Inversion is the mathematical process of calculating cause from a set of observations.)
This is a massive computational task and requires the use of large parallel computers. The 3D inversions are mostly running on high-performance computers, and Lutz and his team are developing their software to run on these platforms with RCC assistance.
RCC computer systems engineer Mark Endrei helped Lutz and his team with HPC application development on escript. Mark is currently helping the team broaden the supported platforms and simplify user access to the developed software.
“We see a lot of potential to get users interested in our software,” said Lutz.
“To improve the acceptance of the software it is crucial that beginners can work and get trained on a platform they are familiar with.
“We didn’t have expertise [within the research group] in developing the software for all the platforms we needed, so Mark has been tremendously helpful in building new platform support for our software, which is important for expanding our user community.”
As a joint RCC and SEES appointment, former RCC staffer Cihan Altinay worked extensively on the Esys-escript project, largely providing expertise on GPU and OpenMP.
In terms of compute power, Lutz and his team are using a “little cluster” within SEES for small jobs. For very large inversions, they are using Perth-based Pawsey Supercomputing Centre’s Magnus HPC, and have used Raijin, Canberra-based NCI Australia’s former supercomputer. (In November 2019, NCI replaced Raijin with a much faster supercomputer called Gadi.)
Using national supercomputers meant the team was able to invert a gravity field anomaly data set (mainly satellite imagery) for the entire Australian continent.
“A single inversion on Magnus took four to five hours, using about 1,000 cores. On a PC this would take around 1.5 months. And we had a few (about 50) runs to do to tune some parameters, which would take a couple of years to complete on a PC, I guess,” said Lutz.
“Based on this code the colleagues at WA now do routinely high-resolution inversion of large-scale gravity and magnetic data on Magnus.”
Currently, Lutz and his team are looking into techniques to focus the calculation to regions where “interesting things” are detected, such as the interior structures of volcanoes, carbon storage in mineral deposits or leaks in dams. “This can be achieved by focusing the local resolution while the inversion (simulation) is running. This approach also helps to reduce computational costs and at the same time enhances the quality of results,” said Lutz.
The underlying software infrastructure for Esys-escript is funded through the Australian Government’s National Collaborative Research Infrastructure Strategy (NCRIS) program.
Assoc. Prof. Lutz Gross
Figures show the results of 3D inversion of an electric field intensity inversion over the Mount St Helens volcano (USA) using 14.7 million tetrahedral elements solving for over 150 million unknows using 200 cores of a compute cluster in about 6 hours. Objective of this calculation using synthetic data was to image the conduit and magma chamber from surface observations. (Image from Dr Lutz Gross.)