UQ is helping the New South Wales government determine the impacts of proposed wastewater infrastructure upgrades on the water quality of Lake Macquarie.
Two researchers from UQ’s School of Chemical Engineering are working with researchers from the New South Wales Office of Environment and Heritage (OEH) to assess the potential ecological impacts on the lake of the upgrades, as well as ensuring human health risks are reduced as the lake, near Newcastle, is used recreationally.
UQ’s Dr Matthew Adams and Associate Professor Kate O’Brien are working on the chemical engineering side of the project, with OEH’s Dr Angus Ferguson leading the project, and with funding from Hunter Water Corporation.
Dr Adams, a postdoctoral researcher, turned to RCC and high-performance computer FlashLite to help him in his contribution to the project, which required running a three-dimensional bio-geochemical model for the various water constituents (sediment, pathogens, carbon, nitrogen and phosphorus), and ecological organisms (seagrass, phytoplankton and microalgae).
He initially emailed the RCC Support Desk in May last year, saying: “Currently I can run these simulations on my Windows PC but they are lengthy (one month’s simulation equals one day on my PC), and I am hoping to obtain additional computing power to be able to do longer runs to report in this project (e.g. two years, which would take a month on my PC.)”
However, these time estimates were for a version of the model with very coarse temporal resolution — and higher resolution predictions were needed, which takes even more time for the model to run. The model also needed to be run numerous times, with different parameter values tested, to calibrate it against collected data.
Using the Delft3D water quality software package, DELWAQ, outputs were required on a 15-minute time-step for 27 months at more than 20 locations of interest in and around Lake Macquarie. Achieving this required solving the model for more than 30,000 separate spatial locations and 10 water layers at each location. “Hence, a very large amount of hard-drive space was needed to store the outputs – and FlashLite was the ideal machine for these computational requirements,” said Dr Adams.
He developed several model extensions to DELWAQ in order to simulate the water quality and ecology of Lake Macquarie, “according to our best scientific knowledge of the lake ecology and nutrient cycling dynamics.”
The native environment for these novel model extensions was Windows. However, Windows is not a typical environment for high-performance computing. Despite there being a more HPC-friendly Linux version of the software, given the urgency of Dr Adams’s project, and the model extensions already developed by Dr Adams in the Windows environment, RCC decided to implement the Windows version.
To obtain outputs of the model within a reasonable time period, the best solution was to create a Windows ‘virtual machine’ on FlashLite with several terabytes of hard-drive space. Three Windows virtual machines were created so that three different scenarios for the lake could be run simultaneously — each virtual machine with 20 cores available for parallel computing. FlashLite was chosen for the virtual machines, primarily for access to its large, local disk — the NVME drives.
RCC HPC staff Martin Nicholls and Dr David Green helped set up the Windows virtual machines on FlashLite and compiled the DELWAQ software on the platform. “It’s a really novel use of FlashLite to run Windows virtual machines with huge, fast, local disk drives,” said David. “When Matthew first approached RCC, his computational throughput was limited — he only had a single, four-core laptop to work on. With our help, he was able to run three jobs with up to 20 cores, and he had these three systems running for about a year. RCC has enhanced his throughput and productivity many-fold.”
The results enabled Dr Adams to run his model on a laptop not specifically designed for scientific computations. “If run on a standard desktop, the simulations would not have been possible due to high computational cost as well as abnormally high hard-drive requirements,” said Dr Adams.
The model predicts the timeframe in which different Lake Macquarie sites will be affected by wastewater flows caused by rainfall. The model also fairly accurately simulates the spatial distribution of seagrasses growing in the lake, which, because it is primarily controlled by underwater light availability, is strongly affected by water quality.
“It was exciting to run a model with such a fine spatial and temporal scale and see how far we could really push the high-resolution predictions of ecology and water quality. Running a model such as this also helps us identify which data will provide the most ‘bang for our buck’ if we are attempting to better understand how the ecosystem functions in Lake Macquarie and in shallow lakes more generally,” said Dr Adams.
The outcomes of the overall project will be used to inform Hunter Water Corporation’s decisions regarding its wastewater overflow system upgrades. The work will also assist the company in satisfying legislative requirements under a new, proposed “effects-based assessment” framework, which aims to assess whether the infrastructure is suitable based on the impacts on humans and the natural environment.
Tech specs:
Each of the three active DELWAQ virtual machines on FlashLite ran Windows 7 Enterprise and had:
- 20 CPU cores
- 64 GB RAM
- 200 GB C: drive hosted on NVME
- 2,400 GB D: drive hosted on NVME.
