A new RCC supercomputer to support next-generation molecular biology, neuroscience and translational research at UQ began operating in trial mode two weeks ago.
Wiener, a Dell EMC-built high-performance computer (HPC), is designed to expedite the pace of research in a diverse range of imaging-intensive science, generated by UQ’s world-leading microscopy facilities.
The new HPC is expected to be fully operational by February 2018.
News summary
- The Wiener HPC system is UQ’s new deconvolution engine supported by RCC and a consortium of microscopy facilities and researchers.
- Wiener will support UQ’s new Lattice Light Sheet Microscope (LLSM) to provide high-resolution biological imaging data in real time, allowing image and pattern recognition as well as deconvolution at extreme scale.
- First use cases in research will focus on pioneering research into cancer biology, inflammation and infection at UQ’s Institute for Molecular Bioscience (IMB), and cutting-edge whole of brain connectome analysis for human brain developmental disorders at the Queensland Brain Institute (QBI).
About Wiener
Wiener was funded by strategic funding from UQ and a consortium of the university’s various cutting-edge microscopy facilities housed within the Centre for Microscopy and Microanalysis (CMM), IMB and QBI.
Using advanced microscope systems, researchers at these facilities generate petascale image data and immense computational workloads. Within this tsunami of microscopy data, there is image noise that has to be extracted algorithmically and computationally by deconvolution. Deconvolution is needed to ‘clean up’ image distortions introduced by some microscopes and sharpen blurred images captured during rapid movements. Achieving rapid deskewing and deconvolution for large volumes of image data will be revolutionised by the Wiener system.
Wiener harnesses the capabilities of the most powerful GPUs (Graphics Processing Units) ever made. Using a mixture of deconvolution algorithms, machine learning and pattern recognition techniques, the supercomputer will provide near real time outputs of deconvolved, tagged and appropriately characterised data, providing researchers with immediate feedback on the quality of data being collected and allowing faster interpretation of microscopy data.
UQ’s aim is to reduce time to discovery and enhance medical and translational capabilities now and into the future for better human outcomes.
Why "Wiener"?
The name of the new supercomputer is based on the creator of the Wiener filter, a mathematical model originally devised by Norbert Wiener, which ‘deconvolves’ images by removing the unnecessary noise.
Big microscopy, big computing, enormous data
The Lattice Light Sheet Microscope (LLSM) is a new microscope that can image live cells or specimens in 4D (3D imaging, over time). It acquires images incredibly rapidly and at high resolution, generating up to 7 terabytes of imaging data per day. Handling and processing this volume of image data is not a job that is feasible or affordable using public cloud computing. Wiener will deliver the image deconvolution capacity needed for this world-leading microscopy and live cell imaging. Wiener is an essential component of the HPC infrastructure now required to support cutting-edge microscopy.
The recent installation of a LLSM at UQ’s IMB, along with the necessary advanced computing infrastructure, is the result of funding from the Australian Cancer Research Foundation and the Australian Research Council.
According to IMB’s Professor Jenny Stow: “The LLSM installed at IMB is one of only a handful of these prototype instruments in the world and the third in Australia. It gives us the ability to image cell behaviour at exceptional speeds to produce full 3D movies of cells in real time, allowing us to detect small but critical changes introduced by gene mutations or drug treatments. This combination of microscopy and image processing on unprecedented scales will really advance biomedical research and drug development.”
Professor Linda Richards at QBI is leading a project to better understand the whole brain connectome model in human brain developmental disorders and syndromes. “We will be able to use this accelerated [Wiener] supercomputer to analyse brain MRI data to obtain detailed maps of brain connectivity. Understanding how brain connections vary between people and in people with brain disorders will underpin new discoveries in brain structure, function and new diagnostic tools,” said Professor Richards.
Wiener’s performance will also impact on deep learning techniques, which are being employed to help understand how information is coded in biological neural networks. Professor Geoffrey Goodhill at QBI said: “Wiener will dramatically increase the speed at which we can derive new insights from experimental data, and also test novel hypotheses about nervous system function.”
For RCC Director Professor David Abramson, Wiener rounds out the computational infrastructure at UQ. “We have platforms that support general-purpose parallel computations [Tinaroo and Awoonga], data-intensive applications [FlashLite], and now compute-intensive accelerator based codes [Wiener],” said Prof. Abramson. “Wiener is primarily intended for image deconvolution, but we are seeing new applications emerging in chemistry, physics, materials science and machine learning. Our Metropolitan Data Caching Infrastructure (MeDiCI) data fabric makes it feasible to process complex pipelines on all of these machines, allowing researchers to choose the right tool for various tasks.”
Driving a new era of research
The Wiener system was first publicly announced in October this year at the 11th eResearch Australasia conference in Brisbane, which Professor David Abramson chaired. UQ announced in February this year that it would design and build a high-performance image processing facility. It engineered a system that would put it on par with systems within the same sector in the USA and Europe.
Following a request for tender in late August, Dell EMC made a submission for a HPC solution and won the tender in early October. The Wiener system is expected to be fully operational by February 2018.
High-performance computing for clarity
Building computational capabilities of this nature for scientific research is about balance and architectural fit for purpose, to meet the overall goals and discovery strategy of an organisation, according to Jake Carroll, Senior ICT Manager (Research) from QBI and Chair of Wiener’s procurement committee.
“Every day we see more and more researchers turning to accelerated computing in order to address their requirements. Wiener’s carefully considered balance of GPUs, robust fabric design and nVME will meet those requirements, allowing us to enhance research experiences and outcomes, along the way,” said Mr Carroll.
As Wiener will provide the processing interface at the edge of scientific instruments throughout the university campus, two large visualisation nodes are included in the design to provide a fully-interactive experience to researchers viewing the data sets in real time.
Technical specifications
Complementing UQ’s investment in the LLSM, the Wiener system is built on the 14th Generation Dell EMC PowerEdge R740 platform. This provides a consistent building block for processing data sets across 15 compute and analysis nodes, along with two additional nodes for visualisation.
The visualisation nodes are comprised of:
- 2 x Dell EMC PowerEdge R740 HPC nodes featuring:
- 2 x Intel Xeon Gold 6132 (28 cores per node)
- 1TB DDR4 RAM
- 3 x NVIDIA Tesla V100 Accelerator units,
- 1.6TB Dell EMC NVMe Flash Storage
- 100Gbps Mellanox EDR InfiniBand HCA’s
- Dell EMC ProSupport Plus:(7x24) Technical Support & Assistance
The Compute and Analysis nodes are comprised of:
- 17 x Dell EMC PowerEdge R740
- 2 x Intel Xeon Gold 6132 (28 cores per node)
- 384GB DDR4 RAM
- 2 x NVIDIA Tesla V100 Accelerator Unit
- 1.6TB Dell EMC NVMe Flash Storage
- 100Gbps Mellanox EDR InfiniBand HCA’s
- Dell EMC ProSupport Plus:(7x24) Technical Support & Assistance
Bridging the Dell EMC PowerEdge platform is a low-latency high-bandwidth interconnect of Mellanox EDR SB7800 series InfiniBand switching, which delivers 7.2Tb/s of non-blocking bandwidth across the Wiener system at a 90-nanosecond latency.
The new NVIDIA Volta V100 Tensor Cores gives the Wiener system a total processing ‘deep learning’ capacity of 4.284 PetaFLOPS. This accumulated processing power comes from the 204,3800 CUDA cores and 25,600 Tensor cores distributed across the compute and visualisation nodes.
Any questions about Wiener, please contact the RCC Support Desk: rcc-support@uq.edu.au.
