QURPA 2020: Using deep learning to detect black hole activity
UQ Honours student Edward Davis reports on his internship in January and February 2020 at the US National Centre for Supercomputing Applications (NCSA) located at the University of Illinois at Urbana-Champaign. Edward, who is studying for a Bachelor of Mathematics, majoring in statistics and computer science, worked on a NCSA project using deep learning to analyse gravitational wave data coming from America’s Laser Interferometer Gravitational-Wave Observatory (LIGO) detectors. The internship was part of the RCC-sponsored Queensland Undergraduate Research Projects Abroad (QURPA) program.
By Edward Davis
During my eight weeks at NCSA, I worked on scaling several deep learning models, which solve problems in astrophysics. Due to the highly complex nature of these problems, it could take days to weeks to train models using a single computer. Instead, by employing the supercomputers available at NCSA we were able to significantly speed up this time and could tackle much larger problems.
The research team I worked with at NCSA, Gravity Group, studies computational problems in numerical relativity and orbital mechanics, typically modelling black holes and neutron stars. These problems are known to be high in computational complexity. In recent years, the use of deep learning techniques has shown to be effective for such problems.
The first set of models I worked on analyse gravitational waveforms. This allows us to learn about events in space, most often black hole mergers. From gravitational wave data observed from LIGO, we can work out when the two black holes merged as well as their individual masses and spin. Another model I worked on classifies images of galaxies (e.g. spiral vs elliptical).
The first milestone was to scale these models onto NCSA’s experimental HAL cluster. HAL holds 64 NVIDIA V100 GPUs, the state-of-the-art GPUs currently found in many of the most powerful supercomputers*. The two frameworks that I used to distribute the training were Horovod (developed by Uber) and Apex (developed by NVIDIA).
In the end, we were able to show that our program was scaling quite well under each framework. The goal from this point, which is still in progress, is to move from tens to hundreds of GPUs using Oak Ridge National Laboratory’s Summit supercomputer (currently the fastest HPC in the world) to yield even better training performance.
Coming into this program, I had very limited experience in research and these two months have enlightened me with what that environment is like for a team of researchers. It was also really cool to be able to directly use NCSA’s computer clusters. I would recommend this experience to all others who are curious about what research is like, particularly in the HPC area.
For me, the highlight of the trip was experiencing university life in a completely new environment. That includes Illinois’ cold winter climate and living on campus near many other students. The trip also gave me the opportunity to visit Chicago and some parts of Canada, which I thoroughly enjoyed.
I’d like to thank everybody who made this trip as great as it was: RCC Director David Abramson and Jay Roloff, NCSA’s Associate Director of Program Management, for organising the QURPA program; my NCSA supervisor Eliu Huerta (Head of the Gravity Group), and everyone in Gravity Group who I got to meet and work closely with. I was given countless amounts of support by the people at NCSA. I hope to continue working with the team as this project continues to evolve.
*RCC high-performance computer Wiener features NVIDIA GPUs.