Major upgrades of UQ’s data storage infrastructure have reached an important milestone, with new hardware and software platforms for the preservation and storage of data collections successfully installed in early January.
The work includes the installation of a new disk storage array for the University’s high-performance computers that is almost in production.
These two projects, representing a significant UQ investment, are replacing ageing technologies with newer, more powerful and resilient hardware, which will enable enhanced scalability and usability, and provide some exciting new features for users.
Data collections will be directly connected with parallel file systems, providing a substantial improvement in data transfer speeds between collections and computational storage. The new technology provides options for workloads with high I/O requirements that have traditionally had latency issues with the existing storage.
User applications will respond more gracefully to storage requests, no matter where data is sitting in the storage hierarchy.
Data will be served by UQ’s MeDiCI data platform, connecting all the University’s supercomputers, desktops and scientific instruments.
Q collections
Only data collections, known as ‘Q collections’, were affected during RCC’s holiday shutdown period from 5pm, Thursday, 24 December 2020 to 9am, Monday, 4 January 2021, in which new hardware and software were installed in Springfield’s Polaris Data Centre.
During the migration process, about five billion objects, representing almost 20 years of UQ research data, were moved from the old to new hardware and software systems.
Users will already be noticing their Q collection returning to them from storage faster than before.
All HPC services remained operational during the shutdown period, although users could not access their Q collections via the HPCs.
Eight UQ staff were involved in the upgrade, including five RCC staff, who worked alongside hardware vendors from Germany, Japan and the US.
Data storage for HPC users
RCC is in the process of replacing the ageing disk storage “scratch” array currently serving HPCs Awoonga, FlashLite and Tinaroo. The array is where data is stored while users run their jobs.
The new storage array has already been installed in the Polaris Data Centre. However, other hardware required to connect it to the HPCs is currently being shipped and due to be installed in mid-March.
Replacing the connection hardware will require shutting down Awoonga, FlashLite and Tinaroo for a few days. RCC’s engineering team is currently working on the migration procedures and will provide all users with an outage window shortly to minimise disruption and help prepare for the downtime. RCC will make every attempt to plan the outage to be as short as possible.
RCC has already notified users of the three HPCs about this upcoming scheduled outage and what steps they need to take to prepare for it and will let them know closer to the time the exact shutdown dates. If you are a FlashLite, Awoonga or Tinaroo user and did not receive an email about this, please contact the RCC Support Desk.
As HPC Wiener has its own data storage, it will be unaffected by the shutdown.
If you have queries about any of the above, please contact the RCC Support Desk.
