By RCC Director Prof. David Abramson
UQ’s MeDiCI data storage fabric will receive considerable enhancement over the next few months, with additional on and off-campus storage, faster network links, and new and improved capabilities.
MeDiCI (Metropolitan Data Caching Infrastructure) works by keeping research data close to the user, scientific instruments and compute resources while it is being referenced, but stored in a variety of long-term back end archives when not actively used. This makes it possible to leverage the best of high-performance, but costly, local cache and cheaper, but slower, archival storage technologies.
MeDiCI’s architecture, designed by RCC in collaboration with UQ ITS, has been recognised by a number of national awards, including a 2017 ACS Digital Disruptors Award.
In the coming months, an additional cache will be added by ITS in UQ’s Prentice Building data centre. A new DNN GS14k will augment the existing DDN GS7k and IBM ESS, bringing the total local cache there to around 2 PB of usable storage.
"What is important is that data is held in the cache while it is actively being used to allow real workloads and moved off as it becomes stale to allow full use of the resource,” said ITS’s Dr Leslie Elliott. “When it is no longer required, it is automatically archived, and retrieved when accessed again. It is quite exciting to see this working in practice.”
MeDiCI caches are also present in QBI, IMB and RCC, bringing the local cache storage on the St Lucia campus to around 3.5 PB.
MeDiCI also underpins the UQ Research Data Management system (UQRDM), providing access to research data collections on a variety of platforms, from supercomputers to laptops. Because of this, PhD students with laptops can seamlessly process their data on the University’s high-performance computers Tinaroo, FlashLite, Awoonga and the new Wiener GPU cluster.
“We are seeing huge demand for Wiener,” said recent UQ Award for Excellence–Innovation winner Jake Carroll. “MeDiCI means we can make a researcher’s data available without them having to do anything special. It just appears.”
As part of the coming enhancements, ITS are upgrading the core campus network to support 100 Gbit/s optical connections, and will begin a rollout to key research buildings. This new capability underpins MeDiCI’s ability to move data seamlessly between the off-site Polaris Data Centre in Springfield and the campus caches.
Another series of rolling enhancements are planned for the Polaris back end storage system over the coming months, increasing the total archival capacity (allowing for three copies of data for resilience) to more than 50 PB.
“We believe this will set the scene to handle the deluge of data we are getting from our supercomputers and scientific instruments,” said RCC Director Prof. David Abramson. “We’ve got the best architecture for this. Now we have to add storage as required to meet the exponential growth we are seeing.”
Users who believe they will have significant data collections are encouraged to either apply for storage through the QRIScloud portal or tick the “HPC” box when completing a UQRDM request. (See the "Research Data Management at UQ" column in RCC News' July 2018 issue for more information.) "This will ensure we allocate the space on the appropriate back end platform,” said Prof. Abramson.
Researchers who wish to attach computers and devices to MeDiCI are encouraged to contact the RCC Support Desk for help: support@rcc.uq.edu.au.
