Nimrod
Nimrod
Parametric computational experiments are becoming increasingly important in science and engineering as a means of exploring the behaviour of complex systems. For example, an engineer may explore the behaviour of a wing by running a computational model of the airfoil multiple times, while varying key parameters such as angle of attack and air speed. The results of these multiple experiments yield a picture of how the wing will behave in different parts of parametric space.
The same process can be applied in other experiments that involve parametric modelling.
Nimrod is a specialised parametric modelling system.
It uses a simple declarative parametric modelling language to express a parametric experiment.
It provides the machinery to automate the task of formulating, running, monitoring, collating, presenting and visualising the results from multiple individual experiments.
Nimrod incorporates distributed scheduling so that the appropriate number and kind of resources to complete the job, e.g., HPC and virtual machines, can be selected and used.
Nimrod helps researchers run computations remotely on the cloud. It can turn your laptop into a supercomputer. With Nirmod you can run many jobs — millions if need be.
Nimrod is the name of a set of tools, including Nimrod-G (Grid), Nimrod-K (Kepler), Nimrod-O (Optimisation) and Nimrod-E (Experiment):
- Nimrod-G is a tool for people to access the computational resources of the grid.
- Nimrod-K is a development platform that allows you to bring together your various computations, modeling, visualisation and storage to create a flow of work (i.e. a scientific workflow). More importantly, Nimrod-K enables the use of HPC and cloud resources for computationally expensive tasks in a workflow. It also enables you to share that workflow with collaborators and other researchers.
- Nimrod-O enables more refined searching across potentially millions of combinations of research parameters.
- Nimrod-E helps you work out what parameters are relevant to your experiment.
RCC provides training for all the Nimrod tools. To request training, contact our Nimrod experts Dr Hoang Nguyen (h.nguyen30@uq.edu.au) or Zane Van Iperen (z.vaniperen@uq.edu.au).
Embedded Nimrod
Embedded Nimrod is a version of the Nimrod high-throughput computing tool that can be utilised within batch jobs on a HPC.
Embedded Nimrod builds a miniature Nimrod environment within your Portable Batch System (PBS) job resources and starts processing the experiment plan file you included with the job submission.
A win-win for UQ’s HPCs
RCC HPC Manager Dr David Green said he could see Embedded Nimrod’s potential from the outset to better manage high-throughput computing workloads on HPC clusters.
“We were often seeing scatterings of small footprint jobs, especially on HPC Tinaroo, that were causing issues for the scheduling of larger footprint jobs. Embedded Nimrod provides us with another powerful option to manage workloads on our HPCs," he said.
User benefits
High-throughput workloads submitted as whole jobs or job arrays are treated as distinct batch system entries and there is potential for delays while scheduling. However, once an Embedded Nimrod job commences running, it stays resident on the HPC until all the tasks to be done are completed.
Embedded Nimrod manages the sequence of tasks internally without the need to go back to the batch system to be allocated resources. The resources you requested are yours until Nimrod has completed all the tasks.
What to note
To use Embedded Nimrod means you need to know how to write a plan file for Nimrod, which is not too difficult and RCC provides documentation to assist you that contains sample code for different scenarios.
You also need to think slightly differently about your job resource request, but again, this is not too difficult, and RCC can assist you (email: rcc-support@uq.edu.au).
Typically, you will want to request a big resource footprint so that multiple tasks can be active at the same time. The queuing for larger footprint resources is usually longer than small job footprints, however, once the wait is over there is no queue-wait after that, just the time it takes for each task to be completed.
What workloads are already using Embedded Nimrod?
Researchers from several disciplines have kindly helped RCC by being early adopters or by providing use cases for us to test the Embedded Nimrod deployment. These include:
- Inland drayage operations (Dr Mahboobeh Moghaddam’s research, School of Economics)
- Traffic simulations (School of Civil Engineering)
- Genome analysis (School of Biological Sciences)
- Molecular simulations (School of Chemical Engineering)
- Threatened species (School of Biological Sciences)
- Spatial ecology (School of Biological Sciences)
- Brain MRI (Centre for Advanced Imaging).
The structural difference between conventional web portal Nimrod and the new Embedded Nimrod variety:
