Home page > HPC access

PRACE Preparatory Access - cut-off evaluation in July 2011

Monday 5 September 2011

Find below the results of the cut-off evaluation of the 1st of July 2011 for the PRACE Preparatory Access Calls.

Type A - Code scalability testing

Project name: NadiaSpectral

Project leader: Marc Buffat, Université Lyon 1, LMFA, Villeurbanne, France
Collaborators: Julien Montagnier / Anne Cadiou / Lionel Le Penven, CNRS, Ecully, France
Research field: Engineering and Energy

Abstract

"Study of by-pass turbulent transition in wall bounded flow" The problem of transition from laminar to turbulent flow has been the subject of intense study in the field of fluid mechanics since the pioneer work of Reynolds (1883). Simple wall bounded flows, as periodic channel flows or zero pressure Boundary Layers, have been intensely studied by many researchers using Direct Numerical Simulation (DNS). In our team, we study study turbulent by-pass transition of boundary layers in the entrance region of a plane channel. For DNS, very high accuracy of the numerical method is mandatory to obtain meaningful results. Furthermore, as the computational domain is very large in the stream-wise direction, we need to solve for large numbers of modes (from 10^7 to 10^9 modes), and use massively parallel computers. We have thus developed a highly parallel, spectral Galerkin method, based on an orthogonal decomposition of the divergence free velocity field into 2 orthogonal solenoidal velocity fields (Buffat et al, Computer & fluids 2011) . We have implemented the method in an efficient numerical tool, NadiaSpectral (http://ufrmeca.univ-lyon1.fr/ buffa...), with a very high parallel efficiency, checked on the IBM BG at IDRIS and SGI at CINES ( up to 80% on BG with 16384 computational core with 10^9 spectral modes) . The objective of the project is to carry on tests of scalability of the NadiaSpectral code on the PRACE computers in order to run Peta-Scale simulations.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Computer sytem: JUGENE, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Scalability testing for large scale simulation of the effects of deep brain stimulation

Project leader: Christian Hauptmann, Forschungszentrum Jülich GmbH, Jülich, Germany
Collaborators: Martin Ebert, Forschungszentrum Jülich GmbH, Jülich, Germany
Research field: Medicine and Life Sciences

Abstract

The occurrence of resting tremor in Parkinson’s disease is correlated to pathological synchronization of particular neuronal populations located in the thalamus and the basal ganglia. In the healthy state these populations fire in an uncorrelated manner. Electrical high-frequency deep brain stimulation (HF DBS) emerged to be the standard therapy for movement disorders like Parkinson’s disease (PD).

A stimulation technique which specifically counteracts pathological neuronal synchronization processes by desynchronization is the so called coordinated reset (CR) stimulation. Apart from the reduction of adverse effects, CR DBS aims at curative or at least long-lasting effects that persist after cessation of stimulation by unlearning of pathological strong synaptic interactions.

The goal of our studies is thus to improve DBS therapy by investigation of new pulse shapes for effective neuronal stimulation, different stimulation techniques (HF and CR) and their influence on a computational model of the basal ganglia.

The ultimate goal of our study is to optimize stimulation protocols and parameters for clinical use.

To study the emergence of synchronized neuronal activity in Parkinson’s disease, we focus on the interaction of two major neuronal populations: A population of bursting neurons from the subthalamic nucleus (STN) interacts with a population of globus pallidum exterior (GPe) neurons. The STN neurons are recurrently inhibited by the GPe neurons.

The STN model neurons interact with each other through chemical synapses. In several experiments it has been demonstrated, that the strength of the synaptic interaction is changed depending on the preceding neuronal activity. Following experimental results obtained in populations of bursting neurons, the activity induced changes of the synaptic strengths are controlled by the timing of the bursts. In this context symmetric and smooth plasticity functions, which control the induced synaptic changes, are experimentally observed. The plasticity rule used in this study will take into account these findings.

HF and CR stimulation will be applied to the modeled target structure. A CR stimulus consists of a sequence of resetting stimuli (typically brief high-frequency stimulus trains) which are administered via the different stimulation sites. The single resetting stimuli are shifted in time with respect towards each other. The delay between the subsequent resetting stimuli should optimally be chosen equal to tau/n, where tau T, T is the mean period of the synchronized oscillation, and n is the number of stimulation sites.

The mathematical models will consist of a set of time delayed differential equations. We will use a Hodgkin-Huxley-type neuron model. Up to 10.000 neurons will be considered in each simulated population. The neurons will communicate with each other through chemical and electrical synapses. The simulation framework NEST will be used to implement the neuronal network.

Computer sytem: JUGENE, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Three-Dimensional Simulations of Core-Collapse-Supernova-Explosions of Massive Stars Applying Neutrino Hydrodynamics

Project leader: Hans-Thomas Janka, Max-Planck-Institut für Astrophysik, Garching, Germany

Collaborators: Florian Hanke / Bernhard Mueller, Max-Planck-Institut für Astrophysik, Garching, Germany; Andreas Marek, Rechenzentrum der Max-Planck-Gesellschaft, Garching, Germany
Research field: Astrophysics

Abstract

Supernova explosions of massive stars are among the most powerful cosmic events. They give birth to neutron stars and stellar black holes, produce strong neutrino and gravitational wave signals, and are a prime source candidates of chemical elements from iron to plutonium. The details of the physical mechanism that leads to the final explosion of the star are not yet fully understood. In this project we perform the currently most advanced simulations of the supernova evolution of massive stars and treat the neutrino-matter interactions in the supernova core with unprecedented accuracy. In this project we plan to move towards three-dimensional models of core collapse supernovae with detailed neutrino-transport.

Due to the limitations of the available computing power neutrino radiation-hydrodynamics simulations have long been forced to assume axi-symmetry (2D), thereby imposing restrictions on the convective flow pattern in the neutrino-heated region behind the supernova shock. While axi-symmetric models capture the interplay of convection and neutrino heating to some extent and thus yield marginal explosions in some instances, three-dimensional simulations will be indispensable to determine true understanding of the mechanism and consequences of core-collapse supernovae.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: NMMB/BSC-CTM porting and scalability test - extension

Project leader: Oriol Jorba, Barcelona Supercomputing Center - Centro Nacional de Supercomputación, Barcelona, Spain
Collaborators: Luca Telloli / Kim Serradell, Barcelona Supercomputing Center - Centro Nacional de Supercomputación, Barcelona, Spain.
Research field: Earth Sciences and Environment

Abstract

The Earth Sciences Department of the Barcelona Supercomputing Center (ES-BSC) is currently developing a new fully on-line coupled chemical weather prediction system for research applications and experimental forecasts at sub-synoptic and mesoscale resolutions on global and regional domains. The new system, namely NMMB/BSC-CTM, is based on the Nonhydrostatic Multiscale Model on the B Grid (NMMB), recently developed at National Centers for Environmental Prediction (NCEP).

The main feature of NMMB/BSC-CTM is its online coupling of chemistry and meteorology. The new chemical system component solves the gasphase tropospheric chemistry and the life cycle of the mineral dust, and will soon include other relevant aerosols (sea salt, black carbon, organic carbon and sulfate). The direct effect of mineral dust on the radiative budget is already implemented, and allows to further study mesoscale processes associated with air pollution and its interactions with meteorology, both at high resolution and on a global scale.

NMMB/BSC-CTM currently benefits of strong collaboration ties between ES-BSC, NCEP, the Technical University of Catalonia, the University of Murcia, the University of California Irvine, the NASA Goddard Institute for Space Studies, and the International Research Institute for Climate and Society. ES-BSC considers porting and evaluating NMMB/BSC-CTM on different high-performance computational platforms a top priority, before making the code generally available to the scientific community. This proposal is the continuation of PRACE project 2010PA0419, and it has the main objective to finalize the porting works started in the first preparatory access project.

We aim to finalize the porting and investigate scalability and performance of the NMMB/BSC-CTM model onto the CURIE supercomputer, which uses a different processor architecture that Marenostrum at BSC-CNS, but a similar memory-to-core ratio. Previous experience on the Marenostrum supercomputer allows us to define specific problems that the next generation of supercomputers should allow to solve. We target two main configurations: one using current resolution of atmospheric models, the other at higher horizontal resolution reaching the foreseen capabilities of next decade numerical weather prediction systems. Initial work on the CURIE system has pointed out some issues to be addressed prior to be prepared for the scalability tests.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Computational Molecular Modeling of the Materials and Processes Relevant to Geological Nuclear Waste Disposal

Project leader: Andrey Kalinichev, Ecole des Mines de Nantes, Laboratoire SUBATECH, Nantes, France
Collaborators: Narasimhan Loganathan / Brice Firmin Ngouana Wakou, Ecole des Mines de Nantes, Laboratoire SUBATECH, Nantes, France
Research field: Earth Sciences and Environment

Abstract

Safe and sustainable management of nuclear waste poses major scientific challenges to make the environmental footprint of nuclear energy as small as possible for a long period of time (up to 1 million years). This requires a detailed understanding of radionuclides’ interactions with natural and engineered barriers (consisting mostly of clay and cementitious materials used to protect the environment) and their behavior in the geosphere over time- and distance- scales spanning many orders of magnitude from the molecular-level chemical reactivity to the larger scale geochemical mobility of radionuclides in macroscopically heterogeneous systems.

Computational molecular modeling of materials for nuclear waste disposal applications is the primary objective of our research in the framework of the industrial chair "Storage and Management of Nuclear Waste" recently created at the Ecole des Mines de Nantes with support from ANDRA, AREVA, and EDF. Clay rock formations of nuclear waste repositories contain significant amounts (up to 1 mass %) of organic matter. The effects of natural and anthropogenic organic molecules on the mobility and toxicity of various elements in the context of nuclear waste storage are not yet well understood and are the priority topics of the present project. We address these problems on the fundamental molecular level by performing detailed quantitative studies of the energetic, structural, and dynamic aspects of different interaction mechanisms between radionuclides, organic matter, and clay particles using computational molecular modeling techniques. Free energies of adsorption and other thermodynamic and structural parameters obtained through the potentials of mean force calculations will then be utilized to significantly improve the predictive capabilities of the thermodynamic and geochemical models used for the performance assessment of nuclear waste repositories.

From a more general perspective, similar clay-like hydrous inorganic interfaces are ubiquitous in many natural and technological environments. Thus, our research will also contribute to the development of viable carbon sequestration technologies, new membranes for water purification, and other research and engineering fields where detailed molecular scale understanding of materials and processes represents one of the most important cross-cutting fundamental problems of materials and environmental chemistry concerned with more efficient production and use of clean energy and clean water.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: ARPEGE-OASIS-NEMO climate model set up for high resolution climate experiments

Project leader: Eric Maisonnave, CERFACS, Global Change, Toulouse, France
Collaborators: Pierre-Antoine Bretonnière / Gabriel Jonville / Sophie Valcke / Christophe Cassou / Laure Coquart, CERFACS, Toulouse , France; Jean-Marc Molines, CNRS, Grenoble, France; Michel Déqué / Jean-Philippe Piédelièvre, Météo-France, Toulouse, France
Research field: Earth Sciences and Environment

Abstract

This project deals with preparing the ARPEGE/OASIS/NEMO Climate model, in order to complement the international intercomparaison exercice (CMIP5), which modelization will provide a basis for the Intergovernmental Panel on Climate Change (IPCC) 5th Report in 2013.

The preparatory project technical objectives are porting, set up on at least 1000 cores and optimizing the 3 coupled model different components ARPEGE,OASIS and NEMO, at high resolution (720 x 360 x 31 for atmosphere, 1442 x 1021 x 46 for ocean) on PRACE architectures.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: The Genetic Hybrid Algorithm

Project leader: Ralf Östermark, Åbo Akademi University, Turku, Finland
Research field: Mathematics and Computer Science

Abstract

GHA is a general purpose platform for parallel processing of large-scale computational problems. Good scalability has been demonstrated on the PRACE supercomputers with up to 8000 cores in large scale (mixed integer) non-linear optimization problems and vector-valued time series problems. An accelerator function is included in critical stages, allowing the connection of external algorithms to the platform. Thanks to the accelerator function, the researcher can pose the important question: How can I solve the current numerical problem using the best available algorithms in the world. The accelerator function also supports the development of special purpose algorithms to be monitored by GHA. Several algorithms for vector-valued time series models and non-linear optimization have been connected as support libraries to GHA and successfully tested. The time series support libraries have been coded, tested and reported by Ralf Östermark based on relevant literature. The source code for the optimization support libraries has been delivered by research groups belonging to the leading developers of non-linear optimization methodology (e.g., NLPQLP by Schittkowski, Univ. of Bayreuth, Germany, SNOPT by Gill, Murray and Saunders from UCLA & univ. of Stanford, FSQP by Lawrence et al and DNCONG included in the IMSL package).

All output is sorted in ascending order based on the value of the merit function BEST = BEST.F + p*BEST.Dev, where p is a penalty based on possible deviations from the feasible solution space and the size of the branch-and-bound tree needed to obtain the solution. The Total time column approximates the time required for the job, including the post-processing done by the root in receiving the solutions from the mesh. Since GHA and its support libraries are installed in every available processor, communication between processors can be minimized. The job could actually be terminated immediately once the solution has been obtained somewhere in the mesh. The processor that has solved the problem could write its output on a file, e.g., result.n and broadcast an asyncronic early interrrupt message to the mesh that will terminate the other processors. However, at least as researchers, we are interested in the solutions generated by all processors. Therefore, we let all processors deliver their best solution to the root for sorting and producing the summary output file once the early interrupt message has been broadcasted. In large MINLP-problems this post-processing will have a negligible impact on the total time absorbance.

Computer sytem: JUGENE, Gauss/FZJ
Resource awarded: 100 000 core-hours

Type B – Code development and optimization by the applicant (without PRACE support)

Project name: Simulations of Turbulent, Active and Rotating Sun and Stars (STARS2)

Project leader: Allan Sacha Brun, CEA-Saclay, Gif-sur-Yvette, France
Collaborators: Rui Pinto / Nicolas Bessolaz / Antoine Strugarek, CEA-Saclay, Gif-sur-Yvette, France
Research field: Astrophysics

Abstract

The STARS2 project, funded by ERC grant 207430, aims at modelling on massively parallel supercomputers in a self-consistent and three-dimensional way, the complex, time dependent and nonlinear dynamics operating in the Sun and stars. In particular we wish to understand how stars generate the wide variety of magnetic activity that is observed, with the Sun - given its proximity and its influence on our technical society - playing a central role in characterizing, studying, and constraining the dynamical processes acting in stellar convection and radiation zones. Studying the solar-stellar connection is crucial because it will allow us to understand why depending on the spectral type of the star considered, this activity can be cyclic (solar type stars), irregular (very low mass stars), or for massive stars, without any activity or simply a modulated one. The mechanism thought to be at the origin of the magnetism seen in solar (late) type stars is likely to be linked to dynamo action in the upper convective layers of such stars. To achieve these challenging scientific goals, the STARS2 project performs realistic high performance numerical simulations of the Sun and of stars of various spectral types using the ASH (Anelastic Spherical Harmonics) code. These simulations are at the front-edge of current research in astrophysics, they require the use of the latest class of supercomputers available in Europe.

Understanding the nonlinear interactions between convection, turbulence, shear, rotation and magnetic fields in stars is the main scientific goal of this ERC project (www.stars2.eu).

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: High Performance Computing For Highly Non-Linear Explosive Fluid-Structure Simulations

Project leader: Vincent Faucher, Commissariat à l’Energie Atomique et aux Energies Alternatives, Gif sur Yvette, France.
Research field: Engineering and Energy

Abstract

The project is dedicated to large scale fluid-structure simulations in explosive situation, i.e. fast transient phenomena, high pressure levels, both geometric and material non-linear structural behavior, structural failure & fragmentation. EUROPLEXUS fast transient dynamics software (CEA/European Commission) will be used, and proposed work is part of REPDYN Project, sustained by French ANR (COSINUS 2009 call), involving CEA, EDF, ONERA, LaMSID, LaMCoS INSA Lyon and INRIA.

EUROPLEXUS is used for a wide range of applications, from nuclear safety to civil protection and aeronautics, through a development consortium involving CEA, EC/JRC, EDF, ONERA and Samtech SA.

Full coupling between fluid and structure is handled, as well as unilateral contact between fragments of structure. Fluid-structure interaction may rely either on conforming meshes for both entities, with ALE rezoning for unstructured fluid grid, or on immersed boundaries approaches, with topologically disconnected meshes for fluid and structure and potential use of a structured fluid grid. Both interaction methods can be mixed within a simulation, for example considering an explosion within a vessel with internal structures. Conforming FSI is used for interaction between fluid and vessel, whereas non-conforming FSI is used for interaction between fluid and immersed structures.

Refined fluid meshes are required to perform accurate simulations, especially when strong shock waves occur or when multi-component fluids with interfaces are considered. Long simulation times, with respect to classical time scales for fast transient dynamics, i.e. several seconds compared to classical stability step for time integration limited to several microseconds, are also required to completely simulate the structural ruin process and its interaction with fluid flow. This yields typically to 5M to 10M cells for fluid mesh, 0.1M to 0.5M cells from structural mesh and 1M to 5M cycles for transient time integration.

The project objectives are : 1. Test scalability of EUROPLEXUS current MPI parallel algorithm for more than 512 CPU cores for this kind of simulations, 2. Identify locks and help design improvements, for both fully distributed parallelism and hybrid shared/distributed parallelism, currently under development within REPDYN project.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Protein-protein interactions : massive free energy computations

Project leader: Michel Masella, CEA, Gif-sur-Yvette, France
Collaborators: Philippe Cuniasse / Guillaume Collet, CEA, Gif-sur-Yvette, France
Research field: Medicine and Life Sciences

Abstract

Protein-protein interactions play a key role in most of the biological processes involved in living cells, and their modeling represents one of the major focus in nowdays research. The development of accurate and efficient numerical methods allowing one to evaluate their nature, strength and selectivity is a major challenge in the computational biophysics field. Such a task is particularely challenging because of the size of microscopic systems to be considered (usually, proteins are made of thousends of atoms), and because of the need of accounting for the incidence of their chemical environnement. We developped recently new, accurate and efficient molecular modeling techniques allowing one the use of sophisticated approaches to model the interactions within proteins, as well as the use of efficient coarse-grained approaches to model the protein chemical environment. These methods are implemented in our POLARIS(MD) code, a parallel simulation program, allowing to evaluate the incidence of mutations at the protein surfaces on the stability of protein/protein edifices. The aim of the present projet is to apply this code to theoretical investigate the impact of a large set of mutations on the interaction of two particular protein-protein systems: the beta-trypsin and alpha-chymotrypsin proteins interacting with a family of bovine pancreatic trypsin inhibitor (BPTI) mutants.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Three-Dimensional simulation of high frequency wave propagation in a Mode Converter (3DMC) Project leader: Jonathan Neudorfer, Universität Stuttgart, Stuttgart, Germany
Research field: Engineering and Energy

Abstract

The 3DMC project deals with the three-dimensional simulation of high frequency waves in waveguides. Such waveguides are used in high power microwave generators. The generated wave modes have to be shaped in a mode converter to assume a gaussian beam that is emitted at the exit of the device. So far, two-dimensional simulation methods have been used to simulate this problem. In recent years, advances in the field of numerics have allowed the three-dimensional simulation of such a mode converter, directly computing the time-dependent Maxwell equations without any algorithmic or geometric approximations. Still, the required resources are enormous. A complete simulation run of a realistic device takes at least 24 hours on 2048 CPU cores. Moreover, the large amount of memory required by the simulation results in problems with the scalability on Infiniband clusters.

This project intends to improve the efficiency of parallel computations on large nodes by adding a shared memory parallelization to the existing MPI parallelization. The improvements are expected to enable simulations of state-of-the-art mode converters, allowing for the first time to inspect the three-dimensional wave propagation in such devices.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Aero-Elastics with Distributed Octrees

Project leader: Sabine Roller, German Research School for Simulation Sciences, Aachen, Germany
Collaborators: Metin Cakircali / Manuel Hasert / Daniel Harlacher / Harald Klimach / Jens Zudrop, German Research School for Simulation Sciences, Aachen, Germany
Research field: Engineering and Energy

Abstract

Aeroelastic phenomena need to take into account the interaction of inertial (dynamic), elastic (structural) and aerodynamic (fluid) forces. The interaction of these forces tackled in fluid-structure interactions (FSI) are of great importance in many engineering fields, for example in the design of bridges or wings. The simulation of unsteady compressible fluid flows around moving and deforming structures is a very complex and computationally expensive task. In our work, we apply the partitioned approach, where fluid and structural domains are solved separately, yet tightly interacting at their interfaces. The part with the highest computational resource demand in this coupled setup is the fluid solver. Therefore this project is mainly concerned with the efficient simulation of this part and a scalable coupling. Suitable methods and data structures are implemented to allow for scalable parallel implementations using a discontinuous Galerkin (DG) method of high order, which is highly local in space and time, and therefore highly scalable. It is implemented as an arbitrary high order PnPm scheme [1].

The scheme is well suited for high locality, hp-adaptivity and capturing of shocks as they might appear in compressible flows. The major challenge in h-adaptivity is to implement a simple, yet efficient parallel algorithm. In our approach, we use the octree data structure on the fluid domain in order to reduce the intensive computational demand of h-adaptivity up to some extent while keeping the algorithm simpler. As the deformation proceeds, the FSI coupling interface and mapping of deformed meshes onto Eulerian domain are also expensive tasks, which can be simplified by the usage of an underlying octree structure. The mesh resolution around the FSI interface has to be high enough to resolve the complex geometries. This results in extreme CPU and memory usage in those regions. These types of problems require thousands of CPU-cores on HPC systems per simulation in order to reach a solution in a reasonable time, especially when considering design cycles needed in industry to manufacture a product. Thus, the method has to be highly scalable throughout the entire simulation cycle as modern supercomputing facilities provide their resources in massively distributed [2]. This is addressed by the specific data layout and highly local numerical scheme.

In order to increase the overall computational efficiency, we separate the fluid domain into two different parts with adapted properties. The Eulerian domain consists of basic cubes, which simplify many computations and therefore reduce the overall computational costs. On the other hand, the Arbitrary Lagrangian Eulerian (ALE) domain is used around the FSI coupling interface only, as the moving elements are computationally more expensive. This allows for a well-resolved interface that enforces the kinematic continuity, while maintaining conservation laws in the fluid domain.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours