Home page > HPC access

PRACE Preparatory Access - cut-off evaluation in March and May, 2011

Tuesday 19 July 2011

Find below the results of the cut-off evaluation of March and May, 2011 for the PRACE Preparatory Access Calls.

Type A - Code scalability testing

Project name: Optimizing the performance of bias-exchange metadynamics for the calculation of free energy in complex biomolecular transitions

Project leader: Laio Alessandro, SISSA, Statistical and Biological Physics, Trieste, Italy
Research field: Chemistry and Materials

Abstract This project aims at optimizing the computational performance on a massively parallel machine of bias-exchange metadynamics (BE-META), one of the state-of-the-art approaches for obtaining free energy landscapes of complex biomolecular systems. BE-META is based on the combined usage of two two softwares. The first one is the Gromacs package, a classical molecular dynamics engine; the second one is PLUMED, a plugin that allows computing free energies within different algorithmic frameworks. Standard molecular dynamics of biomolecular systems are notoriously affected by a limited scalability, due to the collective communications necessary to evaluate long-range interactions and their intrinsic heterogeneity. Bias-exchange metadynamics is based on the simultaneous simulation of several replicas of the same systems. These replicas exchange their positions, according a montecarlo scheme, only relatively rarely. In this way, communications are strongly reduced and an excellent scaling is made possible. The use of this algorithm in Tier-0 machines could lead to explore details of the free energy landascapes of complex biomolecular processes not permitted on the usual supercomputers.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Scalability of ’ELMFIRE’ on JUGENE

Project leader:Jan Åström, CSC, Espoo, Finland
Collaborators: Arthur Signell, Åbo Akademi University, Abo, Finland
Research field: Engineering and Energy

Abstract The ELMFIRE code is one of the important European simulation codes for fusion plasmas. These codes are of vital importance for solving the remaining technical problems related to the functionality of the first power-plant scale fusion reactor, ITER, under construction in France. So far ELMFIRE has been run on computer systems up to 2048 cores. The objective of the project is to port ELMFIRE on JUGENE and test scalability beyond the present maximum.

Computer system: JUGENE, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Testing the scalability of enhanced sampling technique to study the interaction between amyloid-beta peptide and gold-surface in water

Project leader: Luca Bellucci, CNR-Istituto nanoscienze, Modena, Italy
Collaborators: Stefano Corni, CNR-Istituto nanoscienze, Modena, Italy
Research field: Chemistry and Materials

Abstract The interaction between proteins and inorganic material surfaces is of paramount importance in natural system and currently is a ’hot topic’ for both experimental and theoretical disciplines. It is worth remembering that protein/inorganic-surface interaction in recent years have been played a pivotal role in developing smart material, nanotechnology and molecular electronics. Nanoparticles have been proposed as the basis for innovative diagnostic and therapeutic approaches, applications which define the emerging field of nanomedicine. Gold nanoparticles, for example, selectively attach the amyloid beta peptide aggregates (related to the Alzheimer’s disease) and interfere with their growth [1].

Despite the pivotal role of protein-surface interactions in all of these technologically and societally relevant fields, they are still poorly understood. Computational approach represents a powerful methodology to rationalize the peptide/inorganic-surface interface, consequently a series of computational tools (i.e. atomistic or non-atomistic classical force filed, quantum mechanics computational protocols, etc.) have been made and have been tested in affordable peptide/interface systems of interest [2,3].

The purpose of this project is test molecular dynamics simulation protocols based on Hamiltonian Replica Exchange Molecular Dynamics (H-REMD) [4] to improve the conformational sampling of the beta-amyloid/gold system in water. The scalability of this protocol will be used to plan and to submit a new proposal to study the complex interaction between the gold surface and beta amyloid peptide. In fact, although the computational tools have been demonstrated to have predicting power to simulate complex protein-surface systems in water, it is necessary to set up the scalability of the H-REMD protocols and to estimate the amount of the computational resources needed for a comprehensive description of the system behavior.

References
[1] M.J kogan et al. "Nanoparticle-Mediated Local and Remote Manipulation of Protein Aggregation" Nano Lett 6, 110 (2006)
[2] F. Iori, R. Di Felice, E. Molinari, S. Corni, "GoIP: An Atomistic Force-Field to Describe the Interaction of Proteins With Au(111) Surfaces in Water" J. Comput. Chem. 30, 1465 (2009).
[3] M. Hoefling, F. Iori, S. Corni, K.E. Gottschalk, "Interaction of Amino Acids with the Au(111) Surface: Adsorption Free Energies from Molecular Dynamics Simulations" Langmuir 26, 8347 (2010).
[4] R. Affentranger, I. Tavernelli E. E. Di Iorio, "A Novel Hamitonian Replica Exchange MD Protocol to Enhance Protein Conformational Space Sampling", J.Chem. Theory Comput. 2, 217 (2006).

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Modeling late stages of spin-coating evaporation

Project leader: Olivier Benzerara, CNRS, Strasbourg, France
Collaborators: Falko Ziebert, CNRS, Strasbourg, France / Joerg Baschnagel, CNRS, Strasbourg, France / Hendrik Meyer, CNRS, Strasbourg, France / Alexander Blumen, Freiburg University, Freiburg, Germany
Research field: Chemistry and Materials

Abstract Spincoating of dilute polymer solutions onto substrates is a powerful technique that allows the deposition of thin polymer films of nanoscopic dimensions. It is used in numerous technological applications, e.g. for protective coatings of surfaces, microelectronics, lubrication and others. However, the structure and the properties of such glassy polymer films produced by rapid evaporation of the solvent (“solvent quench”) are poorly understood theoretically as the polymer structure is frozen-in into out-of-equilibrium configurations which appear to be hard to relax even by long annealing at elevated temperatures.

Together with experiments done in the experimental polymer physics group in Freiburg, the aim of this project will be to undertake statistical (continuous time random walk) and simulation (molecular dynamics) studies to investigate the properties of spin-coated films and their relaxational behavior.

Questions to be addressed involve:
* Is the film homogeneous in the plane?
* What is the inhomogeneous structure in the normal direction?
* Is stress induced by the out-of-equilibrium production process that might affect the film stability?
* What governs the dynamics inside the thin film, both of the polymers and the residual solvent as a function of temperature and solvent concentration?

Specific predictions will be tested by the experimental group to finally establish a theoretical model that makes testable predictions about experimentally controllable factors which are important for the spin-coating process.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Ion permeation in the single of the bacterial porin NanC

Project leader: Paolo Carloni, German Research School for Simulation Sciences GmbH, Computational Biophysics Lab, Juelich, Germany
Collaborators: Emiliano Ippoliti, German Research School for Simulation Sciences GmbH, Computational Biophysics Lab, Juelich, Germany / Paul Strodel, German Research School for Simulation Sciences GmbH, Computational Biophysics Lab, Juelich, Germany
Research field: Medicine and Life Sciences

Abstract The passage of small ions through biological cell membranes is modulated by channel proteins connecting both compartments involved. Understanding the precise mechanism of the permeation process is not only an important topic for basic research, but opens the way to develop therapeutics for many diseases related to the (mal)functioning of these proteins. It is the aim of this project to explain differences in sodium and potassium permeation through the protein channel NanC from E. coli by computational methods. To this end, we aim to calculate the changes in free energy associated with the passage of each ion species, respectively. This gives insights into the specific atomic interactions contributing to the free energy and so allows working out differences between the sodium and potassium transportation mechanism. Free energy calculations of ion permeation in other classes of ion channels have been already published by various groups, using classical force fields for the energy function. However, it was shown recently that quantum effects might play a decisive role in describing the factors controlling ion permeation in these systems. This calls for the application of quantum mechanical methods to overcome the limitations of classical ion force fields. Since the size of the simulation system precludes treating the whole system by quantum mechanical methods alone, a combined quantum mechanics/molecular mechanics description is appropriate. Here, the core atoms (ions, close amino acids and surrounding water molecules) are treated by a quantum mechanical calculation, and the remaining system bulk is treated by a classical force field. Such calculations are still very expensive, but tractable by means of a supercomputer.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Insights into the mechanism of oncogenesis of the mutant protein PI3Ka from Molecular Dynamics simulations

Project leader: Zoe Cournia, Biomedical Research Foundation, Academy of Athens, Pharmacology, Athens, Greece
Collaborators: Argiris Efstratiadis, Biomedical Research Foundation, Academy of Athens, Pharmacology, Athens, Greece / Gkeka Paraskevi, Biomedical Research Foundation, Academy of Athens, Pharmacology, Athens, Greece
Research field: Medicine and Life Sciences

Abstract The kinase PI3Ka is involved in fundamental cellular processes such as cell proliferation and differentiation. PI3Ka consists of a catalytic (p110a) and a regulatory subunit (p85a). It was recently established that PIK3CA, the gene encoding the catalytic subunit of PI3Ka, is frequently mutated in human malignancies. 80% of these mutations are located in either one of two hotspots: (a) glutamic acid is replaced by lysine in exon 9 (E545K), and (b) a histidine is changed to arginine in exon 20 (H1047R). Both types of these mutations increase the kinase activity of the enzyme, upregulate the downstream AKT pathway and VEGF signaling, and thus stimulate cell transformation, tumorigenesis and angiogenesis. Understanding how these mutations lead to the increased PI3Ka activity is paramount to developing new treatments for cancer. Mutated proteins such as PI3Ka can promote tumorigenesis due to conformational changes that result in upregulated enzymatic activity. For example, the H1047R mutation possibly provides extra mobility to the activation loop of PI3Ka, as a consequence of the disruption of a hydrogen bond that is broken in the mutant structure. This extra mobility is postulated to induce a larger area of the catalytic cleft compared to the wild-type (WT), which facilitates easier entrance of substrates (ATP and PIP2) and could yield enhanced substrate-to-product turnover potentially associated with oncogenicity. Using Molecular Dynamics (MD) simulations it is possible to confirm and capture the dynamics of such structural changes and provide insights into the mechanism driving tumor development in H1047R. Moreover, a positive-negative charge interaction between the two PI3Ka subunits, p85a and p110a, keeps the WT PI3Ka in a low activity state. The E545K mutation results in an amino acid of opposite charge, where the glutamic acid (negative charge) is replaced by lysine (positive charge). It has been recently proposed that in this oncogenic charge-reversal mutation, the interactions of the p110a subunit with the p85 subunit are abrogated, resulting in constitutive PI3Ka activation. MD simulations will be used here to monitor how the p85a-p110a interaction is lost and examine conformational changes differing among the WT and mutant as they occur. Therefore, the aim of this project is to monitor conformational changes between the wild-type and carcinogenic mutants H1047R and E545K and gain insights into the mechanism of oncogenesis. For this purpose, extensive Molecular Dynamics simulations of the wild type and mutants will be performed. Differences between these systems will be identified by means of structural, thermodynamic and kinetic properties calculated from the simulations. The program that will be used is NAMD. NAMD is a highly scalable and portable parallel molecular dynamics program for biophysical simulations, which has been scaled up to 32,000 cores. According to our preliminary work, long runtimes (20-100 ns) and 200,000-400,000 atoms (depending on the different system used) will be required to obtain molecular evidences of the oncogenic mechanisms. Therefore, we need sufficient computational facilities, which justify the need for running on a Tier-0 system.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours Computer system: Jugene, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: SPIC (Scalable Parallel Image Compositing)

Project leader: Xavier Cavin, INRIA Nancy-Grand Est, Villers-les-Nancy, France
Collaborators: Olivier Demengeon, INRIA Nancy-Grand Est, Villers-les-Nancy, France
Research field: Mathematics and Computer Science

Abstract Both in the industry and in the academic world, larger and larger numerical datasets are being produced on a daily basis, whether they are large Computer Aided Design (CAD) assemblies, results of numerical simulations or 3D scanned information just to cite a few. It is widely accepted that a single computer cannot render terabytes of data interactively, unless a huge preprocessing job can be applied to the data. In most cases, however, high-end parallel visualization solutions must be designed to tackle these ever-increasing amounts of data, sometimes on-the-fly as they are being produced.

Sort-last parallel rendering is an efficient technique to render very large datasets in parallel. In this kind of parallel applications, the dataset to be visualized is first partitioned and distributed across multiple processors. Each processor renders its subset of the dataset independently using its local resources; this stage does not imply any communication between the processors and scales perfectly. Then the computed images are composed together by a parallel image compositing algorithm and the final image is delivered. The parallel image compositing stage requires a lot of inter-processor communication, and may become the bottleneck of a sort-last parallel pipeline if it is not designed with sufficient care.

Since the early 1990’s, parallel image compositing has been widely studied, and numerous algorithms have been developed and experimented on many parallel computing architectures, ranging from supercomputers to PC clusters. These image compositing algorithms are commonly classified into three categories: Direct Send, Binary Swap and Parallel Pipeline. The two latest advances in the field include 2-3 Swap, which is an extension of the Binary Swap to non-power of two numbers of processors, and the Radix-k algorithm which unifies the Direct Send and Binary Swap and enables numerous other configurations.

We have developed a new parallel image compositing algorithm, named SPIC for Scalable Parallel Image Compositing. It has been designed to run effectively and scale across large numbers of processors interconnected by a high-speed low latency fat-tree topology. Our algorithm takes the network topology as an input and generates a communication pattern for the image compositing that allows benefiting from the full available bisection bandwidth.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Scalability testing of massively-parallel density functional theory

Project leader: Niall English, University College Dublin, Chemical & Bioprocess Engineering, Dublin, Ireland
Collaborators: John Tse, University of Saskatchewan, Saskatoon, Canada
Research field: Chemistry and Materials

Abstract This project seeks to perform benchmarking studies for massively-parallel ab initio molecular dynamics, with the goal of assessing scalability and performance.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Multi-scale BSP programming

Project leader: Gaetan Hains, Université Paris-Est Créteil (UPEC), Créteil, France
Collaborators: Chong Li, Université Paris-Est Créteil (UPEC), Créteil, France
Research field: Mathematics and Computer Science

Abstract We develop the Scatter-Gather parallel-programming and parallel execution model in the form of a simple imperative language named SGL. Its design is based on past experience with Bulk-synchronous parallel (BSP) programming and BSP language design. SGL’s novel features are motivated by the last decade’s move towards multi-level and heterogeneous parallel architectures involving multi-core processors, graphics accelerators and hierarchical routing networks in the largest multiprocessing systems. The design of SGL is coherent with Valiant’s Multi-BSP while involving a programming interface that is even simpler than the primitives of Bulk-Synchronous parallel ML (BSML). SGL appears to cover a large subset of all BSP algorithms while avoiding complex message-passing programming. It allows automatic load balancing and like all BSP-inspired systems, predictable, portable and scalable performance.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: NMMB/BSC-CHEM

Project leader: Oriol Jorba, Barcelona Supercomputing Center, Barcelona, Spain
Collaborators: Luca Telloli, Barcelona Supercomputing Center, Barcelona, Spain
Research field: Earth Sciences and Environment

Abstract The chemical composition of the atmosphere has been progressively perturbed by emissions of trace gases and aerosols connected to a variety of anthropogenic activities; these changes have important implications for urban, regional and global air quality, and climate change. In this sense, the European Environmental Agency identifies air pollution as responsible for the largest burden of environment-related diseases.

Modeling techniques are a valuable approach for research and assessment of these problems. In this sense, the European mother Directive 2008/50/EC establishes that “fixed measurements may be supplemented by modelling techniques and/or indicative measurements to provide adequate information on the spatial distribution of the ambient air quality”. Historically, air quality modeling systems (AQM) and numerical weather prediction models (NWP) were developed separately. This was plausible in previous decades when the resolution of NWP models was too poor for mesoscale air pollution forecasting, but due to the impressive increase in high performance computing over the past decade, nowadays this approach is under revision. In so-called on-line models, meteorological and chemical equations are solved simultaneously and consideration of feedback processes is now possible.

The Earth Sciences Department of the Barcelona Supercomputing Center (ES-BSC) is currently developing a new fully on-line coupled chemical weather prediction system for research applications and experimental forecasts at sub-synoptic and mesoscale resolutions on global and regional domains. The new system, namely NMMB/BSC-CHEM, is based on the Nonhydrostatic Multiscale Model on the B Grid (NMMB), recently developed at National Centers for Environmental Prediction (NCEP).

The main feature of NMMB/BSC-CHEM is its online coupling of chemistry and meteorology. The new chemical system component solves the gas-phase tropospheric chemistry and the life cycle of the mineral dust, and will soon include other relevant aerosols (sea salt, black carbon, organic carbon and sulfate). The direct effect of mineral dust on the radiative budget is already implemented, and allows to further study mesoscale processes associated with air pollution and its interactions with meteorology, both at high resolution and on a global scale.

NMMB/BSC-CHEM currently benefits of strong collaboration ties between ES-BSC, NCEP, the Technical University of Catalonia, the University of Murcia, the University of California Irvine, the NASA Goddard Institute for Space Studies, and the International Research Institute for Climate and Society. ES-BSC considers porting and evaluating NMMB/BSC-CHEM on different high-performance computational platforms a top priority, before making the code generally available to the scientific community.

In this proposal, we aim to port and investigate scalability and performance of the NMMB/BSC-CHEM model onto the CURIE supercomputer, which uses a different processor architecture that Marenostrum at BSC-CNS, but a similar memory-to-core ratio.

Previous experience on the Marenostrum supercomputer allows us to define specific problems that the next generation of supercomputers should allow to solve. We target two main configurations: one using current resolution of atmospheric models, the other at higher horizontal resolution reaching the foreseen capabilities of next decade numerical weather prediction systems.

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Discrete Logarithms on Elliptic Curves over composite extension fields

Project leader: Antoine Joux, Université Versailles St-Quentin-en-Yvelines, Versailles, France
Collaborators: Vanessa Vitse, Université Versailles St-Quentin-en-Yvelines, PRISM, Versailles, France
Research field: Mathematics and Computer Science

Abstract The goal of this proposal is to scale a recent algorithm for computing discrete logarithms on elliptic curves over composite extension field (mathematical description of the algorithm available at http://eprint.iacr.org/2011/020).

Computer system: CURIE, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Radiative Feedback from Early Cosmic Structures

Project leader: Llian Lliev, University of Sussex, Physics and Astronomy, Brighton, UK
Collaborators: Mellema Garrelt, Stockholm University, Stockholm, Sweden / Kyungjin Ahn, Chosun University, Gwangju, South Korea
Research field: Astrophysics

Abstract Reionization is believed to be the outcome of the release of ionizing radiation by early galaxies. Due to the complex nature of the reionization process it is best studied through numerical simulations. Such simulations present considerable challenges related to the large dynamic range required and the necessity to perform fast and accurate radiative transfer calculations. The tiny galaxies which are the dominant contributors of ionizing radiation must be resolved in volumes large enough to derive their numbers and clustering properties correctly, as both of these strongly impact the corresponding observational signatures. The ionization fronts expanding from all these millions of galaxies into the surrounding neutral medium must then be tracked with a 3D radiative transfer method which includes the solution of non-equilibrium chemical rate equations. The combination of these requirements makes this problem a formidable computational task. We propose to use a combination of new, much larger structure formation simulations and the implementation of more sophisticated modelling of the relevant physical processes to include for the first time in large-scale reionization simulations the effects of small-scale structure and in particular of minihaloes in a self-consistent way. Another strand of inquiry will focus on the effects the early rise of the inhomogeneous X-ray background and its effects on the intergalactic medium. The forming early galaxies, and the stars and accreting black holes within them emit copious amounts of radiation in all spectral bands, which in turn affects future star and galaxy formation.

With this proposal we request preparatory access to port and test the scalability of our structure formation and radiative transfer codes on both CURIE and JUGENE in preparation for the next call for production access.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: First principles design of a biocatalyst for water oxidation

Project leader: Carme Rovira, Parc Científic de Barcelona, Computer Simulation and Modeling Laboratory, Barcelona, Spain
Collaborators: Agusti Lledós, Universitat Autònoma de Barcelona, Belaterra, Spain / Pietro Vidossich, Universitat Autònoma de Barcelona, Belaterra, Spain / David Balcells, Universitat Autònoma de Barcelona, Belaterra, Spain / Victor Rojas, Parc Científic de Barcelona, Computer Simulation and Modeling Laboratory, Barcelona, Spain
Research field: Chemistry and Materials

Abstract A sustainable solution to the high demand of energy is among the big challenges our society is facing. Sunlight is a major source of renewable energy and it would be highly desirable to base our consumption entirely on it. Unfortunately, though we are nowadays able to capture radiative energy and convert it into electricity, deployment on the large scale is not yet possible because of limited storage capacity. The development of an artificial photosynthetic system to produce fuel, thus storing solar into chemical energy, is a major objective in current research. In this respect, water appears as an ideal substrate to produce molecular hydrogen, a carbon-neutral fuel. Photolysis of water (2H2O -> 2H2 + O2) involves two half-reactions: the oxidation of water to oxygen (2H2O -> O2 + 4H+ + 4e-, reaction 1) and the reduction of protons to dihydrogen (2H+ + 2e -> H2, reaction 2). Catalysts driven by visible light have been reported for the reductive half-reaction (reaction 2). However, the efficient catalytic oxidation of water to dioxygen (reaction 1) is a major challenge to be overcome to build sun-driven water-splitting devices. It is the objective of the present research proposal to design a biocatalyst capable of performing water oxidation efficiently, overcoming the limitations in terms of toxicity and cost of current (non-biological) catalysts and thus suitable for its use in the large scale.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Phase diagram of the Hubbard model by quantum Monte Carlo

Project leader: Sandro Sorella, SISSA, Condensed Matter Physics, Trieste, Italy
Collaborators: Frederico Becca, SISSA, Condensed Matter Physics, Trieste, Italy
Research field: Fundamental Physics

Abstract In this project we propose to study the phase diagram of the Hubbard model in two dimensional lattices by using an highly developed quantum Monte Carlo method. The main puprose is to use the large computer power available by the PRACE call to obtain reliable simulations of the model on lattice sizes containing several thousands electrons. For this purpose it is extremely important the use of massively parallel supercomputer and an efficient code, as TurboRVB is expected to be.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours Computer system: Jugene, Gauss/FZJ
Resource awarded: 100 000 core-hours

Project name: Linear-scaling Density Functional Theory of heterogeneous proteins with Conquest

Project leader: Antonio Sánchez Torralba, Spanish National Cancer Research Centre (CNIO), Structural Biology and Biocomputing, Madrid, Spain
Research field: Chemistry and Materials

Abstract Biological macromolecules, particularly proteins, present complex, heterogeneous environments that complicate the balancing of computational effort in massively parallel algorithms for calculating electronic structure properties using quantum mechanical methods. We intend to carry out Density Functional Theory (DFT) calculations of dihydrofolate reductase (DHFR), a small enzyme, using the linear-scaling code Conquest, but need to improve on the default partitioning of work among processors. The code has been designed to distribute atoms and functions on a grid, focusing on matrix multiplication kernels, so that calculating electronic charge densities and total energies of very large systems becomes feasible. Indeed, good scaling behaviour has been observed in a number of materials, including bulk silicon and germanium “huts” on silicon, in systems of thousands and even millions of atoms. However, we expect that performance can be improved for proteins in water. In this project, we plan to systematically assess scaling of DFT calculations of DHFR, with an eye on understanding the limitations and possible improvements of the default partitioning scheme, which uses a space-filling Hilbert curve to decide how atoms are distributed. We will measure the waiting times per iteration in the central loop of the algorithm, which optimises the total energy with respect to the density matrix. Typically, waiting times follow a sigmoidal distribution over processors, but it should be possible to reduce them globally by slight modifications of the distribution of atoms.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: High-fidelity simulations of multiscale-generated turbulence

Project leader: John Christos Vassilicos, Imperial College London, Aeronautics, London, UK
Collaborators: Sylvain Lazet, Imperial College London, Aeronautics, London, UK
Research field: Engineering and Energy

Abstract In this proposal for Preparatory Access we would like to investigate the scalability of our CFD code "Incompact3d" on CURIE. This code simulates complex fluid flows by solving the incompressible Navier-Stokes equations by means of Direct Numerical Simulation (DNS) and Large-Eddy Simulation (LES). This finite-difference code uses a powerful 2D domain decomposition which has lead to excellent parallel efficiency with up to 262144 computational core on the IBM Blue Gene/P JUGENE in Germany, which is based on low frequency quad-core processors. We would like to investigate the behaviour of the code using the high frequency eight-core processors of CURIE. From a scientific point of view, one recent application of Incompact3D is to complement experimental results from wind tunnel and water channel measurements for new flow concepts concerning turbulence generated by multiscale/fractal objects. This class of new flow concepts is offering possibilities for radically-new solutions useful in industrial mixers, silent air-brakes, heating/ventilation and combustion devices. To resolve the turbulent eddies associated with the smallest scale features on the multiscale objects, very high resolution flow simulations are required in order to understand the underlying physics.

Computer system: Curie, GENCI/CEA
Resource awarded: 50 000 core-hours

Project name: Evaluating scale-separation conditions for the larger eddies in turbulent flow

Project leader: Roel Verstappen, University of Groningen, Mathematics and Computer Science, Groningen, The Netherlands
Collaborators: Leo van Kampenhout, University of Groningen, Mathematics and Computer Science, Groningen, The Netherlands / Mohammad Younas, University of Groningen, Mathematics and Computer Science, Groningen, The Netherlands
Research field: Engineering and Energy

Abstract Finding a coarse-grained description is one of the main challenges to turbulence research. A most promising methodology for that is large-eddy simulation (LES). LES seeks to predict the dynamics of spatially filtered turbulent flows. The very essence is that the coarse-grained approximation contains only scales of size greater than the length of an user-chosen spatial filter. This property enables us to perform a LES when it is not feasible to compute the full, turbulent solution of the Navier-Stokes equations. In the present approach we try not to make any specific assumptions (about spectra, e.g.). Rather, we propose to continue the research that has been started at the 2010 Summer Program at the Center for Turbulence Research, Stanford University by addressing the following two questions: "when does the LES-model stop the production of smaller scales of motion from continuing at the filter scale?’’ and "when does the LES-model dissipate any subfilter perturbations at the natural rate?" This yields two approximate, scale separation conditions. We aim to evaluate and further develop these novel scale-separation conditions. To test these conditions numerically, data representing the interactions between scales of size larger than the filter length and residual scales is needed. Therefore full, turbulent solutions of the Navier-Stokes equations are to be computed numerically. This requires high-end computational resources. The goal of the present proposal is to test the scalability of our Navier-Stokes code for at least 10,000+ processing cores.The parallel performance has already been tested successfully at SARA Amsterdam (IBM Power 6) up to 2048 cores.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 100 000 core-hours

Type B – Code development and optimization by the applicant (without PRACE support)

Project name: PRACE 1IP WP7.3b supporting activity

Project leader: Jacques David, CEA, Nuclear Energy Division, Gif-sur-Yvette, France
Research field: Mathematics and Computer Science

Abstract Project gathering supporting activities for PRACE 1IP WP7.3b

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Electromagnetic field effects in massively parallel non-equilibrium dynamics

Abstract This project seeks to introduce external electromagnetic fields into massively-parallel molecular dynamics simulation, with the goal of future applications of nanoscale non-equilibrium MD simulation of electromagnetic field effects.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: HYBRID HERACLES

Project leader: Matthias Gonzalez, Université Paris Diderot, Gif-sur-Yvette, France
Collaborators: Pierre Kesterner, CEA, Maison de la Simulation, Gif-sur-Yvette, France / Edouard Audit, CEA, Maison de la Simulation, Gif-sur-Yvette, France
Research field: Astrophysics

Abstract This project aims at porting and optimizing the HERACLES code on the PRACE Curie computer. Heracles is a multi-purpose Eulerian radiation magneto-hydrodynamics code, based on Godunov scheme and domain decomposition. It is use mainly for astrophysics and laboratory astrophysics. Based on previous experience on French Tier-1 supercomputers (Jade at CINES, Babel at IDRIS), we believe that using and optimising a Hybrid MPI/OpenMP version of the code might be a good way to circumvent potential bottlenecks appearing when going to a very large number of cores (i.e. IO and communications).

From a scientific point of view, we plan in the near future to use HERACLES to study various aspect of interstellar physics. Mainly MHD turbulence and fragmentation of the interstellar medium, and also the formation of pillars around HII regions. Both these topics a closely connected to star formation process and are of prime importance to understand the onset and the regulation of the star formation in Galaxies.

From a technical point of view, the objective is to test and optimize on Curie an hybrid OpenMP/MPI version of the HERACLES code. This version was already tested on the BlueGene machine at IDRIS, but the fat nodes of the Curie machine, and the imminent arrival of massively multicore processor, offers new promizing perspectives for this programming model. More specifically, we expect that coarse graining obtained by using OpenMP and fewer MPI processes will greatly improve the communications and IO performance of the code when going to a very large number of cores.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: noFUDGE: Flow Unsteadiness computed by DG finite Elements

Project leader: Koen Hillewaert, Cenaero, Gosselies, Belgium
Collaborators: Corentin Carton de Wiart, Cenaero, Gosselies, Belgium
Research field: Engineering and Energy

Abstract Many industrially and scientifically relevant flows are dominated by separation and instabilities. To simulate these flows Large Eddy Simulation (LES) based approaches are indispensable. Currently no method is known to provide sufficient accuracy for this approach on unstructured meshes typically used for complex industrial geometries. The Discontinuous Galerkin method (DGM) is expected to bridge this gap. Direct Numerical Simulation (DNS) computations on selected benchmarks will be undertaken to assess the suitability of the method for the computation of unsteady turbulent wall-bounded flows.

A first benchmark consists of the DNS simulation of the flow around the SD0073 airfoil at Re = 60000 and angle of attack of 6°. Due to the low speed, the flow on the suction side consists of a laminar boundary layer separating at about 50% of the chord, and reattaching after transitioning to turbulence. This test case has recently been proposed as a benchmark for higher-order methods. The estimated computational cost to simulate 6 flow passages (ie. the time needed for a flow particle to pass the airfoil) is equal to 600k CPU hours on the BlueGene/P.

A second test case concerns the unsteady turbulent flow around a low pressure turbine blade. These turbines often operate at moderate Reynolds numbers (Re=20.000 - 200.000) and feature high deflection and flow acceleration. On the suction side, the flow is characterised by either very late transition or laminar separation followed by turbulent reattachment. Moreover, the flow at the thick trailing edge often features important vortex shedding. As weight reduction is a prime mover in the evolution of (turbine blades for) aero-engines, constructors try to increase the distance between successive blades. This leads to increased blade loading, in turn resulting in decreased flow stability and guidance. Moreover, modern designs will use a recirculation bubble captivated in the cavity formed by the blade pressure side to virtually provide a thick blade, required for a smooth flow acceleration.

The resulting large scale turbulent features make it very hard for RANS computations, currently used in industry, to correctly represent the flow. Hence more accurate approaches will need to be accepted and used in industry. Using DGM, it should be possible to compute this flow with a DNS approach for the lower ranges of the Reynolds number (Re < 80.000). The DNS approach should give the most accurate description of the flow since it does not rely on any model for the turbulence. The cost should be about 2-3 times that of the simulation of the SD0073 airfoil. These computations will be compared to measurements obtained at the von Karman Institute, and the results of Large Eddy Simulation (LES) simulations performed with a classical finite volume (FVM) code at Cenaero.

Computer system: JUGENE, Gauss/FZJ
Resource awarded: 2 000 000 core-hours (industrial pilot)

Project name: Numerical analysis of the magnetic properties of the polynuclear molecular clusters

Project leader: Grzegorz Kamieniarz, Adam Mickiewicz University, Poznan, Poland
Collaborators: Michal Antkowiak, Adam Mickiewicz University, Poznan, Poland
Research field: Chemistry and Materials

Abstract Molecular-based metallic clusters and chains behave like individual quantum nanomagnets, displaying quantum phenomena on macroscopic scale. In view of potential applications of such materials in magnetic storage devices or in envisaged quantum computer processor as well as in the low-temperature refrigerants, the accurate simulation of these complex objects becomes the key issue. The magneto-structural correlations, the role and mechanism of magnetic anisotropy and intrinsic quantum effects following from the geometrical frustration induced by the topological arrangement of spins or particular interactions count among the new challenges for computer simulations.

The simulations planned in the project address the quantum phenomenological models which are the most reliable theoretical representatives of the physical molecular-based nanomagnets investigated recently and their reliability from the fundamental microscopic point of view assessed by the well established first-principle electronic structure calculations. Exploiting a deterministic verified exact diagonalization technique, the model calculations will be performed without any uncontrolled approximations and will be numerically accurate.

The chromium-based rings which are outstanding materials for quantum information processing and for low-temperature cooling will be the principal objects of investigation. The real challenges appear for the molecules containing more than eight Cr S=3/2 ions and/or are doped by magnetic Ni or Cu ions, nevertheless the exact energy spectra, S-mixing, the total spin oscillations essential for quantum coherence and frustration phenomena important for magnetic refrigeration will be accomplished.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Scale-out of the Propag Electrocardiology Code on Petascale Architectures

Project leader: Rolf Krause, University of Lugano, Institute of Computational Science, Lugano, Switzerland
Collaborators: Mark Potse, Maastricht University, Maastricht, The Netherlands / Frits Prinzen, Maastricht University, Maastricht, The Netherlands / Angelo Auricchio, Fondazione Cardiocentro Ticino, Lugano, Switzerland / Thomas Dickopf, University of Lugano, Institute of Computational Science, Lugano, Switzerland / Dorian Krause, University of Lugano, Institute of Computational Science, Lugano, Switzerland
Research field: Medicine and Life Sciences

Abstract In order to contract effectively, the heart has a complex electrical triggering mechanism. Electrical activation is generated by the individual muscle cells and propagated to their neighbors through intercellular conducting channels. The propagation of the activation in the human heart can be mathematically modeled by a reaction-diffusion system. The reaction side of this system represents the dynamics of the active ion transport through the cell membranes, while the diffusion side represents the electric current flow between cells and in the environment outside the cells. Propag 5 is one of the most realistic mathematical heart models to date. The code has been parallelized with MPI and OpenMP and is optimized for massively parallel architectures. The goal of this project is to address the I/O and scaling bottlenecks of the code to allow for large-scale simulation of structural heart muscle disease (cardiomyopathy). Experiments have shown that for core counts of more than 4000, running in hybrid OpenMP-MPI mode can significantly speed up Propag. However, interfacing to external libraries in hybrid mode (for example for mesh partitioning) does not always give optimal results as most available software packages do not support hybrid execution.

To optimize the setup phase of Propag while maintaining a fully parallel workflow we will implement a separate mesh partitioning tool which can be run as a pre-processing step on an allocation that is optimal for this task. The distributed mesh is stored on disk in an optimized file format (matching the filesystem of the machine).

The mesh partitioner orders cells and nodes according to a space-filling curve to increase OpenMP scalability. To increase the output bandwidth of the code while maintaining the necessary compatibility with other post-processing tools in our workflow, we plan to split the output procedure in two phases. In a first phase, the compute processes perform contiguous independent writes to intermediate files. These are read by a set of writer processes (offline or as part of the simulation) and converted to the Cartesian output format.

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: Operating System Structures for Extreme-Scale Dynamic Resource Provisioning on Supercomputers

Project leader: Jan Stoess, Karlsruhe Institute of Technology, Karlsruhe, Germany
Collaborators: Noah Evans, Alcatel Lucent, Antwerpen, Belgium / Jeff Napper, Alcatel Lucent, Antwerpen, Belgium / James McKie, Alcatel Lucent, Antwerpen, Belgium / Eric van Hensbergen, IBM Austin Research Lab, Austin, USA / Ron Minnich, Sandia National Laboratories, Livermore, USA / Jonathan Appavoo, Boston University, Boston, USA / Dan Schatzberg, Boston University, Boston, USA / Charles Forsyth, Vita Nuova, York, UK / Jens Kehne, Karlsruhe Institute of Technology, Karlsruhe, Germany / Marius Hillenbrand, Karlsruhe Institute of Technology, Karlsruhe, Germany / Aranda Zamora, Karlsruhe Institute of Technology, Karlsruhe, Germany
Research field: Mathematics and Computer Science

Abstract The goal of our project is a new HPC operating system that enables highly dynamic, large scale provisioning of HPC resources while retaining high performance. Our vision is an operating system layer that enables standards-compatible applications to seamlessly operate on Blue Gene within their own domains; but which, on the other hand, also allows all HPC tasks to fully exploit the performance and scalability properties of Blue Gene’s processing and communication facilities.

Our new operating system will build up on top of our work on three existing open-source projects:

Kittyhawk, an effort to enable cloud computing on Blue Gene (kittyhawk.bu.edu)
L4, a micro-kernel with virtualization capabilities that, amongst other platforms, runs on Blue Gene compute nodes (l4ka.org) and
HARE, an effort to explore operating system and runtime services at scales of up to a million cores (www.fastos2.org/projects/hare); these will be sister projects with close involvement and collaborations; we already have gathered experience running all existing platforms on Blue Gene on two different sites in the US, at IBM TJ Watson Research Center and at Argonne National Labs.

Building on top of that work, we will construct a new, light-weight kernel layer that enables applications to utilize supercomputer resources dynamically at large scale, but in a performance and latency-sensitive way. Our experiences with Kittyhawk and HARE will serve as foundation for basic operating system primitives that scale up to millions of nodes. The basic system allows users to acquire and release supercomputer resources in a dynamic fashion, grouped into isolated computing domains, similar to infrastructure-level cloud-computing offerings. On top of that system layer, we will then place a hybrid library operating system configuration: we will use virtualization to construct a standards-compatible platform that eases installation, booting, bring-up, execution, and debugging of traditional supercomputer HPC tasks as well as general-purpose workloads. At the same time, we will exploit the potential of micro-kernel based systems to facilitating low-latency communication, to shortening the path to applications and lowering operating system noise, and to bringing guaranteed real-time performance and quality-of-service to applications, in order to allow running HPC tasks with predictable, almost-native performance.

Our project is a system-level endeavor, and there exists a broad range of applications we would like to explore. We plan to run three different types of benchmarks on our operating system layer: 1) traditional supercomputer applications such as large-scale, MPI-style simulations; 2) Performance and scalability demanding tasks from the commodity systems world such as big data processing algorithms 3) Latency-sensitive workloads such as online data streaming or transaction processing algorithms.

Specific milestones of our project are:
M1: Constructing a basic, scalable platform on Jugene to run custom kernels (1 month)
M2: Getting existing kernel environments up: Kittyhawk Linux, L4 microkernel + VMM + virtualized Linux, Plan 9, Osprey (2-3 months)
M3: Exploring, based on the existing kernel foundations, novel operating system primitives for dynamic yet latency- and throughput-oriented computing (>3 months)
M4: Demonstrating applications and benchmarks (1-2 months, pursued in parallel to M3)

Computer system: JUGENE, Gauss/FZJ
Resource awarded: 250 000 core-hours

Type C – Code development with support from experts from PRACE

Project name: Optimizing a 6D global Vlasov simulation of Earth’s magnetosphere

Project leader: Sebastian von Alfthan, Finnish Meteorological Institute, Earth Observation Unit, Helsinki, Finland
Collaborators: Arto Sandroos, Finnish Meteorological Institute, Earth Observation Unit, Helsinki, Finland / Ilja Honkonen, Finnish Meteorological Institute, Earth Observation Unit, Helsinki, Finland / Minna Palmroth, Finnish Meteorological Institute, Earth Observation Unit, Helsinki, Finland
Research field: Astrophysics

Abstract Space weather is a term used when the near Earth space environment threatens technological systems or humans. Most of the world depends on spacecraft that traverse the unpredictable space environment, where small dynamical events can cause systems failure. Space weather forecasts are currently being developed to model the electromagnetic plasma system within the near Earth space including the ionized upper atmosphere (ionosphere), the space governed by the Earth’s magnetic field (magnetosphere), and beyond (solar wind) FMI develops a Vlasov-hybrid simulation (Gux) where electrons are fluid and ions are distribution functions, enabling the description multi-component plasmas without noise and in scales unreachable by existing techniques such as magnetohydrodynamic (MHD) simulations. A large-scale Vlasov-hybrid simulation is highly challenging, as in every grid cell of the ordinary space resides a velocity space, making the simulation 6-dimensional (6D). This sets demanding quality criteria for the solver, because the simulation needs to be executed in 10^6 spatial grid cells for 10^6 time steps indicating peta-scale computations.

In this project the remaining bottlenecks of Gux are to be removed so that large-scale production simulations may be performed. This will enable global Vlasov simulations of Earth’s magnetosphere for the first time, opening up new avenues of research. As Gux will be distributed as open source software the code will be of great benefit to a wider space research community.

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: Direct numerical simulation and turbulence modeling for fluid-structure interaction in aerodynamics

Project leader: Marianna Braza, CNRS-Institut de Mecanique des Fluides de Toulouse, IMFT, Toulouse, France
Collaborators: Gilles Harran, INPT, Toulouse, France / Yannick Hoarau, INPT, Toulouse, France / Fernando Grossi, CNRS-Institut de Mecanique des Fluides de Toulouse, IMFT, Toulouse, France / Marcel Thibaud, CNRS-Institut de Mecanique des Fluides de Toulouse, IMFT, Toulouse, France / Marc Gual Skopek, CNRS-Institut de Mécanique des Fluides de Toulouse, IMFT, Toulouse, France / Sylvie Saintlos, UPS, IMFT, Toulouse, France
Research field: Engineering and Energy

Abstract The present proposal focuses on code development with support of experts concerning multi-level optimisation of an efficient and robust CFD solver in aeronautics: the code NSMB, Navier-Stokes MultiBlock. This solver, used during the decade of 1990’s and 2000’s by principal european research institutes and european industry in aeronautics, has been significantly upgraded in the last ten years in terms of implementing new FLOW PHYSICS MODELLING, able to capture complex phenomena arising in aeronautics and aeroelasticity, by the collaborative efforts of CFS-Engineering, IMFT, IMF-Strasbourg, EPFL. A specific advanced version of the code containing Turbulence Modelling Approached developed by IMFT will be the object of the present optimisation, thanks to the competences of experts, especially coming from french supercomputing centres, although there is no limitation concerning a general panel of experts. The present project aims at improving the performances of this version of the code in terms of MPI and MPI/OPEN-MP platforms, in order to use efficiently the largest number of processors currently available. This will allow to investigate NEW PHYSICS in the turbulence processes arising in aeronautics, thanks to an increase by orders of magnitude the number of degrees of freedom and by refining the time-steps. Fully implicit numerical schemes will be privileged, dealing properly with the non-linearity of Turbulence in the flow equations. Furthermore, new methods of turbulence modelling and of coupling with deformable solid structures will be implemented in an optimal basis of MPI architecture, to increase the predictive capability of the solver comparing to other currently existing methods, concerning the unsteady aerodynamic loads, the nuisance instabilities as buffet and flutter, as well as strong flow detachments. These constitute currently the most stiff problems in aeronautics, fluid-structure interaction and flow control.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Improvement of the multi-level communication scheme for better scalability of the code BigDFT

Project leader: Luigi Genovese, Commissarial à l’Energie Atomique, INAC, Grenoble, France
Collaborators: Thierry Deutsch, Commissarial à l’Energie Atomique, INAC, Grenoble, France
Research field: Chemistry and Materials

Abstract High Performance Computing performances are of paramount importance in the domain of electronic structure calculations. The high complexity of the formalism increase the computational workload such that an efficient usage of computing resources is mandatory for effective progresses in the scientific outcomes. In the context of Density Functional Theory (DFT) codes, distributed computations are in general parallel-intensive runs. The intrinsic nature of the approach, where electrons are described by wavefunctions, requires that data have to be communicated for evaluating scalar product and orthogonality conditions. For codes using systematic basis sets approaches, where the number of degrees of freedom is large, the performances of the communication scheme have at least the same importance of the computational behaviour of the code. In this proposal, we will study the behaviour of different parallelisation strategies of the BigDFT code, based on Daubechies Wavelets. The latter is a systematic, real-space based basis set, which account for optimal precision and flexibility. In addition, the formalism of Daubechies Wavelets is optimal for building codes with optimal efficiency in massive and massively parallel architectures, which account for multi-level parallelisation. For this reason, multi-level MPI-OpenMP parallelisation has been implemented in BigDFT in the recent years. The code is also able to run efficiently in Hybrid CPU/GPU architectures, thanks to its formalism and data distribution. Moreover, Wavelets are also interesting for DFT due to their compatibility with Order N methods, and a O(N) version of the BigDFT code is at present under preparation.

This version will be able to benefit from the existing multi-level parallelisation layers. To this aim, the efficiency of different communication schemes of the BigDFT code will be investigated, in particular with respect to the multi-level parallelisation features. Simple collective, blocking communication schemes will be tested against more elaborated non-blocking, point-to-point parallelisation algorithms, both with traditional and O(N)-like data repartitions. These investigations should help in finding the correct dimensioning of the approach with respect to the particular system under investigation. This fact should improve the know-how on parallel DFT approaches, such as to improve performances of other codes with similar features.

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Quantum Monte Carlo methods for biological systems

Project leader: Leonardo Guidoni, Università degli Studi de L’Aquila, Dipartimento di Chimica, Ingegneria Chimica e Materiali, L’Aquila, Italy
Collaborators: Sandro Sorella, Sissa, Trieste, Italy / Andrea Zen, La Sapienza, Università di Roma, Roma, Italy / Emanuele Coccia, Università degli Studi de L’Aquila, Dipartimento di Chimica, Ingegneria Chimica e Materiali, L’Aquila, Italy
Research field: Chemistry and Materials

Abstract Quantum Monte Carlo (QMC) methods are a promising technique for the study of the electronic structure of correlated molecular systems. QMC algorithms are highly parallel in nature and thanks also to the relatively small memory requirements also for large systems, they have good performances and scalability for highly parallel computers. Recent implementations in the TurboRVB code were able to calculate in an efficient and scalable way the ionic forces, providing us with the possibility to perform full geometry optimization of molecules at the Variational Monte Carlo level. Three different run types are usually necessary for QMC calculations: Variational Monte Carlo (VMC), optimization of the variational wavefunction and Lattice Regularized Diffusion Monte Carlo (LRDMC). We reported for all 3 run types good scalability up to 4000 cores on the CINECA BlueGene machine (BGP). Anyway, a linear algebra part of the code, which is currently not parallelised, is a bottleneck to further scaling. The technical of the present project is to demonstrate the scalability of the TurboRVB code for a series of systems having different properties in terms of number of electrons, number of variational parameters and size of the basis set. The scientific goal will be the geometry optimization of a protein chromophore (about 100 atoms and 500 electrons) within the field of the protein matrix. The achievement of the targets of the present proposal will produce an accurate electronic structure tool with almost unique scaling capabilities among all other packages.

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: Turbulent convection and dynamos in spherical wedges

Project leader: Petri Käpylä, University of Helsinki, Helsinki, Finland
Collaborators: Maarit Mantere, University of Helsinki, Helsinki, Finland / Axel Brandenburg, Nordita, Stockholm, Sweden / Piyali Chatterjee, Nordita, Stockholm, Sweden / Gustavo Guerrero, Nordita, Stockholm, Sweden / Dhrubaditya Mitra, Nordita, Stockholm, Sweden
Research field: Astrophysics

Abstract The Sun exhibits magnetic activity at various spatial and temporal scales. The best known example is the 11-year sunspot cycle which is related to the 22-year periodicity of the Sun’s magnetic field. The sunspots, and thus solar magnetic activity, have some robust systematic features: in the beginning of the cycle sunspots appear at latitudes around 40 degrees. As the cycle progresses these belts of activity move towards the equator. The sign of the magnetic field changes from one cycle to the next and the large-scale field remains approximately antisymmetric with respect to the equator. This cycle has been studied using direct observations for four centuries. Furthermore, proxy data from tree rings and Greenland ice cores has revealed that the cycle has persisted through millennia. The period and amplitude of activity change from cycle to cycle and there are even periods of several decades in the modern era when the activity has been very low. Since it is unlikely that the primordial field of the hydrogen gas that formed the Sun billions of years ago could have survived to the present day, the solar magnetic field is considered to be continuously replenished by some dynamo mechanism.

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Visualization of output from large-scale brain simulation Project leader: Anders Lansner, KTH, Computational Biology, Stockholm, Sweden
Collaborators: Simon Benjaminsson, KTH, Computational Biology, Stockholm, Sweden / David Silverstein, KTH, Computational Biology, Stockholm, Sweden
Research field: Medicine and Life Sciences

Abstract The proposed neural visualization application should work as a tool used to better understand activity resulting from large-scale neural simulations and to generate animations of this. It could possibly also be used in real-time visualization, including zooming into specific parts of a network.

We are interested in visualizing changing properties of simulated neural systems and generated synthetic output: A: Properties could be neuron membrane potentials, spiking-state of neurons, synaptic strengths between neurons or neural assemblies. More than one property may be of interest to visualize, either separately or simultaneously. Coordinates of neural elements could be generated from the simulation and may in some cases be changing over time, e.g. in fiber growth. Neuron types could be visualized with different geometries, e.g. pyramidal cells could be pyramids.

B: Synthetic output could be VSD, LFP, MEG or other output from a simulation that can be related to experimental data. These are not necessarily located at a specific neural location but could be seen as something measured from an area or collection of neurons. It could also be an output based on a specific measure of a simulated population, e.g. a synchrony measure of population activities or visualizing waves of activities.

A user should be able to present the information in a specified geometrical way or mapped in a predefined geometry as specified from a model of a brain or a brain area, e.g. mapping simulated activity to a whole-brain wireframe model. In total, this gives three different ways to visualize data: Fixed geometry as defined by simulation (neurons are not moving), changing geometry (fiber growth) or mapped activity to an externally defined geometry (whole-brain wireframe model). We can use it with our own developed simulators, ANSCore and BrainCore and the widely used NEURON simulator. This can enable prototyping and fast testing of visualization techniques and data storage code for our neural models as well as validating the models themselves. If the tool and input/output-specifications are general and prove to be useful, this will be of interest to the larger computational neuroscience community.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: Petascale Astrophysical Simulations of Accretion Processes with PLUTO

Project leader: Andrea Mignone, Univiversity of Torino, Dipartimento di Fisica Generale, Torino, Italy
Collaborators: Gianluigi Bodo, Osservatorio Astronomico di Torino, Pino Torinese, Italy / Petros Tzeferacos, Univiversity of Torino, Dipartimento di Fisica Generale, Torino, Italy
Research field: Astrophysics

Abstract During the past decades, our comprehension of the rich and complex phenomenology associated with astrophysical plasmas has largely benefitted from large-scale numerical simulations. Nevertheless, much of what is currently known crucially depends on the computational grid sizes affordable in present numerical simulations and thus several challenges are still open and several questions have not been properly addressed. Among these, a large number of unsettled issues directly concerns the process of angular momentum transport and the physics of magnetically-driven turbulent accretion disks. Present days simulations rely on the so-called shearing-box approximation in which instead of simulating the entire disk one considers only a small computational domain with radial extent much smaller than its orbital radius. Although such an approximation permits a huge computational saving and allows simulations to be carried out at much higher effective Reynolds numbers, recent results suggest that the shearing-box approximation in its simplest form is probably unable to capture the physics of the Magneto Rotational Instability (MRI) in disks. As a consequence, future simulations will probably give up the local approach in favour of increasingly more costly global disk simulations. In this sense, the PLUTO code for astrophysical fluid dynamics, actively developed by our group and considered among the top-rated state-of-the-art current numerical tools, can deliver to the scientific community a suitable parallel computational framework for tackling large-scale fluid dynamical problems demanding intensive resources.

In this perspective we aim at improving the present parallel capabilities of the code by overcoming the present potential bottlenecks targeting Petascale simulations. To successfully accomplish this target a number of improvements in the parallelization strategies as well as in the I/O performance are mandatory and reachable through a close interplay and coordinated development with high performance computing experts.

Computer system: Jugene, Gauss/FZJ
Resource awarded: 250 000 core-hours

Project name: High resolution ocean simulations with NEMO modeling system

Project leader: Jean-Marc Molines, CNRS, LEGI, Grenoble, France
Collaborators: Bernard Barnier, CNRS, LEGI, Grenoble, France / Julien le Sommer, CNRS, LEGI, Grenoble, France / Albanne Lecointre, CNRS, LEGI, Grenoble, France / Anne-Marie Treguier, CNRS, Plouzané, France / Claude Talandier, CNRS, LEGI, Grenoble, France / Gurvan Madec, CNRS, Paris, France / Rachid Benshila, CNRS, Paris, France / Claus Böning, IfM-Geomar, Kiel, Germany / Arne Biastoch, IfM-Geomar, Kiel, Germany / Marcus Scheinert, IfM-Geomar, Kiel, Germany / James Orr, CEA, Gif-sur-Yvette, France / Andrew Coward, NERC, Southampton, UK / Adrian New, NERC, Southampton, UK / Marina Levy, CNRS, Paris, France / Christian Ethé, CNRS, Paris, France
Research field: Earth Sciences and Environment

Abstract This project aims at preparing the high-resolution ocean/sea-ice realistic modelling environment implemented by the European DRAKKAR consortium for use on PRACE Tier-0 computers. DRAKKAR participating Teams jointly develop and share this modelling environment to address a wide range of scientific questions investigating multiple-scale interactions in the world ocean. Each team relies on the achievements of DRAKKAR to have available for its research the most efficient and up-to-date ocean models and related modelling tools. Two original realistic (by their representation of coastlines, topography, forcing, …) model configurations, based on the NEMO modelling framework, are considered in this proposal. They are designed to make possible the study of the role of multiple-scale interactions in the ocean variability, in the ocean carbon cycle and in marine ecosystems changes. These 2 configurations, which are among the flagship configurations of the DRAKKAR consortium, are:

1) ORCA12, a 1/12 degree realistic global configuration, with 75 vertical levels (10^9 grid points). We already use this ORCA12 configuration on Tier-1 computer (JADE) at CINES (using 2032 cores), but with lesser vertical resolution (46 levels). Porting ORCA12 to Tier-0 machine (and targeting the use of 5000 cores) will allow an increase of the vertical resolution to 75 levels (more dynamically consistent with the high vertical resolution used) and will give the possibility of longer runs (which are necessary to reach a state of quasi-equilibrium), and of coordinated (ensemble) sensitivity runs.

2) PERIANT8_BIO, a realistic regional 1/8 degree circumpolar Southern Ocean configuration including biogeochemistry (carbon cycle modelling) and 46 vertical levels (10^8 grid points). For PERIANT8, the ocean-only configuration is also running on JADE at CINES (using 520 cores). Adding the coupling to biogeochemistry (about 23 passive tracers) will considerably increase the required memory and the computational burden. On CURIE, we target the use of about 4200 cores for this configuration. In this proposal, we will mimic the use of the biogeochemical model (PISCES) by adding 23 independent passive tracers that will be advected and diffused in the ocean by the TOP component (i.e. the passive tracers module) of NEMO. This strategy avoids the problem of initialization and parameter tuning for the biogeochemical model, which by itself is not a computational issue.

This preparatory access project will serve many purposes: i) the porting of the existing configurations from JADE to CURIE ( code, scripts), ii) the fine tuning (optimization) and the assessment of the performances on CURIE and iii) validation of the I/O strategy used so far and its possible evolution for biogeochemistry. Finally (iv) it will set the bases of the next modelling environment that the DRAKKAR consortium will exploit with PRACE in the future, and what new scientific questions it will permit to address. In particular, this proposal is preparatory to a proposal that will be submitted to a regular PRACE Call in fall, with the objective to study the role of multiple-scale physico-biogeochemical interactions in the South Indian Ocean. Higher resolution (1/24°) is aimed at for this application (BIOCOSM project).

Computer system: Curie, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: Self organization, pattern formation and morphological instabilities in suspensions of microswimmers

Project leader: Ignacio Pagonabarraga, University of Barcelona, Barcelona, Spain
Collaborators: Giovanni Giupponi, University of Barcelona, Barcelona, Spain / Francisco Alarcon, University of Barcelona, Barcelona, Spain / Andrea Scagliarini, University of Barcelona, Barcelona, Spain / Ricard Matas, University of Barcelona, Barcelona, Spain / Silvio Rene Morales Suarez, University of Barcelona, Barcelona, Spain
Research field: Fundamental Physics

Abstract Suspensions of microswimmers (either model microorganisms or microrobots) constitute materials which are intrinsically out of equilibrium. The collective, dynamical behavior, of these systems is far from well understood. In particular, the internal motion and the coupling of these particles to the fluid environment promote the self-assembly of supra-particle structures which lead to patterns and new type of aggregates.

Appropriate computational schemes, which focus on the dynamic coupling between swimmers and solvent fluid dynamics are required to reach the long time and spatial scales on which they develop while accounting for the basic physical mechanisms responsible for the emergence of such patterns. We will analyze the possibility of exploiting a lattice Boltzmann platform developed for complex fluids, called Ludwig, on large supercomputing facilities. We will benchmark the code and assess critically how particle data limits the code performance.

Facing these difficulties and improving the code correspondingly will allow us to study the possibility that the hydrodynamic coupling provides a new mechanism for a flocking transition and in general, a means to promote inhomogeneous patterns. We will subsequently analyze how different kinds of particle interactions compete to induce anisotropic patterns and will pay special attention to the possibility to generate active polymers out of such swimmers of microrobots.

Computer system: JUGENE, Gauss/FZJ
Resource awarded: 250 000 core-hours

Computer system: CURIE, GENCI/CEA
Resource awarded: 200 000 core-hours

Project name: New algorithms in Octopus for Petaflop computing

Project leader: Joseba Alberdi Rodriguez, Universidad del Pais Vasco/Euskal Herriko Unibertsitatea, Donostia-San Sebastian, Spain
Research field: Chemistry and Materials

Abstract The main objective of this project is enhance the parallel capabilities of the first-principles simulation code Octopus (http://www.tddft.org/programs/octopus) to reach a highly efficient massive parallel version to meet the petaflop (and more) challenges. We have identified some important bottlenecks that has hampered the massive performance of the code and those should be overcame during the present project with the help of PRACE. Octopus is used to study the properties of the excited states of large biological molecules and nanostructures of complex solids, using first principle simulations. For instance, Octopus is presently used to understand the mechanisms of absorption of the light and the energy transfer in photosynthetic complexes, including both electronic and ionic dynamics triggered by the absorption of light (“light-induced photophysical processes in biocomplexes”). The range of applicability of the code suite spans nano, bio and materials science with a clear multidisciplinary development contribution from physics, chemistry, biology and computer science.

The theoretical framework Octopus relies on the time-dependent density functional theory (TDDFT) formulation of quantum mechanics. The main quantities to represent are three dimensional functions: the density and the single particle orbitals. The single particle orbitals are evolved following the time dependent Kohn-Sham equations taking as initial condition in most cases the solution of the ground state density functional theory problem, also obtained by Octopus. In the code the functions are represented in a real space grid, and differential operators are approximated by high-order finite differences.

We have developed a parallel version of Octopus that, for instance, has been used to study the behavior of systems around 2,600 atoms. But, to deal with bigger systems in a bounded time we need to highly improve the current version of the parallel code.

Therefore, our objective is to exploit the capabilities of the new High Performance Computing (HPC) facilities to address, for the first time, the excited states properties of large biological molecules by first principle simulations.

Computer system: JUGENE, Gauss/FZJ
Resource awarded: 250 000 core-hours