Swiss National Supercomputer Center (CSCS) - CAMERA Collaboration:
Algorithms, Workflow, and Supercomputers for Big Data Processing and Analysis
A joint project between the Swiss Supercomputer Center (CSCS) in Lugano, ETH Zurich, The Adolphe Merkle Insistute, The Technical Universityof Munich (TUM), the Advanced Light Source and CAMERA is aimed at developing the infrastructure needed to meet the challenges of the new generation of synchtrotron light sources.
Traditionally, users collect data during their assigned and limited "beamtime" and then spend many months analyzing the results. With the huge increase in data volume, this is no longer possible. As a consequence, typically only a fraction of the collected data is fully analyzed and used. However, synchrotron beamtime is an expensive resource, and scheduling beam time and appropriate resources is challenging. Second, advanced algorithms and software are needed to
deal with bursts of information when large amounts of data are created in short period of time. Third, it is impractical to collect, set up, process, and analyze data in a manuall-driven step by step process. Instead, what is needed are
automatized data analysis pipelines that can help the user fine-tune an experimental run during beam-time.
The goal is to addressing the synchrotron data deluge by developing an automatized data analysis tool that aims to solve the current mismatch in Big Data generation at synchrotron national facilities and the ability to analyze these data. As a testbed environment, the above partners are building a protoptype synchrotron open framework for interactive analysis and automation, equipped with the tools necessary to move, archive, visualize, reduce, analyse, modelling and simulating the synchrotron data in near real time by taking advantage of the cutting edge super-computing facilities at CSCS in Switzerland.
Taking advantage of a jointly sponsored TUM/ALS/CAMERA summer school on grazing incidence small angle scattering in Munich, students designed and excuted experiments from laptops in Munich, running x-ray scattering experiments at the ALS, and remotely analyzing the data at CSCS using CAMERA analysis algorithms. Synchrotron experiments were carried out using robotics without human intervention on a large amount of samples from completely uncorrelated and diversified research projects from 20 international research groups.
The data was then streamed to Munich, and then simulated and processed using CAMERA's High Performance Grazing Incidence Small Angle X-Ray Scattering (HipGISAXS) algorithm running remotely on dedicated nodes on the Piz Daint supercomputer at CSCS in Switzerland seamlessly with the push of a single button through the Xi-CAM interface.
The next steps include extending this computational and workflow framework to others modalities, such as tomography. and extending the computational capabilities from synchrotron to neutrons.
On the ETH Zurich/CSCS, the effort includes work of Thomas Schulthess and Claudio Gheller (ETH Zurich/CSCS), .Alessandro Sepe (Adolphe Merkle), Eva Herzig (TU-Munich), and CAMERA scientists Alexander Hexemer, Ron Padnolfi, Dinesh Kumar and Hari Krishnan.