Argonne - CAMERA Collaboration on Software for Tomographic Reconstruction









Since its invention a few decades ago, tomography has found continually increasing application across many areas of science. The Advanced Photon Source (APS) at Argonne National Laboratory has more than 10 beamlines capable of performing tomography at least part of the time; the Advanced Light Source (ALS) at Lawrence Berkeley National Laboratory, has 7 beamlines that can carry out tomography experiments. These include soft and hard X-ray beamlines, as well as Infrared beamlines.


The algorithms used to convert raw data into useable 3D images are complex and computationally intensive. In 2013 the APS started the TomoPy project [1], aimed at providing a collaborative and community-driven framework for the analysis of synchrotron tomographic data in an open source environment which now has active developers around the world.

The Goal:


In 2015, the Center for Advanced Mathematics for Energy Research Applications (CAMERA) initiated collaboration with Argonne. The shared goals have been to bringing closer interaction between Argonne and CAMERA algorithm developers, to provide users with an extensive set of tools for tomographic reconstruction, and accelerate the adoption of these tools across DOE facilities.



Current Progress:


Working together, CAMERA and Argonne have proceeded in three steps.


The first step was to bring TomoPy to the ALS, and this process started during a weeklong collaboration meeting in Berkeley in February 2016. At that meeting,  Doga Gursoy, the lead developer of TomoPy, from the APS, was a highlight leading participant. Key other participants included Daniel Pelt (formally a visiting graduate student at the APS and now a CAMERA postdoctoral fellow) who developed the Python ASTRA package for high-performance GPU primitives for 2D and 3D tomography [2], as well as Luis Barroso-Luque, Singanallur Venkatakrishnan, and Stefano Marchesini, who are members of CAMERA on the tomography team.


During this time, TomoPy was deployed at NERSC, and new beamline computers were set up specifically to run TomoPy.  Code was added to the Exchange package (which handles TomoPy file format and i/o functions) that allows it to handle data collected at one ALS beamline. Additionally, TomoPy was installed to ALS GPU clusters to take advantage of fast execution of the TomoPy/ASTRA GPU supported functions.


The next step was customize and extend some of the TomoPy functionality for ALS beamlines, and include new auto-center of rotation detection  algorithms and new ring removal algorithms. Together with the TomoPy team, these new functionalities have been added to TomoPy to benefit all users of TomoPy, including those at other synchrotron facilities.


The third step was to incorporate these tomographic algorithms into Xi-CAM environment, built by CAMERA, which aims to provide an easy-to-use and natural framework for algorithms and tools across many DOE facilities. Xi-CAM was originally developed for small angle scattering experiments at the ALS, and aims to provide a community driven platform for multimodal analysis in synchrotron science, and is designed to be highly extensible and portable across platforms, beamlines, and facilities. The robust plug-in infrastructure of Xi-CAM encourages continuing development to add further functionality.


Coupling TomoPy algorithms, CAMERA algorithmic extensions, and Xi-CAM’s environment together now allows users to run preview reconstructions, providing fast visual feedback to users making it easy to compare reconstructions using different set of parameters. It provides tools for quick intermediate data processing step inspections and for exploring algorithm behavior for ranges of parameters. It also provides a 3D rendering viewer with clipping planes and easy color-map adjustment for viewing the 3D result.



This coupled environment is now being used at ALS Beamline 8.3.2 of the ALS, and, in recent weeks has been extended three different ALS user groups. While still in beta mode, it is already becoming a valuable tool, and soon set to be made available to the large community of TomoPy users.

The Future: 


















The Team: 

On the Argonne side, the effort includes work of Doga Gursoy.  CAMERA scientists tasked to work with Argonne include Dula Parkinson, Daan Pelt, Ron Pandolfi, Luis Barroso-Luque, Singanallur Venkatakrishnan, and Stefano Marchesini.

[1] Gürsoy D, De Carlo F, Xiao X, and Jacobsen C, “TomoPy: a framework for the analysis of synchrotron tomographic data”, Journal of Synchrotron Radiation, 21(5): 1188–1193, 2014.

[2] Pelt D, Gürsoy D, Palenstijn WJ, Sijbers J, De Carlo F, and Batenburg KJ, “Integration of TomoPy and the ASTRA toolbox for advanced processing and reconstruction of tomographic synchrotron data”, Journal of Synchrotron Radiation, 23(3): 842–849, 2016.

[3] Bicer T, Gürsoy D, Kettimuthu R, De Carlo F, Foster I, “Optimization of tomographic reconstruction workflows on geographically distributed resources”, Journal of Synchrotron Radiation, 23(4): 997-1005, 2016.

Screenshot of CAMERA's Xi-CAM interface for tomographic reconstruction, built using the customized algorithms of the APS's TomoPy toolbox. 

CAMERA and APS scientists are working together to considerable extend these capabilities for deployment at the APS, at the ALS, and at other facilities. One of the most important future challenges will be making it easy and automatic for users to match up the location of their data with appropriate computing resources. For certain reconstruction algorithms or certain data sizes, a GPU cluster or a supercomputer is the best place, but for others a laptop will suffice (see for example [3], which examines optimization of tomographic reconstruction workflows utilizing geographically distributed resources.  Xi-CAM now enables users to select the location of the data and the location of their desired compute resources, seamlessly handing details of the data transfer, etc.: the goal is to provide a transparent  interface and workflow which naturally distributed resource optimization.

Xi-CAM environment for remote workflow execution for tomography