As the volume of data generated by large-scale experiments continues to grow, the need for rapid data analysis capabilities is becoming increasingly critical to new discoveries.
At the U.S. Department of Energy’s (DOE) Argonne National Laboratory, the co-location of the Argonne Leadership Computing Facility (ALCF) and the Advanced Photon Source (APS) provides an ideal proving ground for developing and testing methods to closely integrate supercomputers and experiments for near-real-time data analysis.
For over a decade, the ALCF and APS, both DOE Office of Science user facilities, have been collaborating to build the infrastructure for integrated ALCF-APS research, including work to develop workflow management tools and enable secure access to on-demand computing. In 2023, the team deployed a fully automated pipeline that uses ALCF resources to rapidly process data obtained from the X-ray experiments at the APS.
To demonstrate the capabilities of the pipeline, Argonne researchers carried out a study focused on a technique called Laue microdiffraction, which is employed at the APS and other light sources to analyze materials with crystalline structures. The team used the ALCF’s Polaris supercomputer to reconstruct data obtained from an APS experiment, returning reconstructed scans to the APS within 15 minutes of them being sent to the ALCF.
The researchers detailed their efforts in their article “Demonstrating Cross-Facility Data Processing at Scale With Laue Microdiffraction,” which was recognized with the Best Paper Award at the 5th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP 2023) at the Supercomputing 2023 (SC23) conference in November. Led by APS software engineer Michael Prince, the team includes Doğa Gürsoy, Dina Sheyfer, Ryan Chard, Benoit Côtê, Hannah Paraga, Barbara Frosik, Jon Tischler and Nicholas Schwarz.
Read more on Argonne website
Image: Argonne researchers Hannah Parraga (far right), Michael Prince (second from right) and Nicholas Schwarz (third from right) lead a demo at the SC23 conference on using integrated computing resources to accelerate discoveries at the Advanced Photon Source.
Credit: Argonne National Laboratory