data analysis – Lightsources.org

PANOSC consortium signs Memorandum of Understanding with the European Open Science Cloud

2025/12/182025/12/19

The Director General of the ESRF, Jean Daillant, representing the 11 partners of the Photon and Neutron Open Science Cloud (PaNOSC) , has signed the EOSC Federation Memorandum of Understanding with the EOSC Association today, in presence of ILL representatives.

The European Open Science Cloud (EOSC) Federation aims to create a seamless system where researchers across the continent can easily find, access, and use data and services to drive innovation. By linking hundreds of data repositories and tools, EOSC will make it simpler for scientists to find, share, analyze, and reuse FAIR (Findable, Accessible, Interoperable, and Reusable) research outputs.

PaNOSC as the EOSC Node of the Photon and Neutron Open Science Cluster (PaNOSC), which includes all synchrotron and neutron sources in Europe, aims to connect the Photon and Neutron European research infrastructures to EOSC. Currently 11 Photon and Neutron Research Institutes have committed to providing data and services to the EOSC Federation through the PaNOSC EOSC Node – these are ESRF (as host institute), ALBA, DESY, ELETTRA, ESS, European XFEL, HZDR, ILL, MAX IV Laboratory, PSI, and SOLEIL.

Read more on the ESRF website

Image: The signature took place in December at the ESRF. The DG of the ESRF, Jean Daillant, signed on behalf of PaNOSC. Mark Johnson (first left) represented the ILL in the event.

Credit: Alexia Daurat

Argonne rapid cross-facility data processing

2024/03/122024/03/26

As the volume of data generated by large-scale experiments continues to grow, the need for rapid data analysis capabilities is becoming increasingly critical to new discoveries.

At the U.S. Department of Energy’s (DOE) Argonne National Laboratory, the co-location of the Argonne Leadership Computing Facility (ALCF) and the Advanced Photon Source (APS) provides an ideal proving ground for developing and testing methods to closely integrate supercomputers and experiments for near-real-time data analysis.

For over a decade, the ALCF and APS, both DOE Office of Science user facilities, have been collaborating to build the infrastructure for integrated ALCF-APS research, including work to develop workflow management tools and enable secure access to on-demand computing. In 2023, the team deployed a fully automated pipeline that uses ALCF resources to rapidly process data obtained from the X-ray experiments at the APS.

To demonstrate the capabilities of the pipeline, Argonne researchers carried out a study focused on a technique called Laue microdiffraction, which is employed at the APS and other light sources to analyze materials with crystalline structures. The team used the ALCF’s Polaris supercomputer to reconstruct data obtained from an APS experiment, returning reconstructed scans to the APS within 15 minutes of them being sent to the ALCF.

The researchers detailed their efforts in their article “Demonstrating Cross-Facility Data Processing at Scale With Laue Microdiffraction,” which was recognized with the Best Paper Award at the 5th Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP 2023) at the Supercomputing 2023 (SC23) conference in November. Led by APS software engineer Michael Prince, the team includes Doğa Gürsoy, Dina Sheyfer, Ryan Chard, Benoit Côtê, Hannah Paraga, Barbara Frosik, Jon Tischler and Nicholas Schwarz.

Read more on Argonne website

Image: Argonne researchers Hannah Parraga (far right), Michael Prince (second from right) and Nicholas Schwarz (third from right) lead a demo at the SC23 conference on using integrated computing resources to accelerate discoveries at the Advanced Photon Source.

Credit: Argonne National Laboratory

ExPaNDS webinar series to showcase achievements and look to the future

2023/01/262023/01/26

We’re pleased to announce our upcoming topic-based webinars which will take place during the coming month before the end of our grant in February 2023. The webinar topics have been selected with the help of our work package leaders and some of the highlighted use cases taken directly from the PaN community throughout our grant.

The series will provide a great opportunity to showcase some of the outcomes of our grant to the PaN facility user communities. We will present some key findings from the recently conducted data consultation, which was sent to over 14,000 PaN facility users.

The ongoing work of ExPaNDS has been very important to the PaN community and we have invited senior community figures to discuss the future needs and requirements for their respective discipline or technique to keep the momentum going beyond the grant.

We will have flash talks from our work packages with focus being on FAIR, data catalogue services, data analysis and an overview of the PaN training platform.

Read more on the ExPaNDS website

Image: Chairman of the DESY Board of Directors – Professor Dr Helmut Dosch

New software based on Artificial Intelligence helps to interpret complex data

2022/12/202022/12/21

Experimental data is often not only highly dimensional, but also noisy and full of artefacts. This makes it difficult to interpret the data. Now a team at HZB has designed software that uses self-learning neural networks to compress the data in a smart way and reconstruct a low-noise version in the next step. This enables to recognise correlations that would otherwise not be discernible. The software has now been successfully used in photon diagnostics at the FLASH free electron laser at DESY. But it is suitable for very different applications in science.

More is not always better, but sometimes a problem. With highly complex data, which have many dimensions due to their numerous parameters, correlations are often no longer recognisable. Especially since experimentally obtained data are additionally disturbed and noisy due to influences that cannot be controlled.

Helping humans to interpret the data

Now, new software based on artificial intelligence methods can help: It is a special class of neural networks (NN) that experts call “disentangled variational autoencoder network (β-VAE)”. Put simply, the first NN takes care of compressing the data, while the second NN subsequently reconstructs the data. “In the process, the two NNs are trained so that the compressed form can be interpreted by humans,” explains Dr Gregor Hartmann. The physicist and data scientist supervises the Joint Lab on Artificial Intelligence Methods at HZB, which is run by HZB together with the University of Kassel.

Read more on the HZB website

Calculating the “fingerprints” of molecules with artificial intelligence

2022/06/132022/06/28

With conventional methods, it is extremely time-consuming to calculate the spectral fingerprint of larger molecules. But this is a prerequisite for correctly interpreting experimentally obtained data. Now, a team at HZB has achieved very good results in significantly less time using self-learning graphical neural networks.

“Macromolecules but also quantum dots, which often consist of thousands of atoms, can hardly be calculated in advance using conventional methods such as DFT,” says PD Dr. Annika Bande at HZB. With her team she has now investigated how the computing time can be shortened by using methods from artificial intelligence.

The idea: a computer programme from the group of “graphical neural networks” or GNN receives small molecules as input with the task of determining their spectral responses. In the next step, the GNN programme compares the calculated spectra with the known target spectra (DFT or experimental) and corrects the calculation path accordingly. Round after round, the result becomes better. The GNN programme thus learns on its own how to calculate spectra reliably with the help of known spectra.