Brookhaven Lab team pioneers interactive virtual companion to accelerate discoveries at scientific user facilities
UPTON, N.Y. — A team of scientists at the U.S. Department of Energy’s (DOE) Brookhaven National Laboratory have dreamed up, developed, and tested a novel voice-controlled artificial intelligence (AI) assistant designed to break down everyday barriers for busy scientists.
Known as the Virtual Scientific Companion, or VISION, the generative AI tool – developed by researchers at the Lab’s Center for Functional Nanomaterials (CFN) with support from experts at the National Synchrotron Light Source II (NSLS-II) — offers an opportunity to bridge knowledge gaps at complex instruments, carry out more efficient experiments, save scientists’ time, and overall, accelerate scientific discovery.
The idea is that a user simply has to tell VISION in plain language what they’d like to do at an instrument and the AI companion, tailored to that instrument, will take on the task — whether it’s running an experiment, launching data analysis, or visualizing results. The Brookhaven team recently shared details about VISION in a paper published in Machine Learning: Science and Technology.
“I’m really excited about how AI can impact science and it’s something we as the scientific community should definitely explore,” said Esther Tsai, a scientist in the AI-Accelerated Nanoscience group at CFN. “What we can’t deny is that brilliant scientists spend a lot of time on routine work. VISION acts as an assistant that scientists and users can talk to for answers to basic questions about the instrument capability and operation.”
VISION highlights the close partnership between CFN and NSLS-II, two DOE Office of Science user facilities at Brookhaven Lab. Together they collaborate with facility users on the setup, scientific planning, and analysis of data from experiments at three NSLS-II beamlines, highly specialized measurement tools that enable researchers to explore the structure of materials using beams of X-rays.
Tsai, inspired to alleviate bottlenecks that come with using NSLS-II’s in-demand beamlines, received a DOE Early Career Award in 2023 to develop this new concept. Tsai now leads the CFN team behind VISION, which has collaborated with NSLS-II beamline scientists to launch and test the system at the Complex Materials Scattering (CMS) beamline at NSLS-II, demonstrating the first voice-controlled experiment at an X-ray scattering beamline and marking progress towards the world of AI-augmented discovery.
“At Brookhaven Lab, we’re not only leading in researching this frontier scientific virtual companion concept, we’re also being hands-on, deploying this AI technique on the experimental floor at NSLS-II and exploring how it can be useful to users,” Tsai said.
Talking to AI for flexible workflows
VISION leverages the growing capabilities of large language models (LLMs), the technology at the heart of popular AI assistants such as ChatGPT.
An LLM is an expansive program that creates text modeled on natural human language. VISION exploits this concept, not just to generate text for answering questions but also to generate decisions about what to do and computer code to drive an instrument. Internally, VISION is organized into multiple “cognitive blocks,” or cogs, each comprising an LLM that handles a specific task. Multiple cogs can be put together to form a capable assistant, with the cogs carrying out work transparently for the scientist.
“A user can just go to the beamline and say, ‘I want to select certain detectors’ or ‘I want to take a measurement every minute for five seconds’ or ‘I want to increase the temperature’ and VISION will translate that command into code,” Tsai said.
Those examples of natural language inputs, whether speech, text, or both, are first fed to VISION’s “classifier” cog, which decides what type of task the user is asking about. The classifier routes to the right cog for the task, such as an “operator” cog for instrument control or “analyst” cog for data analysis.
Then, in just a few seconds, the system translates the input into code that’s passed back to the beamline workstation, which the user can review before executing. On the back end, everything is being run on “HAL,” a CFN server optimized for running AI workloads on graphics processing units.
Read more on BNL website
Image: VISION aims to lead the natural-language-controlled scientific expedition with joint human-AI force for accelerated scientific discovery at user facilities.
Credit: Brookhaven National Laboratory
