Tech Notes: AI and the Data Lifecycle in NSF Major Facilities: From Experimental Tool to Embedded Intelligence

These Tech Notes are authored by Charles Vardeman and Donald Brower. 

View Tech Notes

Abstract:

Artificial intelligence has demonstrated extraordinary potential to accelerate scientific discovery, but realizing that potential within the operational realities of NSF Major Facilities remains a challenge. Enterprise platforms such as Snowflake’s in-warehouse copilots, Microsoft’s Power BI Copilot, and Databricks’ agentic orchestration systems illustrate how AI can function and augment cloud infrastructure, and yet we find that even well-resourced organizations struggle to move from experimentation to integration. The same holds true for the scientific enterprise: AI adoption is advancing, but unevenly, constrained by data silos, legacy systems, and cultural as well as governance barriers.

For Major Facilities, this unevenness is amplified by scale and mission. Some facilities operate mature data ecosystems with dedicated informatics and AI professionals; others rely on small, multidisciplinary teams balancing instrument operation, curation, and analysis. Across this spectrum, the question is no longer whether to use AI, but how to do so in ways that enhance understanding, accountability, and scientific trust.

This Tech Note introduces a framework for integrating generative and agentic AI across the data lifecycle through four archetypes of intelligence—augmented, automated, agentic, and explanatory. In this framework, the scientific copilot serves as the Human-AI Interface (HCI): a conversational collaborator that allows scientists to reason with their data, formulate hypotheses, and interpret results interactively. This interaction point becomes the natural gateway for integrating background agentic processes such as metadata capture, documentation, and provenance tracking—tasks that operate silently but maintain scientific integrity. While these capabilities can extend throughout the lifecycle, embedding them within the copilot interface makes them immediately accessible to researchers without adding cognitive burden. Automated processes enhance throughput without increasing staffing. Agentic orchestration coordinates data and compute resources across instruments and facilities. Explanatory intelligence bridges data and theory, strengthening understanding and trust. Together, these forms of intelligence can help facilities to evolve toward sustained, mission-driven integration of AI as part of their operational and scientific infrastructure.

The path forward depends on strategy as much as technology. Facilities must pair experimentation with governance by starting with targeted applications having clear value, cultivating iterative learning loops between humans and AI systems, and calibrating expectations with evidence. For instance, in a randomized trial on experienced open-source developers, using early-2025 AI tools increased completion time by approximately 19% despite users predicting a speedup. The goal of the strategy should not be simply to automate work, but to build the capacity to learn and adapt, turning a constraint into a catalyst for innovation thus ensuring that AI serves as a partner in reasoning rather than a black box in the workflow.