Virtual Workshop 2025 - Data Management: From Instrument to First Storage - Day 3 - Maintaining sanity while moving, storing, and cataloging terabytes of data and millions of files on a nightly basis by Robert Tawa, NSO

Maintaining sanity while moving, storing, and cataloging terabytes of data and millions of files on a nightly basis

By Robert Tawa, National Solar Observatory (NSO)

Large volumes of data contained in millions of files have become the norm of production in large facilities, which the Daniel K. Inouye Solar Telescope (DKIST) is a member of. The DKIST produces data from a multitude of instruments, creating large amounts of data in both small and large files which number in the millions. Moving said amount of data and files from the telescope to the storage and processing facilities on a regular basis, without data loss and with transmission assurance regardless of transmission channel glitches is basic to fulfilling the facility’s mission. The DKIST utilizes a slew of technologies and a scalable microservices architecture to move, store, inventory, and catalog the data such that the first part of the DKIST mission (retrieve and store) is completed without failure. The architecture and technologies also serve to make the scientific data searchable, discoverable, and easily retrievable by users.

NSF CI Compass hosted the U.S. National Science Foundation (NSF) CI Compass virtual workshop "Data Management: From Instrument to First Storage" from January 22-24, 2025.

View Virtual Workshop - Data Management: From Instrument to First Storage event page