Tech Notes: A Review of Streaming Systems and Architectures

A part of CI Compass' Tech Notes series, "Tech Notes: A Review of Streaming Systems and Architectures" was published on April 30, 2024. 

The paper is authored by CI Compass' Erik Scott and Steve Petruzza.

Introduction:

Sometimes science happens slowly, sometimes it comes at us like a firehose. Experiments that produce huge volumes of data, e.g. scientific instruments producing streaming data, are becoming more common with time. Capturing, processing, storing, and sharing high-volume, high-speed scientific data is a challenging problem and one that often requires a mix of off-the-shelf tools and custom, bespoke solutions.

In this technical note, we will take a short look at the challenges in more detail, highlight some existing tools that are useful to address these challenges and discuss some general strategies and design principles for those times that you have to take data matters into your own hands.