Tech Notes: Evaluation of Scalable Solutions for Time Series Database Streaming

This Tech Note presents an evaluation of scalable solutions for streaming time-series data, critical for real-time analysis in large-scale national research facilities like the NSF Laser Interferometer Gravitational-Wave Observatory (LIGO). The study assesses various time-series databases (ClickHouse, InfluxDB, TimescaleDB) and communication protocols (Kafka, Arrow Flight), focusing on query performance, data ingestion, and scalability. ClickHouse and Kafka emerged as preferred solutions, providing high performance and flexibility for environments with large-scale data requirements. The evaluation is based on use cases from facilities like LIGO, aiming to improve real-time data processing capabilities in NSF Major Facilities.

This Tech Notes is authored by: Valerio Pascucci, Giorgio Scorzelli, Steve Petruzza, Erik Scott, Jarek Nabrzyski, Anirban Mandal, Jameson Rollins, Patrick Godwin, Jonathan Hanks, and Martin Beroiz.