The Rubin Observatory Prompt Processing System

PO
Not scheduled
20m
Wichernhaus

Wichernhaus

Board: A130
poster presentation Automation of data pipeline and workflows Poster

Speaker

Krzysztof Findeisen (University of Washington)

Description

NSF-DOE Vera C. Rubin Observatory's upcoming Legacy Survey of Space and Time (LSST) will process 20 terabytes of raw images into 10 million transient alerts per night, every night for ten years. The Prompt Processing system deployed at the SLAC National Accelerator Laboratory automatically handles incoming images and generates alerts in near real time. To meet the ambitious throughput, latency, and reliability requirements for the survey, Prompt Processing uses a distributed, highly parallelized architecture based on cloud-native technologies, and provides redundancy against failures in both communication channels and processing nodes. The data pipeline itself is abstracted through the Rubin Data Management Middleware framework (Jenness et al. 2022), allowing the system to select between alternative pipelines based on data availability or image metadata, or to be reconfigured as appropriate. Prompt Processing was deployed starting early in Rubin Observatory commissioning, and we present performance metrics based on commissioning data.

Affiliation of the submitter University of Washington
Attendance in-person

Primary author

Krzysztof Findeisen (University of Washington)

Co-authors

Dan Speck (Burwood Group) Eric Bellm (University of Washington) Erin Howard (University of Washington) Hsin-Fang Chiang (SLAC National Accelerator Laboratory) Ian Sullivan (University of Washington) Kian-Tat Lim (SLAC National Accelerator Laboratory)

Presentation materials