CSST Data Processing Pipeline: Architecture and Development

P2
10 Nov 2025, 11:15
15m
Synagoge

Synagoge

oral presentation Automation of data pipeline and workflows Plenary Session 2

Speaker

BO ZHANG (National Astronomical Observatories, CAS)

Description

The Chinese Space Station Survey Telescope (CSST) is China's first major space-based optical survey facility, equipped with advanced instruments including the Main Survey Camera (MSC), Multi-Channel Imager (MCI), Integral Field Spectrograph (IFS), Cool Planet Imaging Coronagraph (CPIC), and High Sensitivity Terahertz Detection Module (HSTDM), covering wavelengths from near-ultraviolet (NUV) to near-infrared (NIR). Over its 10-year operational lifetime, CSST is expected to generate petabytes of multi-level data products, enabling breakthroughs in fields such as cosmology, galaxy evolution, and exoplanet detection.

To process this data efficiently, we designed the CSST Scientific Data Processing Pipeline based on a ​cloud-native architecture, leveraging elastic cloud resources. Key algorithms—such as instrumental effect correction, photometric calibration, and spectral extraction—are encapsulated into containerized, reusable ​Operators. The pipeline is orchestrated via ​DAG-defined workflows​ and managed on a ​Kubernetes cluster, ensuring high-throughput processing and dynamic resource optimization.

The underlying data management system supports efficient storage, retrieval, and cross-matching of tens of billions of source catalog entries. This is achieved through ​distributed database technologies​ and customized indexing strategies, ensuring efficient handling of complex queries on large-scale catalog data and future integration with external datasets.

For scientific exploration, we are developing a ​cloud-based scientific workbench. This platform allows astronomers to submit custom Python scripts directly adjacent to data storage (compute-near-data), enabling advanced analysis, simulations, and visualizations without downloading large data products, significantly improving efficiency and reducing migration risks.

Affiliation of the submitter National Astronomical Observatories, CAS
Attendance in-person

Primary author

BO ZHANG (National Astronomical Observatories, CAS)

Presentation materials