Speaker
Description
The first Euclid Quick Data Release (Q1) encompasses approximately 30 million sources across 63.1 square degrees, marking the beginning of a mission providing petabytes of imaging data through Data Release 1 (DR1) and subsequent releases. Systematic scientific exploitation of these datasets frequently requires extraction of source-specific cutouts; however, the scale of modern surveys renders permanent storage of such derivative products impractical. Standard tools like astropy's Cutout2D process sources individually, creating significant bottlenecks when generating cutouts for large catalogues. 'To address this, we introduce Cutana, a parallelised, memory-efficient pipeline optimised for batch processing, both locally and in cloud-native environments.
The software implements automated parallel thread management with explicit memory-aware scheduling, accommodating both local systems and containerised deployments such as ESA Datalabs' Kubernetes infrastructure. Input consists of source catalogues containing celestial coordinates and source sizes , paired with corresponding FITS tile collections. Rather than iterative single-source extraction, Cutana employs fully vectorised NumPy operations to extract large batches of cutouts simultaneously from loaded tiles, substantially reducing per-source overhead. The pipeline supports flexible output specifications including variable bit precision (8-32 bit), the Zarr format alongside traditional FITS, and multiple normalisation schemes (linear, logarithmic, asinh, zscale). An ipywidget interface provides parameter configuration, preview generation, and real-time monitoring of processing metrics including memory usage and throughput.
Performance benchmarking demonstrates processing rates of hundreds to thousands of cutouts per second on individual systems (8 cores, 64GB RAM) while minimising memory footprint by tile-based batching. Integration with ESA Datalabs infrastructure is underway for inclusion in the Euclid IDR1 release as well as an ESA open-source licensing process to ensure community access.
| Affiliation of the submitter | European Space Agency |
|---|---|
| Attendance | in-person |