% O1-3.tex — ADASS 2025 Proceedings Paper
% Follows ADASS_template.tex (24 Oct 2025)
% No packages except asp2021

\documentclass[11pt,twoside]{article}

\usepackage{asp2021}

\aspSuppressVolSlug
\resetcounters

\bibliographystyle{asp2021}

% Running heads (modify as needed)
\markboth{Kruk et al.}{ESA Datalabs and Euclid Data Space}

\begin{document}

\title{Addressing Big Data Challenges for ESA Space Science Missions}

\author{
Sandor~Kruk$^1$,
Vicente~Navarro$^1$,
Sebastian~Maksym$^1$,
Pablo~G\'omez$^1$,
Guillaume~Belanger$^1$,
Jan~Reerink$^1$,
Christophe~Arviset$^1$,
Pierre~Ferruit$^1$,
Valeria~Pettorino$^2$,
Bruno~Altieri$^1$,
Jos~de~Bruijne$^2$,
Filip~Marinic$^1$,
Marcos~L\'opez-Caniego$^3$,
H\'ector~C\'anovas$^4$ and
Daniela~Coia$^4$
}

\affil{$^1$European Space Astronomy Centre (ESAC), European Space Agency (ESA),\\
Villanueva de la Cañada, Madrid, Spain; \email{sandor.kruk@esa.int}}

\affil{$^2$European Space Agency (ESA), European Space Research and Technology Centre (ESTEC), Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands}

\affil{$^3$Aurora Technology for ESA, European Space Astronomy Centre (ESAC),\\
Villanueva de la Cañada, Madrid, Spain}

\affil{$^4$Telespazio UK for ESA, European Space Astronomy Centre (ESAC),\\
Villanueva de la Cañada, Madrid, Spain}

% ADS metadata


% \aindex entries optional

\begin{abstract}
The rapid growth of data volumes from the European Space Agency's (ESA) space science missions has created a pressing need for scalable, interoperable, and collaborative analysis environments. ESA Datalabs is a science platform designed to provide researchers with ready-to-use computational environments located close to the ESA Science Archives, enabling data analysis without massive data downloads. The platform integrates Jupyter-based services, GUI tools, shared workspaces, and direct access to ESA multi-mission data.

In particular, the Euclid mission introduces unprecedented data volume and complexity. To support the mission’s scientific community, ESA has developed the Euclid Data Space (EDS), a unified portal combining the data discovery (Euclid Archive), visualization (ESASky), and analysis (ESA Datalabs). We present the motivation, architecture, and operational experience of ESA Datalabs and EDS, describe key functionalities, and highlight use cases including the Euclid Quick Data Release (Q1), large-scale cutout generation and machine-learning use cases. We conclude with lessons learned and prospects for Euclid DR1 and other ESA missions.
\end{abstract}

% Subject indices (add relevant ones)
% \ssindex{data analysis!science platforms}
% \ssindex{surveys!Euclid}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

The era of Big Data in astronomy is characterised by exponential growth in data volume and complexity. Space science missions, ranging from \emph{HST} and \emph{XMM-Newton} to \emph{Gaia}, \emph{JWST} and \emph{Euclid}, collect and process data products at unprecedented scales. Figure \ref{fig:archives}  shows the growth of this data stored in the ESAC Science Data Center. \emph{Euclid} alone is expected to generate almost 30 PB of calibrated and intermediate data over its operational lifetime.

Traditional workflows relying on data downloads are no longer feasible at petabyte scale. Also, as scientific workflows evolve toward machine learning, image level analyses, and multi-mission cross-correlation, users increasingly require compute resources close to the data. Science platforms provide a paradigm shift, enabling researchers to analyse mission data directly where it resides. 

ESA Datalabs\footnote{\url{https://datalabs.esa.int/}} (Figure \ref{fig:platform}) is the science platform of ESA's Science Directorate \citep{Navarro2024}. It is designed to increase data accessibility, operational efficiency, and scientific return for ESA missions by providing integrated tools, compute resources, and collaborative features.

\articlefigure[width=\textwidth]{168_f1.jpg}{fig:archives}{Growth of the data in the ESA Space Science Archives over time, categorised by ESA mission. The ESA \emph{Euclid} mission contributes most of the data in the archives.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The ESA Datalabs science platform}

\textit{Why a science platform?} Historically, scientific workflows follow a model in which astronomers search for the data online, download the data needed, then analyse the data locally and publish the results. With petabyte‑scale missions, this model becomes inefficient or impossible. Science platforms reverse this paradigm by moving computation to the data.

ESA Datalabs is a science platform that provides researchers with preconfigured scientific environments hosted at ESAC. It is built around three core objectives:

\begin{enumerate}
\item \textbf{Boost Access to Tools} – Pre‑configured, ready‑to‑use JupyterLab and GUI applications,
\item \textbf{Boost Access to Data} – Direct access to ESA mission archives via NFS‑mounted volumes and VO‑compliant APIs,
\item \textbf{Boost Science Collaboration} – Shared workspaces, environments, and collaborative analysis.
\end{enumerate}

The \textit{bring code to the data} model minimizes the data movement and supports reproducible and collaborative research.

\articlefigure[width=1\textwidth]{168_f2.png}{fig:platform}{The landing page of \url{https://datalabs.esa.int/}.}

\subsection{System infrastructure}

The ESA Datalabs computing cluster includes both mission-dedicated and general purposed nodes hosted in the data center at ESAC. Each compute node provides large-memory, multi-CPU, and GPU-enabled resources:

\begin{table}[!ht]
\smallskip
\begin{center}
{\small
\begin{tabular}{lc}
\tableline
Resource & Specification \\
\tableline
System Infrastructure & General pool + mission-dedicated nodes \\
Memory per node & $\sim 1.25$~TB RAM \\
CPU per node & $\sim 128$ vCPUs \\
GPUs per node & 2$\times$NVIDIA L4 GPUs \\
Network & 100 Gbps \\
\tableline
\end{tabular}
\caption{Representative ESA Datalabs Node Resources}
}
\end{center}
\end{table}


The pool of general-purpose nodes supports most missions, while dedicated \emph{Euclid} nodes ensure capacity for several hundred concurrent users during major data releases. ESA Datalabs is built on modern container‑orchestration technologies to ensure scalability and consistent user environments. The platform deploys applications as Docker containers managed by Kubernetes, while Rancher provides a unified interface for cluster administration and workload scheduling. This architecture allows the system to scale efficiently during peak activity and ensures reproducible software environments across all users.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{User access and community growth}

Access to ESA Datalabs is currently moderated to ensure responsible use of the shared computational resources. Scientists may request access using an institutional e-mail address and by providing either a short statement of motivation or an invitation code provided by ESA missions, collaboration consortia, or by workshop organisers.

Since its launch, ESA Datalabs has experienced rapid growth in its user community. Figure~\ref{fig:users} shows the cumulative number of registered users, which has now reached over 3,000. The number of active users (defined as those who have launched at least one application within the preceding 72~hours) is currenly approximately 100. 

\articlefigure[width=\textwidth]{168_f3.png}{fig:users}{The growth in the number of registered ESA Datalabs users over time.}

Science platforms such as ESA Datalabs have proven particularly effective for workshops, hackathons, and hands-on trainings. Because all tools and mission data are provided through a browser-based interface, participants are not required to install software locally or configure computing environments across different operating systems. Organisers can instead prepare shared workspaces, datasets, and instructional material in advance, ensuring a uniform access for all participants and enabling collaborative workflows during hands-on sessions.

To date, ESA Datalabs has supported more than twenty workshops across astronomy, planetary science, heliophysics, and data science disciplines. Recent events include:
\begin{itemize}
    \item Machine Learning in Heliophysics Workshop\footnote{\url{https://ml-helio.github.io/}}, 
    \item Working with Euclid Q1 Data\footnote{\url{https://www.cosmos.esa.int/web/working-with-euclid-q1/home}},
    \item ESA Datalabs Ariel Hackathon\footnote{\url{https://www.ariel-datachallenge.space/esa-datalabs25/}},
    \item \texttt{.Astronomy}\footnote{\url{https://www.dotastronomy.com/}}.
\end{itemize}

These activities have contributed substantially to community adoption of platforms by lowering technical barriers and promoting best practices in reproducible, cloud-enabled research.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Euclid Data Space (EDS)}

The ability to enable large data releases directly within science platforms depends on seamless integration between three traditionally distinct components: the long-term data archives, the tools used for data visualisation, and the interactive environments used for scientific analysis. Historically, these systems evolved independently, reflecting the different needs of their respective times. Science archives were established first, providing curation, metadata management, and preservation of mission data. Later, tools such as ESASky\footnote{\url{https://sky.esa.int/}} offered essential visualisation capabilities for exploring complex, multi-instrument datasets stored in the archives. More recently, science platforms such as ESA Datalabs have emerged to support interactive analysis, multi-mission access, and collaborative development directly next to the data.

\articlefigure[width=1.0\textwidth]{168_f4.png}{fig:EDS}{Representative layout of the Euclid Data Space portal (\url{https://euclid.dataspace.esa.int/}), showing the integration of the Euclid Archive, ESASky visualisation services, and ESA Datalabs analysis environments.}

The \emph{Euclid} mission brings these requirements together. To support both discovery and analysis workflows, Euclid Data Space\footnote{\url{https://euclid.dataspace.esa.int/}} (EDS) provides a unified, single-entry portal that integrates the Euclid Archive, ESASky visualisation services, and ESA Datalabs computational environments (Figure \ref{fig:EDS}). Through this integration, users can move easily from querying the archive, to visual inspection, to interactive analysis without transferring large datasets locally or navigating multiple websites and interfaces.

EDS implements role-based access control, enabling differentiated view for public users and members of the Euclid Consortium. It further provides topic-oriented views, embedded applications, and mission-tailored interfaces that simplify navigation across \emph{Euclid}’s diverse data products. An internal release of EDS for the Euclid Consortium took place in late 2025, with the first public release planned for 2026.

The use of ESA Datalabs and EDS to enable the \emph{Euclid} use cases and data releases is detailed in the following subsections.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Euclid pilot studies}

In preparation for \emph{Euclid}'s early data releases, ESA Datalabs supported a series of pilot studies to validate data-access strategies, show representative analysis workflows, and evaluate performance for large-scale in-situ processing. These studies covered a broad set of science and operations use cases, including: identifying moving Solar System objects across repeated Euclid exposures; detecting transient sources such as supernovae; reprocessing individual observations with alternative sky-subtraction strategies to enhance low-surface-brightness structures; stacking Euclid spectra; and using the Cosmology Likelihood for Observables in Euclid (CLOE; \citealt{Cardone2025}) for cosmology forecasts.

The pilot studies provided important input for the platform optimisation, helping to assess resource needs, refine user workflows, and identify missing functionalities prior to the wider release. They also established proven patterns for scalable analysis within ESA Datalabs, thereby informing the design and integration of \emph{Euclid}-specific tools within the Euclid Data Space.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Euclid Quick Data Release (Q1)}

The first \emph{Euclid} data release supported on the platform was the Euclid Quick Data Release (Q1)\footnote{\url{https://www.cosmos.esa.int/web/euclid/euclid-q1-data-release}}, published on 19~March~2025. Q1 consisted of three deep fields covering a total of $\sim 63~\mathrm{deg}^{2}$ and approximately $35$~TB of data \citep{Aussel2025}. ESA Datalabs played a central role in enabling rapid community access to these data products by providing:

\articlefigure[width=1\textwidth]{168_f5.png}{fig:EuclidQ1}{View of the \textit{Euclid Q1} application in ESA Datalabs, showing the mounted \texttt{euclid\_q1} volume, a \textit{Getting Started Guide} describing the available data products, and links to example Jupyter Notebooks.}

\begin{itemize}
    \item mission-mounted \emph{Euclid} data volumes,
    \item a dedicated \textit{Euclid Q1} JupyterLab application (see Figure \ref{fig:EuclidQ1}),
    \item astroquery-based programmatic access via VO-compliant protocols, returning file paths to data already mounted within ESA Datalabs,
    \item a suite of ready-to-use Jupyter tutorials for \emph{Euclid}, including examples for accessing Q1 data via astroquery, generating image cutouts, visualising images and spectra, and performing data downselection using ADQL queries\footnote{These notebooks are openly accessible on GitHub at \url{https://github.com/ESA-Datalabs/Euclid-Q1}},
    \item collaborative workspaces for teams,
    \item monitoring dashboards (e.g.\ Grafana) to track system load and user activity.
\end{itemize}

The Q1 release represented the first large-scale use of ESA Datalabs for scientific analysis. 11 out of 27 Euclid Q1 science papers (about 40\%) reported using ESA Datalabs for tasks such as catalogue queries and cross-matching, direct inspection of FITS files, large-scale cutout generation, AI-based inference workflows, and collaborative analysis. A survey of participating scientists provided further information into their workflows, highlighting missing software packages, desired improvements to the user experience, and priorities for future development.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Representative use case: Galaxy Zoo for Euclid}

One representative example of early science enabled by ESA Datalabs is the creation of a large, high‑quality morphological catalogue of \emph{Euclid} galaxies using a combination of volunteer classifications via \textit{Galaxy Zoo} and machine learning \citep{Walmsley2025a}\footnote{\url{https://www.esa.int/Science_Exploration/Space_Science/Euclid/Euclid_Galaxy_Zoo_help_us_classify_the_shapes_of_galaxies}}.

%integration of Euclid imaging into the Galaxy Zoo citizen‑science platform 

\articlefigure[width=1\textwidth]{168_f6.png}{fig:GZ}{Workflow for the Galaxy~Zoo: \emph{Euclid} project. Image cutouts are generated within ESA Datalabs and submitted to Galaxy~Zoo for volunteer classification, which is used to train the Zoobot machine‑learning model \citep{Walmsley2023}. Zoobot is then run in inference mode on ESA Datalabs to produce large‑scale morphological catalogues of \emph{Euclid} galaxies.}

The first step involved producing large-scale image cutouts of bright and extended galaxies directly within ESA Datalabs. These cutouts were generated and provided to \textit{Galaxy~Zoo} for human classification by almost 10,000 volunteers, supplying the training data required for Zoobot to achieve high performance in automated identification of morphological features such as spiral arms, bars, tidal distortions, and signatures of galaxy mergers. Zoobot was then run in inference mode within ESA Datalabs to predict the morphology of approximately $380{,}000$ galaxies. The current catalogue represents only $\sim 0.4\%$ of the galaxies expected to be observed over \emph{Euclid}'s Wide Survey, implying that the final morphological catalogue will exceed previous surveys by at least an order of magnitude.

ESA Datalabs played a key enabling role in this effort by providing scalable in situ cutout production, shared workspaces, and GPU‑backed environments for running machine learning pipelines. The same methodology has been applied to other early science cases. Using a combination of AI‑based preselection, citizen science inspection, expert vetting, and subsequent modelling, a first catalogue of roughly 500 galaxy–galaxy strong-lens candidates has been built \citep{Walmsley2025b}, demonstrating the potential of the platform for large‑scale discovery workflows\footnote{\url{https://www.esa.int/Science_Exploration/Space_Science/Euclid/Euclid_opens_data_treasure_trove_offers_glimpse_of_deep_fields}}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Preparing for Euclid DR1}

Euclid Data Release~1 (DR1) represents a huge increase in data volume and complexity compared to the Q1 release\footnote{\url{https://www.cosmos.esa.int/web/euclid/data-release-dr1}}. At approximately $1.5$~PB and covering $\sim 1900~\mathrm{deg}^2$  \citep{Mellier2025}, DR1 is more than thirty times larger than Q1, placing fundamentally new demands on storage, data access, and interactive analysis. Unlike Q1, DR1 is far too large to be downloaded and analysed locally by most users, making analysis on science platforms not only advantageous but essential.

DR1 was prepared for Euclid Consortium access in late 2025, with a public release planned through the Euclid Data Space in 2026. In anticipation of this milestone, substantial work has been undertaken to ensure that ESA Datalabs, ESASky, and the Euclid Archive operate together as a coherent and robust system capable of supporting thousands of users during the release.

Current preparation efforts focus on scaling the underlying infrastructure and refining the user experience. Key areas of development include:

\begin{itemize}
\item scalable data-mounting strategies,
\item fair-share computing for hundreds of active users,
\item enabling GPU-backed ML workflows,
\item EDS seamless integration with Euclid Archives, ESASky, and Datalabs with a single-sign-on.
\end{itemize}


\section{Conclusions}

ESA Datalabs has emerged as a central component of ESA’s data‑analysis ecosystem, providing a unified, scalable, and analysis‑ready environment for a wide range of space‑science missions. By collocating computation with mission data and offering a consistent, browser‑based interface, ESA Datalabs lowers the barriers to accessing and analysing complex, PB‑scale datasets. Its integration within the Euclid Data Space (EDS) represents a major step toward fully end‑to‑end scientific workflows, enabling users to transition smoothly from data discovery, to visualisation, to interactive analysis without downloading large data volumes. The Euclid Q1 data release showed the platform’s scientific impact, and preparations for DR1 highlight its essential role for future ESA missions facing PB-scale data.

Beyond \emph{Euclid}, ESA Datalabs is already supporting a diverse set of missions and scientific use cases across astronomy, planetary science, and heliophysics. Workshops, hackathons, and training events have shown that science platforms are efficient mechanisms for building expertise, fostering collaboration, and aligning best practices across disciplines. As the landscape of data‑intensive research continues to evolve, platforms such as ESA Datalabs will play an increasingly central role in enabling open, reproducible, and community‑driven science.

The experience gained through the development and operation of ESA Datalabs and the Euclid Data Space has highlighted several key lessons:

\begin{itemize}
\item Bringing compute to the data is essential for PB-scale missions.
\item Science platforms complement rather than replace archives.
\item VO protocols ensure data discoverability via the archives and effective use through mounted volumes on the science platforms.
\item Platforms must support both novice and expert users.
\item Clear communication of resource limits manages expectations.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgements
We thank the ESAC Data Science Section, ESAC Science Data Centre, the Programatic IT Section, the ESA Datalabs and Euclid Data Space Development Teams, and our industrial partners (CGI, ACRI-ST, Edisoft, Uninova). We also thank ESA mission scientists and archive teams for their support and feedback.

\bibliography{168.bib}

% \bookpartphoto[width=1.0\textwidth]{conference_photo.jpg}{Conference photo (placeholder)}

\end{document}


