The 35th annual conference on Astronomical Data Analysis Software & Systems 2025 in Görlitz, Germany

This tutorial will provide a hands-on introduction to automating unit testing in Python using the industry-standard pytest framework. Participants will learn how to design software components that are easily testable, write effective unit tests using pytest to ensure software robustness and maintainability, and generate and utilize code coverage metrics to verify the comprehensiveness of their test suite. The session will emphasize how these practices contribute to a more reliable software stack that enables reproducible research.

List of what participants will need:
- Wifi capable laptop in any operating system that can run python 3. Exercises are tested on MacOS and Linux.
- Code exercise repository: https://github.com/Smithsonian/adass2025-tutorial-pytest
- Python3 and pre-installed python packages (listed in code repository)

Instructions to Participants
1. Please setup your wifi capable laptop that can run Python 3 and has git version control installed which you must bring to the session.
a. Install anaconda https://conda-forge.org/download/
b. git clone https://github.com/Smithsonian/adass2025-tutorial-pytest.git
c. cd adass2025-tutorial-pytest
d. conda env create -n adasstest -f environment.yml
e. conda activate adasstest
f. python -c “print(‘hello world’)” # Must type out, do not copy paste because quote marks will be wrong.
If you see 'hello world', you are good to go.
If you do not, feel free to email gregory.ciccarelli@cfa.harvard.edu with the error message.

The European Space Agency (ESA) Euclid Quick Release 1 (Q1) consists of some 35 TB of single-epoch imaging, spectroscopy, and catalogues covering three Euclid Deep Fields of combined size 63 deg2 on the sky. The data are available at the ESAC Science Data Centre (ESDC) through the Euclid Science Archive as well as the ESA Datalabs e-science platform. ESA Datalabs offers direct access to the data, without the need for downloading them, and augments the user experience with, among others, tutorial notebooks and team collaboration facilities. This tutorial focuses on providing hands-on training on accessing and analysing public Euclid Q1 data in the Science Archive and in ESA Datalabs. It also offers do's and don'ts for optimal workflow design, geared towards forthcoming petabyte-level data releases.

Prerequisites for participants
- Charged laptop with a web browser and internet connection (e.g., eduroam)
- ESA Datalabs account. Important: to join the session, you will need an ESA Datalabs account.
Please register by 2 November (end of business) at the following link:
https://www.cosmos.esa.int/web/datalabs/self-registration When registering, please enter
“EUCLIDQ1” in the “Your interest in ESA Datalabs/invitation code” field to ensure your
account is approved. Unfortunately, on-the-spot registration for an account is not possible.
- Some basic knowledge of Python
- Familiarity with ADQL / astroquery is an asset but is not mandatory
- Familiarity with the Euclid mission or Euclid data is not assumed
- In case of questions about the session or issues with the creation of the Datalabs account,
please contact the Euclid Helpdesk at: https://www.cosmos.esa.int/web/euclid/euclid-
helpdesk

Obtaining a clear picture of a new or even existing codebase is difficult. This
is especially true if the aim is to ascertain which parts of the code are
crucial to the primary function or performance of the software, and which are
for handling edge-cases, memory management, input/output (IO), or even argument
parsing. One of the most effective means by which to learn such information is
to let the program tell you, by profiling how the code is executing on your
hardware. Profiling provides insight into how the program navigates
instructions, interacts with the system memory and caches, the underlying
operating system kernel, network devices, the file system, thread
communication, or how a modern CPU is pipelineing and speculatively executing
performance critical components.

This workshop will provide a hands-on introduction to software profiling for
codebase exploration, debugging, and performance optimisation. It will cover the
basic concepts behind statistical and frame-based methods in a generally
language and system agnostic manner. The hands-on sessions are specific examples
using compiled binaries, perf events, the Tracy graphical profiler, and more.

The main goal of this tutorial is to teach participants how to use hierarchical Virtual Observatory (VO) standards allowing construction, exploration and querying of all-sky datasets. The Hierarchical Progressive Survey (HiPS) and the Space-Time Multi-Order Coverage map (ST-MOC) standards can be used by data providers to expose their datasets (images or catalogues), and astronomers can use them to perform complex queries and operations on all-sky datasets.

Introduction and requirements for participants can be found at
https://cds.unistra.fr/help/tutorials-more/adass-2025/

Large astronomical facilities produce enormous amounts of data that require automatic processing to be able to analyze and publish it in a timely fashion. When the ALMA observatory was planned more than 25 years ago, one of the key requirements was an automatic pipeline to make the data products readily available for the whole community, independent of the PI's background at a particular wavelength and its corresponding processing techniques. For a radio interferometer this was fundamentally different compared to how existing facilities operated at the time. The ALMA data products were supposed to be close to ready for scientific analysis and checked for quality. This led to the development of the CASA/ALMA pipeline with tailored heuristics and supplementary systems to schedule the processing and archiving of the data. In this talk I will review the pipeline developments, its important features in terms of heuristics and book-keeping, data product quality assessment and weblog presentation of the results. The pipeline is now able to process almost all standard observing mode data sets. The planned ALMA upgrade will bring orders of magnitude more data and the challenge to still process it in good time. But the basic concepts of the existing pipeline remain valid and will be transferred to the new system.

The growing volume of alerts from time-domain and multi-messenger astronomy, often with poor localization, necessitates automated and optimized follow-up scheduling. We present tilepy, an open-source Python package designed to meet this challenge by automatically generating efficient observation plans.
tilepy processes HEALPix-based sky maps to derive pointing schedules using various strategies, from 2D probability integration to 3D methods that exploit distance estimates to correlate event localization with galaxy catalogs. Its scheduling engine is highly flexible, handling complex operational constraints for both ground-based (e.g., zenith limits, sky brightness) and space-based observatories (e.g., occultations, SAA passage). The framework also handles diverse and non-circular FoV shapes, including rotation, to accurately model various instrument footprints.

Designed for broad integration, tilepy is available as a public package on GitHub and PyPI, accessible via a cloud-based REST API at tilepy.com, and is seamlessly integrated into the Astro-COLIBRI multi-messenger platform for user-friendly web and mobile access. The framework is in operational use by major collaborations like H.E.S.S. and CTAO/LST-1, providing a robust and accessible solution for efficient transient follow-up.

PySE (Python Source Extractor) was developed by Spreeuw & Swinbank between 2005 and 2010, as part of the LOFAR Transients Key Project. It has been in continuous use since 2017 within the Amsterdam–ASTRON Radio Transients Facility and Analysis Center (AARTFAAC) pipeline. More recently (2023), major performance enhancements reduced runtime dramatically: offline processing of typical 2300²-pixel AARTFAAC images with ~2000 sources improved from ~20 seconds to 0.9 seconds. Over the past two years, we have benchmarked PySE against a representative Square Kilometre Array (SKA) artificial test image of 4096² pixels, measuring all 167,000 inserted sources, within 0.9 seconds on a high-end consumer CPU. This was made possible by vectorizing the source extraction and measurement stages with Numba’s guvectorize decorator.

This talk will focus on the usability of the new PySE, in particular its “sky-eater” mode for rapid processing of very large images. We will briefly compare its advantages and limitations with other astronomical source extractors. We will then demonstrate PySE’s straightforward installation via pip and show how images can be processed from an IPython shell with just a few commands. Selected source parameters are collected in a Pandas dataframe, which can conveniently be stored as an HDF5 file.

Real-time performance is scientifically crucial: transient surveys typically generate a new sky image every second. If source extraction lags behind acquisition, detections are lost and follow-up becomes impossible. PySE’s ability to keep pace ensures that astronomical “alerts” can be issued within seconds, allowing other telescopes worldwide to repoint rapidly—an essential capability for catching rare events such as the merger of two neutron stars.

PySE is not limited to radio astronomy: it is a versatile package that takes astronomical images as input and delivers robust source-parameter measurements as output, making it suitable for integration as a library into pipelines across other wavelength domains.

The Chinese Space Station Survey Telescope (CSST) is China's first major space-based optical survey facility, equipped with advanced instruments including the Main Survey Camera (MSC), Multi-Channel Imager (MCI), Integral Field Spectrograph (IFS), Cool Planet Imaging Coronagraph (CPIC), and High Sensitivity Terahertz Detection Module (HSTDM), covering wavelengths from near-ultraviolet (NUV) to near-infrared (NIR). Over its 10-year operational lifetime, CSST is expected to generate petabytes of multi-level data products, enabling breakthroughs in fields such as cosmology, galaxy evolution, and exoplanet detection.

To process this data efficiently, we designed the CSST Scientific Data Processing Pipeline based on a ​cloud-native architecture, leveraging elastic cloud resources. Key algorithms—such as instrumental effect correction, photometric calibration, and spectral extraction—are encapsulated into containerized, reusable ​Operators. The pipeline is orchestrated via ​DAG-defined workflows​ and managed on a ​Kubernetes cluster, ensuring high-throughput processing and dynamic resource optimization.

The underlying data management system supports efficient storage, retrieval, and cross-matching of tens of billions of source catalog entries. This is achieved through ​distributed database technologies​ and customized indexing strategies, ensuring efficient handling of complex queries on large-scale catalog data and future integration with external datasets.

For scientific exploration, we are developing a ​cloud-based scientific workbench. This platform allows astronomers to submit custom Python scripts directly adjacent to data storage (compute-near-data), enabling advanced analysis, simulations, and visualizations without downloading large data products, significantly improving efficiency and reducing migration risks.

The first Euclid Quick Data Release (Q1) encompasses approximately 30 million sources across 63.1 square degrees, marking the beginning of a mission providing petabytes of imaging data through Data Release 1 (DR1) and subsequent releases. Systematic scientific exploitation of these datasets frequently requires extraction of source-specific cutouts; however, the scale of modern surveys renders permanent storage of such derivative products impractical. Standard tools like astropy's Cutout2D process sources individually, creating significant bottlenecks when generating cutouts for large catalogues. 'To address this, we introduce Cutana, a parallelised, memory-efficient pipeline optimised for batch processing, both locally and in cloud-native environments.

The software implements automated parallel thread management with explicit memory-aware scheduling, accommodating both local systems and containerised deployments such as ESA Datalabs' Kubernetes infrastructure. Input consists of source catalogues containing celestial coordinates and source sizes , paired with corresponding FITS tile collections. Rather than iterative single-source extraction, Cutana employs fully vectorised NumPy operations to extract large batches of cutouts simultaneously from loaded tiles, substantially reducing per-source overhead. The pipeline supports flexible output specifications including variable bit precision (8-32 bit), the Zarr format alongside traditional FITS, and multiple normalisation schemes (linear, logarithmic, asinh, zscale). An ipywidget interface provides parameter configuration, preview generation, and real-time monitoring of processing metrics including memory usage and throughput.

Performance benchmarking demonstrates processing rates of hundreds to thousands of cutouts per second on individual systems (8 cores, 64GB RAM) while minimising memory footprint by tile-based batching. Integration with ESA Datalabs infrastructure is underway for inclusion in the Euclid IDR1 release as well as an ESA open-source licensing process to ensure community access.

ESA's Euclid cosmology mission was launched in 2023 and during its five year life time it will observe one third of the sky with unprecedented resolution with its optical and near infrared instruments.

The Euclid data will be hosted by the Euclid Science Archive, within the ESAC Science Data Centre (ESDC) in Madrid. The ESDC is in charge of the development, operations and maintenance of the science archives for all the ESA astronomy, planetary and heliophysics missions, as well as for the ESA Human Robotic and Exploration investigations.

The Euclid Science Archive released the Quick Release 1 (QR1) in May 2025, making available to the public 35 Terabytes of data.
The next major milestone is the Internal Data Release 1 (IDR1) in October 2025, whose data will grow to 2.2 Petabytes.
By the end of the mission, over 25 Petabytes of official products from the Euclid mission will be hosted at the Euclid Science Archive.

The purpose of this presentation is to show in detail the software, hardware and architectural progress needed for the critical transition to the IDR1 milestone, and how the Euclid Archive is getting ready to prepare for future, bigger releases.

The topics to be discussed will range from database technology specific for big data, parallelization of download and ingestion of products from several Euclid Data Centres across Europe and the US, as well as custom configuration of filesystem architecture to guarantee a transfer and ingestion performance of up to 70 Terabytes per day.

The Keck Observatory Archive (KOA) curates all observations acquired at the W. M. Keck Observatory. The archive is expected to grow rapidly as complex new instruments will soon be commissioned and as the expectations of archive users have expanded. In response, KOA has been implementing a new Python based VO-compliant query infrastructure. We have deployed real time ingestion of newly acquired data, and a dedicated interface for observers to manage these newly acquired data. Our poster at ADASS 2024 identified the new technologies chosen: Plotly-Dash, a low-code framework that exploits event-driven callbacks to simplify the handling of user interactions; R-tree spatial indexing to speedup spatial searches by x20; a VO-compliant TAP middleware, used already at the NASA Exoplanet Archive and the NEID archive; and mViewer, a visualization engine in the Montage Image Mosaic toolkit that is optimized for astronomy images.

These technologies will underpin new services that can be hosted on web pages or in Jupyter notebooks, and when completed, will replace the current query infrastructure. We have completed two new services now in beta release. The Data Discovery Service is a web-based dashboard which returns spatial and temporal queries of the entire archive in seconds, It supports filtering observations by keywords, previewing results in a interactive data grid, visualizing images, and offer data downloads The second is a Jupyter notebook that performs interactive Visualization of Keck Observations of protostars in the Rho Oph Dark Cloud, and uses data from CDS and IRSA as well as KOA.

This presentation demonstrates these services.

The Euclid satellite is an ESA mission that launched in July 2023. Euclid targets to observe an area of 14,000 deg^2 with two instruments, the Visible Imaging Channel (VIS) and the Near IR Spectrometer and imaging Photometer (NISP) down to VIS=24.5mag (10 sigma). Ground based imaging data in griz from surveys such as the Dark Energy Survey and Pan-STARRS complement the Euclid data to enable photo-z determination.

After the Quick Data Release 1 (Q1), that published 63.1 deg^2 of wide-survey science data publicly in March 2025, the focus has shifted to the preparation of Data Release 1 (DR1). As the first large data release of Euclid DR1 is scheduled for November 2025 internally (public release: November 2026) and has a sky coverage of about 1900 deg^2.

In this contribution we discuss the data processing for DR1, which required the coordination of several processing pipelines for various input data (VIS, NIR and external) that are run on thousands of CPU's distributed internationally over several dedicated data centers. We will put a particular focus on the generation of the the multi-wavelength catalogs of Euclid and ground based data, which is a central part of the Euclid data processing system. We show the different photometric measurements we offer to our users and show that these fulfill the tight requirements on photometric accuracy. We also present the other measurements such e.g. morphology that are included in our catalogs. We list all output products of the cataloging pipeline which go far beyond the object catalogs.

For the processing of this vast amount of data the automatic validation of the results has become a central issue, and we show the software and procedures we developed and implemented to achieve this difficult task with a minimum of interactive work.

BlueMUSE is a new blue-optimized, medium spectral resolution, panoramic integral field spectrograph being developed for ESO's VLT. While building on the legacy of the much requested MUSE instrument, its blue wavelength coverage to the atmospheric cutoff (~350 nm) will make it unique. In Galactic, extragalactic, and high-redshift domains, BlueMUSE will enable new science not possible with other instruments.

In this presentation we will show our efforts regarding science software that is developed together with the instrument to carry out simulation, reduction, and analysis of data. We briefly show how we prepare simulated raw calibration data before demonstrating our plans for comprehensive sets of raw simulated science data which will subsequently be used to test the reduction software. We will then explain the BlueSi software that is being used to simulate (reduced) BlueMUSE cubes for a number of science cases with realistic noise estimates.

We will show detailed plans for the data reduction pipeline, how its design evolved from the MUSE pipeline and which new capabilities we will implement, including spatially-resolved line-profile propagation, handling of rectangular spaxels, and exposure combination at the level of intermediate data and for datacubes.

Finally, we will present ideas for a comprehensive data analysis software environment for automated analysis of the datacubes produced by BlueMUSE.

Most traditional data reduction pipelines are on run an investigators local machine or remote machines requiring a manual touch to be executed. This approach leads to data discoverability and reproducibility issues. Additionally, observatory sites are also often remote and one of the major challenges is facilitating data transfer from site to site. Data Central's Apache NiFi system aims to combine data transfer, archival workflows and triggering pipeline execution to produce discoverable data products while managing the complexity of secure site to site data transfer. This talk will cover the implementation and design of this system for Data Central's archive of the ANU 2.3m Telescope at Siding Spring Observatory and how we have designed our system to enable scalable data transfers, data transformation and pipeline execution for incoming observations. The talk will also highlight how archival processes are integrated to improve data discoverability ensuring data products are easily located, shared and reused while also improving observability at every stage of the process. We will also discuss possible future directions including automatic notification systems (such as for transient events) and multi instrument or multi observatory abilities.

Bayesian imaging of astrophysical measurement data shares universal properties across the electromagnetic spectrum: it requires probabilistic descriptions of possible images and spectra, and instrument responses. To unify Bayesian imaging, we present the the Universal Bayesian Imaging Kit (UBIK). Currently, UBIK allows X-ray satellite data imaging for Chandra and eROSITA and soon radio interferometric imaging. UBIK is based on information field theory, it is open source, it can provide spatio-spectral image cubes, it allows the joint analysis of data from several instruments, and it is able to separate diffuse emission, point sources and extended emission regions.

Do you remember which computer were you using in 1995? What were the favourite operating systems and programming languages in the ADASS community by then? The World Wide Web was just born, and it has changed everybody’s life to a point nobody ever imagined.
When ESA decided to build the ISO Data Archive in 1995, no one expected this would be the beginning of a journey towards establishing the ESAC Science Data Centre. ESA’s Digital Library of the Universe is now covering more than 30 astronomy, planetary and heliophysics missions’ science archives and associated data discovery portal (ESASky) and close integration into data exploitation platform (ESA Datalabs).
Information Technology is changing rapidly, and data archives also went through major changes over the last three decades, as well as their associated science data management hardware and software systems, being storage, databases, graphical user interfaces, data distribution, interoperability and now data exploitation through science platforms.
So, let’s go back in time and let me drive you in a 30-year journey through science data management, sharing the Lessons Learned along the way!

The Nightly Digest (ND) is a web application that condenses Rubin observatory operations into clear visual indicators and key metrics of efficiency, downtime, and key events for a large and diverse cohort of stakeholders.
Developed by the Astronomy Data and Computing Services (ADACS) team in Australia in collaboration with the Rubin Telescope and Site Software team as part of Australia’s in-kind contribution, it is designed for senior managers, astronomers, and night staff to transform raw logs into an accessible overview that supports faster handovers and situational awareness.

The talk will cover:

• The design process we followed to transform indefinite requirements and a Jupyter notebook into a clean yet flexible and feature-rich dashboard.
• The design and UX choices made to balance operational detail with clarity, prioritising metrics and visualisations that support decision-making.
• Lessons learned from user feedback, iterative development, and collaboration with the Rubin team, shaping both features and workflow adoption.

By sharing this process, we aim to highlight the impact of collaborative design in astronomy software, showing how user-driven approaches lead to more effective and sustainable operational tools.

The Mikulski Archive for Space Telescopes (MAST) is the primary archive for the soon-to-launch Nancy Grace Roman Space Telescope. As part of that mission, a science platform known as the Roman Research Nexus (RRN) is being built to make it possible for the user community to view and analyze data at the scales Roman will produce it. As part of this effort, MAST is developing a set of tools that will allow both Roman and other MAST missions to be accessed directly from inside Jupyter notebooks on the RRN and other similar platforms.

I will describe these tools, with a particular focus on the tools to make catalog access more natural for a science user (particularly one with limited VO fleuncy), and visualization tools that smoothly transition from exposure-focused views to all-sky views. I will also describe some key principles and related lessons learned from prototypes of this work and how these prototypes and an iterative development process help yield more practically useful tools faster.

The largest astronomical catalogues now exceed the capacity of a single machine, hence multiple groups have been experimenting with clustered database solutions to be able to scale past a single machine. This BoF will provide a forum for sharing what solutions work and do not work, issues around licensing and possible solutions, and how to take existing astronomical database extensions and use them in clustered databases. This BoF also aims to start a wider collaboration to ensure that working solutions are maintained and can be effectively used by the whole community.

ADASS used to hold a regular FITS BoF. Over time, this morphed into a ‘data formats’ BoF. The early days of the FITS BoFs were a Golden Age when most data was written as FITS files that could be read and displayed by programs like SAOImage/DS9 with standardised WCS coordinates. This is reflected in the shiny ADASS Software prizes awarded to SAOImageDS9 and the CFITSIO and WCSLIB libraries. But times move on, and FITS is now an older format competing with a number of new ones, including various different schemas using the flexibility of hierarchical data formats to store data, coordinates and associated metadata with varying degrees of compatibility. Recent data formats BoFs have not reversed the trend and the ship of one format to rule them all may well be over the horizon by now.

This BoF proposal aims to focus discussion on a less ambitious question: have we lost the ability to write general-purpose programs able to get basic information out of most data files of astronomical origin? If so, does this matter? If it matters, can we do something about it? What would it take for a display program to find a data image and its coordinates in both FITS files and a number of different HDF schemas, for example? In effect, can we accept the existence of all the different data formats, but still find common ground in how they might be read?

In computer science, the field of entertainment computing covers aspects such as game design, computer graphics, human-computer-interaction, and artificial intelligence. On the first glance, the application area seems far away from astronomical research. On second thoughts, we discover many challenges that both areas encounter.

Dealing with large and complex data is a common challenge appearing in astronomy. The different branches of entertainment computing address this challenge as well. Game development requires advanced strategies to render large amounts of visual data in real time, often combined with non-trivial interaction designs. Furthermore, the incorporation of design patterns from human-computer-interaction elements increases the quality of scientific and educational software. Finally, the aspects of serious gaming and gamification empower data acquisition and labelling via crowd-sourcing. We are convinced that we can transfer these insights to the preparation, visualization and analysis of astronomical data.

In this BoF session, we will provide an overview of the entertainment computing research area. We will discuss the above mentioned techniques and speculate on how we can apply them to the astronomy domain. Moreover, we will talk about examples where astronomy or other scientific domains benefitted from an entertainment computing component. This session will be a room to exchange experiences, discuss ideas and gain inspiration for future projects.

2025 marks thirty years since the first release of WCSLIB, a software library closely linked to FITSWCS, the FITS World Coordinate System standard. From the start, WCSLIB informed the development of FITSWCS and now provides one of several practical implementations. In this talk I will describe WCSLIB's origins, its close connection with FITSWCS, and major milestones in their evolution. I conclude with some general observations on software development as I have experienced it in relation to WCSLIB over the past decades.

As astronomy data sets become larger, efficient data ingestion systems are required to ensure science ready data products are promptly available to the community. Within Data Central’s Ingestion system, bottlenecks were identified with the use of py-spy and diagnostic queries against the ingestion database. Rectifying inefficient database usage resulted in an 8 times speed up of some ingestion components. This presentation covers the identification and rectifying of these bottlenecks, some useful features and tricks to be aware of when using py-spy and the importance of proper logging and metrics of long running systems.

The SIXTE (SImulation of X-ray TElescopes) software is a general end-to-end simulation toolkit for X-ray observations, covering the full observation process from source photon generation to detector readout and the production of high-level output files. It is the official simulator for existing and future X-ray missions, such as eROSITA, NewAthena, THESEUS and AXIS.
Originally being designed as a simulator for eROSITA, the addition of new instrument and telescope types over several years have made the original codebase increasingly difficult to maintain. As such, we have refactored the code, changing languages from C to C++ and switching to a more modular software design to facilitate the implementation of new models.
This talk will highlight some of the design patterns used during the refactoring as well as its effects on maintenance, new feature development and user support one year after release of the refactored codebase.

Implementing a test process for a long living scientific software with complex dependencies and many layers of code requires a change in perspective and culture within the entire development team. But it can be done! I will present some of the challenges we have faced to test the CASA software and how we went from having a few tests to having too many tests and why this needs to be controlled.

Astronomical research increasingly depends on complex web-based user interfaces for data exploration, pipeline configuration, and visual inspection of results for quality assurance. As these interfaces grow in sophistication and user expectations rise, ensuring their reliability and usability across diverse environments becomes a critical challenge.
At Data Central we integrate automated UI testing into the development lifecycle of astronomical software tools, with a focus on both internal researcher-facing tools and web platforms. We explored, compared the use of frameworks such as Playwright and Selenium for end-to-end (E2E) testing, and how they can be integrated with CI/CD pipelines to catch regressions early and ensure consistent behaviour across browsers and platforms.
By sharing our experience, methods and tooling, we aim to encourage the broader astronomical software community to adopt automated UI testing as a standard practice, ultimately improving the reliability, maintainability, and user trust in research-critical applications.

The Cherenkov Telescope Array Observatory (CTAO) is the next-generation
very-high energy gamma-ray observatory currently under construction.
With tens of telescopes planned at two sites in both hemispheres, it
will provide a significant improvement over current instruments in
sensitivity, energy range, and resolution. CTAO will generate tens of
petabytes per year, with a first analysis of every night of observations
made available within a day at the latest.

The bulk of the CTAO data will be processed by the Data Processing and
Preservation System (DPPS), yielding high-level data ready for
elaboration by astronomers. DPPS will leverage high-throughput computing
and big data storage distributed over six data centers, orchestrated by
a suite of central services. Assembly, Integration, and Verification
(AIV) is a key element of the DPPS development lifecycle, integrating
its subsystems into a cohesive system, and verifying that it meets its
requirements and quality standards.

DPPS AIV embraces the synergy of dynamic Continuous Delivery workflow
with Systems Engineering, enabling rapid development while producing an
exhaustive project documentation trail. DPPS AIV is closely aligned with
the DPPS deployment strategy, built upon DevOps principles and
containerized cloud-native deployments. It allows fully reproducible
deployments and execution of automated Test Cases traced to DPPS Use
Cases and requirements in well-defined local development environments,
in ephemeral environments of GitLab continuous integration pipelines,
and in persistent GitOps-defined staging, pre-production, and production
deployments.

This contribution introduces the DPPS AIV strategy, processes, and
tools, highlighting the challenges of carrying out requirement
verification and quality assurance of a complex distributed data
processing and preservation system, and the solutions we adopted to
address these challenges while achieving several first releases of DPPS.
We will also discuss the challenges and opportunities created for
software quality assurance and requirement verification by growing
uptake of generative AI tools in the software development.

The data processing task of the Gaia mission is large and complex. One of its central elements is the Astrometric Global Iterative Solution (AGIS), which produces and delivers the core astrometry data products. A major challenge in the software producing Gaia’s astrometric solution is the creation of a calibration model accurate enough to capture subtle effects, which may have an impact on the quality of the solution at the micro-arcsecond level.
Among AGIS related data, a key product is the post-fit residuals. These are the differences between the observations and the predictions obtained using the AGIS source, attitude and calibration model.
This work introduces a framework for the automated analysis of residuals and the detection of anomalies that can be either indicators of non-convergence of AGIS, or problems in the calibration model. One of the methods in the framework consists of a statistical approach, which uses user-defined thresholds to identify deviations of the estimated distribution of anomalous observations with respect to the one of nominal points.
Another method, based on ML, interprets the residuals as time-series, analysing the observations across key dimensions (such as magnitude, pixel value, star color). After having identified anomalous segments, they are grouped into similar classes by means of a clustering algorithm.
Finally, a classifier is trained to distinguish between the identified anomaly classes. By analysing the classifier feature importances with the SHAP library, we can reveal which features influence the model decisions the most for each anomaly class, offering insights into the underlying patterns.
Given the absence of ground truth and the unknown characteristics of each anomaly, the framework is evaluated by comparing the results of the two methods and by manually checking randomly selected anomaly examples from each detected class.

Software tools and the algorithms underlying them have become critical to the advancement of astronomical research. The contribution of those who develop astronomical software can and should be directly linked to the discoveries made using these tools. The American Astronomical Society Journals, including the Astrophysical Journal and the Astronomical Journal, explicitly welcome articles whose purpose is to describe the design and function of software with relevance to research in astronomy and astrophysics.

In this talk, I will describe what a software paper is, some best practices on how to write one, and the editorial processes that happen behind the scenes to get your manuscript from an idea to a published paper. I will further discuss a few features of publishing software papers within the AAS journals, including Living Papers and our partnership with the Journal of Open Source Software.

In science, the lifecycle of software products is typically managed with limited resources while facing unlimited demand. Scientific software requirements are necessarily often dominated by internal project specifications and deadlines, but these internal priorities, while beneficial for the community as a whole, do not always align with the individual needs of our ultimate customers: general users. For software products to have the broadest reach, ideally the general user community should be involved in all aspects of the data lifecycle, but reality is that user expectations (and sometimes also developer expectations) need to be managed. As User Liaison for the team that develops and maintains the CASA software for radio astronomy, I will show ways for software teams to interact with general users, even when facing limited resources for user support. I will discuss how realistic support and involvement of users and user groups can benefit both the user community and software development teams.

The Astro Data Lab science platform recently marked eight years of operational service, a significant milestone in the fast-evolving domains of big data research, software development, and computational infrastructure. Initially designed to host and analyze data from the Dark Energy Survey, Data Lab has expanded its scope far beyond these (modest) first goals. Now integral to the success of all-sky surveys such as Rubin's LSST, Euclid, and Roman, Data Lab supports a growing community of currently over 4,000 registered users.

This talk will briefly outline the evolution of Data Lab, emphasizing its ability to scale alongside both increasing data volumes and user demands, while navigating the challenges of uncertain and diminishing budgets. Key recent developments will be highlighted, including the launch of a new integrated Web Portal, VO Registry integration, and the introduction of Apache Airflow for managing data ingestion workflows. Additional advancements include leveraging Rubin’s Felis for metadata handling, incorporating Gemini's DRAGONS pipeline for data reduction, and conducting our first annual user survey to gauge interest in specific directions of developement.

Looking ahead, I will present plans for near-future enhancements, such as enabling GPU support for machine learning applications, improving cross-platform query capabilities, optimizing query performance to meet the demands of upcoming large datasets, and leveraging LINCC's HATS & LSDB to power large-scale cross-matching capabilities. By maintaining a focus on scalability, user needs, and innovation, Data Lab aims to remain a critical tool for the next decade of astronomical research and discovery.

LOFAR is a high throughput data facility facing several non-trivial technical
challenges in data processing and storage. Since start of science operations, LOFAR has accumulated over 50 petabytes of data in its science data archive. Following a major upgrade of the instrument, it is expected that over the course of the next five years of operations the archive will grow to well over 100 petabytes of science data. Other astronomical instruments that will start and ramp up over the coming years will result in similar (RUBIN) or significantly larger (SKA) data volumes. There are viable technical solutions to scale storage to the required capacity, but sustainability considerations (storage cost, network and processing capacity, data access and re-use by a wide community) are increasingly putting constraints on acceptable volumes of data delivery and long-term storage. In this contribution we will present a spectrum of measures that are applied to data systems and operations to address the sustainability challenges for LOFAR. The measures include the application of (lossy and lossless) data compression, retirement of archived data without legacy value, the application of data retirement policies for raw and intermediate level data, the adoption of full lifecycle data management plans for science projects, engaging the community to realize more efficient data processing, data storage, as well as scaling out to newly available infrastructure. We will discuss the relation to FAIR practices, impact on scientific legacy value, and concerns in, and needs from, the science community.

Astronomy’s data lifecycle is no longer defined solely by storage and processing technologies, but equally by the social and organizational structures that enable long-term usability, interoperability, and trust. Within ESA’s ESAC Science Data Centre (ESDC), we face these challenges at scale through missions such as Gaia and Euclid, and in the development of the Euclid Data Space (EDS) project. These projects illustrate how technical choices—ranging from distributed databases and data access protocols to standards adoption—are inseparable from community processes, governance, and collaboration across international teams.
This talk will share some lessons learned from designing and operating large-scale data systems in support of ESA’s Science Programme for over 25 years, focusing on three dimensions: (1) the technical evolution required to support massive, heterogeneous datasets across their lifecycle; (2) the social aspects of aligning diverse stakeholders, from mission teams to external research communities; and (3) the interplay between standards (e.g., VO protocols) and innovation in shaping sustainable service ecosystems. By highlighting concrete experiences from ESA’s data archives and science platforms, I will discuss how technical and social aspects together determine the success of data lifecycle management, and how these lessons may inform future astronomical and interdisciplinary data infrastructures.

It has recently become possible to numerically simulate large, representative volumes of the Universe. These cosmological (magneto)hydrodynamical simulations solve for the coupled evolution of gas, dark matter, stars, and supermassive black holes interacting via the coupled equations of self-gravity and fluid dynamics, all within the context of an expanding spacetime.

The IllustrisTNG simulations exemplify the current state-of-the-art in this context. They simultaneously resolve tens of thousands, to millions, of individual galaxies - with properties and characteristics in broad agreement with observational data of real galaxy populations. This enables many theoretical studies on galaxy formation and evolution, as well as large-scale structure and cosmology.

I will give a tour of the TNG simulations, touching on a few scientific applications and novel insights. I will showcase the information content and breadth of a virtual Universe, and describe our efforts to publicly release these large datasets through a powerful, online science platform (www.tng-project.org) that is democratizing access to cosmological simulations and paving the way for Open Science and Open Data in theoretical astrophysics.

The Science Archives, managed by the ESAC Science Data Centre (ESDC), continue to serve as the definitive repositories for ESA mission data, safeguarding scientific knowledge and ensuring global accessibility for the research community.

This year marks a major milestone: the publication of the first operational release of the Integral Science Legacy Archive (ISLA), now established as the official archive for INTEGRAL mission data. Developed in close collaboration with mission scientists, ISLA delivers a modern, user-centric experience, transitioning from conventional data access to specialized science portals tailored for high-energy astrophysics.

ISLA introduces a visually engaging and highly interactive interface, powered by cutting-edge web frameworks such as Angular and a suite of reusable components, together with a robust backend built on Spring Boot. The integration of the ESASky portal, via its API, enables seamless exploration of high-energy sources, exposure maps, GRB catalogs, and related publications, as well as dedicated sections for community products and catalog exploration.

ISLA also provides a direct link from its data search interface to the data exploitation platform, ESA Datalabs, where INTEGRAL data can be processed with OSA, without the need to install software or download data.

Importantly, ISLA has served as a precursor for many of the concepts and reusable widgets now adopted across other ESA Science Archives, while also benefiting from shared components developed in parallel by other teams. This collaborative and modular approach ensures a smooth, intuitive, and user-friendly experience, accelerates the integration of new features, and supports rapid development and future scalability.

This operational release demonstrates ESA’s commitment to technological innovation, robust software development, and international collaboration in the realm of astronomical data access and discovery.

Image generation is an important step in the modern astronomy data analysis workflow. It provides quick-look diagnostics on raw data or during the data reduction stages, enabling visual identification or classification of sources and features, and the presentation of the data to the larger scientific community. Traditionally, these images are created from stacking three (or more) scaled single band images as colour channels, in a standard RGB (sRGB) image. This technique belies the complexities of colour science and perception, where even the simplest of images require complex transformations between the input gamut from the camera sensors to the final output sRGB. In this talk, we present a method of combining any number of astronomical single band images into sRGB images using the Hue-Saturation-Value (HSV) colour space. This method produces output images that are more reflective of noise properties of the underlying data, better representative of details and features, more intuitive, and more visually appealing.

Twitter spent ten years as the de facto online platform for astronomy networking and outreach. However, semi-recent events have seen it devolve into a politicized and ineffective platform for science communication and networking. The loss of Twitter has shown how fleeting online spaces can be. It begs the question: can we do better, or are astronomers doomed to always have their online homes tied to the whims of a billionaire?

In this talk, I will present The Astrosky Ecosystem: An open source project to build independent social media infrastructure with the AT Protocol, the social media protocol that powers Bluesky (a prominent Twitter challenger). I will start by presenting a brief outline of the AT Protocol, showing how the technology to build and run independent and interoperable social media infrastructure now exists, bringing incredible new possibilities.

Next, I will present our project's previous two years of progress. We began in 2023 as a small, indie project to move astronomers to a new platform while running a 'custom feed' of astronomy posts. Two years on, our project now hosts over a dozen custom feeds, viewed over 2 million times per month by over 20,000 unique users. Some of our nearly 2000 posters include large astronomy organizations like ESA, ESO, the AAS, and Rubin. Our project also recently progressed in its sustainability, adding new developer and moderation teams in addition to working towards fiscal sustainability through crowdfunding.

I will discuss our future plans to start doing secure, EU-based AT Protocol/Bluesky account hosting - meaning that astronomers and astronomy organizations will never need to change platform again. Finally, I will discuss how other astronomy software projects could integrate with our ecosystem, given the almost-limitless extensibility of the AT Protocol, in addition to sharing lessons learnt in our quest for >99.9% uptime.

The exponential growth in size and complexity of astronomical datasets from space missions presents significant computational and infrastructural challenges. ESA’s Euclid mission has already produced petabytes (PB) of processed data and is projected to produce 30 PB over its operational lifetime. Analysing and processing data on this scale requires specialised infrastructure and toolchains.

ESA has developed a science platform, ESA Datalabs, which provides essential infrastructure to access and analyse data from missions such as the Hubble Space Telescope, James Webb Space Telescope, Gaia, and Euclid. Leveraging software like JupyterLab, users can interact with mission data without downloading it. The platform fosters collaborative science by enabling direct connection to ESA archives and shared computational workspaces, facilitating creation and deployment of user-built applications and analysis pipelines, and ensuring accessibility to a broad research community.

In this presentation, we outline the need for science platforms in the Big Data era, the motivation behind ESA Datalabs, its key functionalities, and its role in addressing challenges such as scalable data processing, infrastructure development, and reproducible research. We demonstrate how integration of archives, visualisation tools and science platform into a unified portal for Euclid, Euclid Data Space, creates a powerful, single-entry point for the mission’s scientific community.

We showcase recent use cases, including ESA Datalabs’ role in the first public Euclid quick data release and the first large internal data release. Additionally, we highlight how the platform supports analysis of data stored in ESA science archives using data mining and machine learning techniques, for use cases such as large-scale classification of galaxies and identification of anomalies.

Our discussion highlights how science platforms can maximise the scientific potential of current and future space missions and shape the future of data-intensive space science.

The Canadian Advanced Network for Astronomical Research (CANFAR) science platform has reached its 1.0 release, marking a significant evolution in its capacity to address the challenges of astronomical analysis in the era of large surveys and facilities. In this presentation, we will detail the evolution of the CANFAR architecture, which provides researchers with a suite of services, including browser-accessible desktops, notebooks, and development and analysis tools, built upon a Kubernetes-managed cloud backend.

We will focus on recent major technical developments, including the deployment of a cloud-native batch processing system using Kueue, which enables the efficient execution of large-scale, automated analysis pipelines, and promotes the transition from interactive, exploratory work to batch computation. We will also detail the release of a new CANFAR client and software API, facilitating programmatic access for complex workflows that can span networks of CANFAR deployments. Drawing from years of operational experience supporting a diverse user base—from individual researchers to large collaborations —we will share insights and metrics related to platform monitoring, reliability, and user support strategies.

Finally, we will outline our roadmap, which prioritizes support for data-intensive projects driven by SRCNet while also accommodating workflows from Euclid, LSST, ALMA or CHIME-FRB. This talk will demonstrate how these integrated components establish CANFAR as a robust, scalable, and accessible platform for modern astronomy.

The discovery of transient phenomena—such as Gamma-Ray Bursts (GRBs), Fast Radio Bursts (FRBs), stellar flares, novae, and supernovae—together with the emergence of new cosmic messengers like high-energy neutrinos and gravitational waves, has revolutionized astrophysics in recent years. To fully exploit the scientific potential of multi-messenger and multi-wavelength follow-up observations, as well as serendipitous detections, researchers need a tool capable of rapidly compiling and contextualizing essential information for every new event. We present Astro-COLIBRI, an advanced platform designed to meet this challenge.

Astro-COLIBRI is a comprehensive platform that combines a public RESTful API, real-time databases, a cloud-based alert system, and user-friendly interfaces including a website and mobile apps for iOS and Android. It ingests alerts from multiple sources in real time, applies user-defined filters, and situates each event within its multi-messenger and multi-wavelength context. The platform provides clear data visualization, concise summaries of key event properties, and evaluations of observing conditions across a wide network of observatories worldwide.

In this contribution, we will present the architecture of Astro-COLIBRI, from the data pipelines that manage real-time alert ingestion and processing to the design of the RESTful API, which enables seamless integration with other astronomical software and services. We will illustrate how this framework supports applications in high-energy time-domain astrophysics through concrete use cases, thereby establishing Astro-COLIBRI as a key enabling tool for the multi-messenger community.

The GAIA Datamining Platform provides interactive, JupyterHub-based access to the GAIA Data Release 3 dataset, which comprises 7TB of data.

The GAIA Data Release 4 dataset is expected to be in excess of 600TB.
We describe our progress in evolving the GAIA Data Mining Platform to a modern, kubernetes-based, platform-independent deployment, named Astroflow, adding dask functionality to existing large scale Spark analytical processing.

In conjunction with the closely related SPACIOUS project, we report our findings and successes in deploying the platform to both on-premise (OpenStack) and commercial (Google) cloud platforms.

We outline our plans to incorporate Apache Iceberg into our architecture to efficiently scale up to and support the forthcoming GAIA DR4 release, and to use the data lake model to support and combine future multiple data sources for large scale analytical processing and data mining in our interactive environment

The James Webb Space Telescope is producing a firehose of extragalactic imaging data through its diversity of legacy programs. Community organized initiatives, such as the Dawn JWST Archive, have come to fill the gap between archive products to uniformly-reduced data that enable large-scale exploration and analysis. These programs are catalyzing further initiatives to generate value-added catalogs of inferred parameters, such as those from SED fitting, morphology extraction, and machine learning algorithms. Using and comparing these catalogues will become increasingly challenging as more data is made available and new parameters are created.

Through the J-HIVE initiative, we have created a platform to generate purpose-specific catalogs that combine the community generated data products in a versionable and traceable manner. The platform also generates schema to enable access to all of this data in a discoverable and documented manner. The Python/JSON based system can be easily modified to add in new catalogues or datasets as they are available. It also performs simple transformations and filters on the data, which can be added to as needed. The output catalogue can then be easily pushed to visualization mechanisms for exploration and analysis. In this talk, we will present how the platform was developed and the benefits it provides to researchers using JWST data now and in the future.

Modern astronomical surveys such as HST, JWST, Euclid, and LSST are generating petabyte-scale imaging archives across multiple wavelengths and epochs. Traditional image retrieval methods, which are based solely on metadata, such as sky position, filter, or exposure time- are insufficient to identify objects with similar visual or physical characteristics. To enable efficient discovery in these massive datasets, content-based search methods that operate directly on image representations are required. Early studies in this area relied on handcrafted features or supervised convolutional neural networks (CNNs), which require extensive labelled data and thus limit scalability. More recent advances employ unsupervised or self-supervised learning to extract intrinsic representations of astronomical images. In particular, Teimoorinia et al. (2021) introduced a fully unsupervised two-stage framework that combines self-organizing maps (SOMs) and CNN autoencoders for image modelling and similarity search, demonstrating the feasibility of data-driven discovery without labels. Building on this foundation, we present a practical and scalable framework that integrates state-of-the-art self-supervised learning into the Canadian Astronomy Data Center’s Common Archive Observation Model (CAOM). Using techniques such as variance–invariance–covariance regularization (VICReg) and deep embedded clustering (DEC), our system generates high-dimensional embeddings that capture both the visual morphology and physical properties of astronomical sources. We demonstrate that these learned representations enable efficient, scientifically meaningful image retrieval within large, heterogeneous archives, paving the way for next-generation, content-based discovery tools in data center environments.

With the new generations of large-scale surveys, we are faced with an avalanche of data that are no longer “images” but “cubes”, and whose third dimension is either temporal or spectral. In this new area, traditional hierarchical science platform visualisation methods must evolve to exploit this third dimension.
Building on the Hierarchical Progressive Survey method – endorsed by the IVOA and standardized in 2017 – we present new results from a study and implementation of extending this visualisation method to a third physical dimension: the HiPS3D solution. We will explain the principles behind this generalization, the scientific and technical choices, and we will present the new prototype tools: Hipsgen for the generation and Aladin (Desktop & Lite) for the visualisation and integration in various science platform solutions. We will report on the first statistics obtained on test cube surveys: GalfaHI, SKA Challenge 2, MUSE excerpt, LGLSHI, DHIGLS. Finally, we will give an overview of the prospects for a large-scale implementation of this method for existing cubic surveys such as ALMA, or future ones such as SKA.

Arrays of Cherenkov telescopes detect ultra-short (nanosecond) flashes of blue light produced when high-energy gamma rays hit Earth’s atmosphere, triggering particle cascades. The upcoming Cherenkov Telescope Array Observatory (CTAO) will generate hundreds of petabytes of data annually, requiring extensive atmospheric monitoring and rich metadata to reconstruct event lists, images, spectra, and light curves. These “software telescopes” depend on complex pipelines and statistical models, not just the instrument and acquisition settings, but also the full processing configuration and detailed provenance, making precise data and processing descriptions critical.
We developed an open-source web platform that enables data search, visualisation and quick-look analysis of Cherenkov Astronomy data products. This platform manages all the necessary metadata to support search criteria adapted to the specific nature of Cherenkov data. Built on the Virtual Observatory framework, the platform implements the IVOA ObsCore standard to describe the spatial, temporal, and spectral characteristics of observations. Additionally, we are contributing to a High-Energy extension of ObsCore, enhancing its capacity to describe high-energy Cherenkov data.
The platform couples a FastAPI micro-service layer to a React front-end, with authentication based on the OpenID Connect standard. The metadata is exposed via an IVOA TAP server receiving IVOA ADQL queries. The matched products are returned in a sortable table, plotted on an Aladin Lite sky map, and through interactive Plotly charts of temporal and energy coverage. Signed-in users benefit from a reproducible search workflow: all ADQL requests are stored in a PostgreSQL database and can be replayed to regenerate identical results. Selected records can be collected into persistent “baskets” — named lists preserved across sessions for reuse.
Open-source licence and strict adherence to IVOA standards ensure alignment of the platform with FAIR and open-science principles.

The amounts of raw data in next-generation observatories, such as the Square Kilometre Array Observatory (SKAO), will be so large that they cannot be archived in their entirety, but must be significantly reduced. This is well known in high-energy physics, particularly at the Large Hadron Collider (LHC), where the data streams captured by the detectors are reduced by several orders of magnitude during the data acquisition phase using sophisticated real-time algorithms.

At the LHC, proton collisions are repeated over and over again under the same initial conditions, which means that even rare events can be observed multiple times if the observation period is long enough. In astronomy, on the other hand, the initial conditions cannot be influenced. Furthermore, the experimental boundary conditions can change in unpredictable ways. Consequently, the established workflows in high-energy physics must be expanded to allow realtime optimization of telescope control parameters. Several years ago, Michael Kramer, Stefan Wagner, and the author proposed the “Dynamic Life Cycle Model.” A characteristic feature of the model is the introduction of two feedback loops:
- from the data centers next to the observatories and
- from the archives in data centers distributed worldwide
to the telescopes in order to control them in realtime or near-realtime.

The presentation provides a brief introduction to the model, followed by a discussion of selected computational challenges and an overview of the current status and future work.

The latest SDSS Data Release 19 comes with a new suite of tools for helping astronomers and students visualize and analyze the vast richness of this dataset. In this demo we will showcase several of these tools, including the Zora web application - a modern and reactive interface for searching SDSS data, exploring observed target metadata, and visualizing or accessing spectral data - and the new Navigate and SQLxMatch tools on the SkyServer web portal - used respectively for browsing through SDSS source catalogs, footprints, and geometries on an interactive sky map, and for running on-the-fly cross-matches of SDSS objects against more than 50 other astronomical catalogs. Lastly, we will illustrate some DR19 science use cases with Jupyter Notebook tutorials in SciServer Compute.

Alongside groundbreaking new hardware capabilities for existing and future facilities, we are entering a new era of optimisation and efficiency that will be driven by software innovation. The dramatic rise of practical artificial intelligence in recent years carries significant implications for how we effectively operate our future facilities, a trend that has already been in place for years with the general shift towards increasingly automated and remote facility operations. And yet, it is also clear that the best outcomes will come from the considered application of the most suitable approach for a given scenario, ranging across a spectrum from highly human-driven to completely hands-off.

Considering specific case studies of the Australian SKA Pathfinder (ASKAP) and the Deep Synoptic Array (DSA-2000), I will outline the approach taken in moving towards autonomous operations, especially in the realm of scheduling science observations for surveys. Our experience to date shows that telescope operations can be highly autonomous even with a relatively low level of AI, freeing up human resources and effort while increasing the overall efficiency and success of scheduling. I will also explore logistical, technical and sociological elements that we have identified as part of the process of increasing operational autonomy, with the goal of conveying our lessons learned for application to other contexts.

The true capability and potential of current and next-generation facilities will only be achieved if we seek to adapt and transform the ways in which we operate our facilities, treating the evolution of our operational models as a core and critical part of the technical innovation that will broaden and deepen the parameter space we can explore.

Advanced science platforms must handle large data volumes, complex workflows, and collaborations that span multiple disciplines and partners. While scientific questions may differ between fields, the challenges of building reliable and reproducible data-driven research infrastructures are very similar.

This talk demonstrates how a geophysics-oriented science platform integrates well known open-source services to support the full lifecycle of data-intensive research. For monitoring and observability, we employ Prometheus and Grafana to collect and visualise performance metrics, alongside Fluentd and Kibana for log aggregation and analysis. These tools provide real-time insights into the health and efficiency of the platform and scientific workflows. They are also generating metrics and usage reports that satisfy accountability requirements of funding agencies.

Beyond infrastructure monitoring, collaboration and reproducibility are enabled through Gitea for distributed version control and lightweight code hosting, allowing users to develop and run custom applications on our platform. To support the increasing role of machine learning in scientific workflows, we integrate tools such as Weights & Biases for experiment tracking and model management, and Label Studio for collaborative dataset curation and annotation. The platform supports execution on remote workers and HPC clusters, enabling scalable computation and flexible integration with existing research infrastructures.

Our experience demonstrates that combining these tools into a modular ecosystem creates science platforms that are scalable, funding-compliant, and cultivate reproducibility. Key lessons stress the need to balance automation with researcher control, design platforms that enable cross-disciplinary collaboration, and adopt testing frameworks to ensure platform reliability. Although developed for geophysics, this approach is directly transferable to astrophysics and other data-intensive fields. The talk will highlight practical insights on building platforms that support scientific workflows and collaboration in the big data era.

Whilst the Research Data and Software (RDS) team at the AAO has more than doubled in size, the nature of our funding is such that there is very limited scope to hire additional infrastructure support personnel. Taking inspiration from the Southern African Large Telescope (SALT) Astronomy Operations user support model, we have introduced the concept of a "DevOps Roster" to ameliorate the load of growing operational tasks within our group. The roster aims to spread the support load over the larger team, while simultaneously upskilling the team with an emphasis on documenting gaps in our processes. The introduction of the roster has also generated interesting discussions about the long term funding and maintenance of operational tasks. In this talk, I will cover what has worked, possible pitfalls of such a model, and how other groups could benefit from its application.

Next year the Fifth Generation of the Sloan Digital Sky Surveys (SDSS-V) will launch the 20th SDSS public data release, 25 years after its very first early data release appeared on-line in 2001. Much has changed in SDSS over that time: telescopes have been added, new instruments have been built, and old instruments have been retired. What has remained however is the commitment to make SDSS data publicly available and accessible, which over the years has led to over 10,000 published papers that use SDSS data, and various educational and outreach projects that incorporate SDSS data. Behind all that lies a data infrastructure that constantly adapts to the needs of the SDSS collaboration and the astronomical community. This data infrastructure encompasses not just archive and catalog servers, but also visualisation web apps and computing platforms, that help both experienced and starting astronomers to access and explore the SDSS data.

In this talk I will present changes made and lessons learned for the public data releases of the SDSS. I will provide an overview on how data releases are created, what they consist of, and how they are managed. I will also discuss how we involve SDSS collaboration members in the data release process, as they create value-added catalogs, and contribute to documentation and tutorials. I will also discuss some very recent changes in the SDSS data systems to support the transition from fibre plug plates to robotic focal plane systems (FPS) in SDSS-V, and the effect that this has had on our data releases.

The advances of generative AI are staggering and progress is expected to continue at high speed. In the near future astronomers will likely be able to use AI agents to accompany them in the entire process from proposal preparation to archival search, data analysis and publication. How to leverage the advantages for astronomy? How to mitigate the risks?

In this BoF we try to look into the consequences of this evolution:

What does this evolution mean for astronomers and their daily work (user-experience)?
What does this evolution mean for observatories and the tools and interfaces they provide (e.g. gui vs. programmatic)?
What should observatories do to become 'AI ready'?
What should observatories do to help astronomers?

Already in May this year - and thus just 2.5 years after the first release of ChatGPT - a generative AI has demonstrated fully autonomous PhD-level research capability in computer science (https://www.intology.ai/blog/zochi-acl). It is conceivable, and maybe even likely, that in some years from now AI will be powerful enough to do autonomous research in astronomy.

In this BoF we also try to anticipate this scenario and ask ourselves:

What would this evolution mean for astronomers?
What would this evolution mean for observatories?
What should be done now?

ADASS POC conducted a community survey. We would like to use that to start the conversation on the future of ADASS.

By combining data from different messengers (electromagnetic radiation, gravitational waves, neutrinos, cosmic rays, ...), one can gain a better understanding of the physics in the universe. A milestone was the merger of a binary neutron star (2017) seen in gravitational waves by LIGO/Virgo, followed by gamma-ray burst, optical/infrared kilonova, X-ray, and radio counterparts.

There are important challenges, for example:
- many messengers are hard to detect,
- transients often evolve quickly, i.e. a rapid observation of follow-ups is needed,
- the variety, the velocity, and the volume of data is large in general.

The German Center for Astrophysics (DZA) will provide archives with data from observatories and telescopes around the world. One of the main tasks will be to make it easier for different communities (EM, GW, EM, neutrino, cosmic ray) to cross-match data. This requires standardized data Formats, open data, and uniform access.

This session will provide a brief overview of the topic, followed by presentations from representatives of a few archives on their workflows and possible strategies for facilitating access, including job processing, federated authentication and authorization infrastructures (AAI) as well as accounting.

The aim of the subsequent discussion is to better understand the challenges involved in implementing multi-messaging and the needs of the various messenger communities, ultimately enabling the DZA to build a suitable infrastructure that best meets the communities' requirements.

In 2022 I transitioned from research in computational astrophysics to healthcare system modelling. I found most of the tools, techniques, and skills acquired during my work in astrophysics to be readily translatable to the new field, and the collaboration with experts from different backgrounds an extremely positive and stimulating experience for all those involved. In this talk, I will discuss my personal experience of this transition, and why I became a believer in multidisciplinary collaborations. I will also discuss my current work leveraging generative AI to optimise the performance of these simulations.

Current and upcoming astronomical surveys (e.g., SKA, LSST, Euclid, or even ALMA) present a significant data processing challenge, with data volumes that overwhelm traditional, single-node analysis workflows. Many of our community's essential analysis tools are built within the Python ecosystem, but they often struggle to scale to the high-performance computing (HPC) resources required for these future datasets.

Heat (GitHub) is a Python library designed to bridge this gap. As a cross-disciplinary tool developed within Germany's Helmholtz Association, it is already employed in applications ranging from climate modeling and neuroscience to aerospace engineering. Heat provides a distributed, NumPy-like array library that enables scientists to scale existing data analysis codes with minimal modification. This allows a seamless transition from a laptop to thousands of CPU cores or GPUs on HPC clusters. By using a data-parallel architecture, MPI for communication, and PyTorch as a backend, Heat is optimized for efficient execution on heterogeneous hardware.

In this talk, we will introduce the core concepts of Heat, show its potential as an HPC backend for the Python array ecosystem, and demonstrate the first few applications in astronomy.

The quantities of data produced by next generation instruments such as the SKA, the DSA2000 and the ngVLA require new software ecosystems to convert observational data into science ready data products.

Traditionally, such scales of data and compute are solved using traditional HPC software and infrastructure. While this approach is still relevant going forward, the advent of (1) ubiquitous cloud compute (2) the expansion of the scientific Python (PyData) ecosystem and (3) the concurrent explosion of data processing techniques used in Machine Learning and AI, has pioneered a complementary approach that sacrifices some performance for the flexibility to rapidly prototype and develop distributed processing pipelines by embedding “experts in the loop”.

The Pangeo project has pioneered this approach in the geosciences domain, building cloud-based pipelines built on a software stack of Xarray for dataset representation, Zarr for distributed storage and Dask, NumPy and SciPy for compute. It has also seen adoption within the Radio Astronomy community in software ecosystems such as Africanus and the adoption of these technologies within the Measurement Set v4 Working Group xradio prototype.

This talk will discuss how, through the adoption of these new interfaces and formats, Radio Astronomy stands on the cusp of a new wave of software built upon Open Source technologies. It will demonstrate how, through the use of the Xarray interface, data scientists and radio astronomers will be able to manipulate large scale datasets to produce science. Additionally, it will discuss pertinent developments in the broader open-source community relevant to Radio Astronomy going forward.

After about 15 years of operations, much of the software and hardware of the LOFAR radio telescope is upgraded to deliver LOFAR 2.0. Part of this upgrade is a renewed software architecture.

LOFAR 1 consisted mostly of in-house software products and protocols for its functionality, information exchange, and service management. These codes ran on open source OSes and libraries.

LOFAR 2 instead leverages open-source third-party products as the basis of its architecture. We connect open-source tooling such as Nomad, Consul, Docker, Prometheus, Grafana, MinIO, Tango Controls to provide a powerful base infrastructure and service management system for our telescope management and information flows. These tools provide us with high-level functionality out of the box, often a natural integration between them, and a powerful basis for flexible development.

In this talk we present an overview of this architecture, and some of the lessons from transitioning to it. What we learned when taking other departments along in the transition. How it affected the work flows of operations and ICT, and the requirements for the hardware engineers. Both were handed a fundamentally different set of tools to work with. And about the effects on the design and security aspects of the underlying IT infrastructure, as software abstraction layers make that process more opaque.

XMMGPT is a dual-purpose project which aims to serve as a unique access point to aid astronomers in their research with XMM-Newton data, and as an exploration of language models and AI systems within European Space Agency (ESA) workflows. 

The system is comprised of 4 main parts, a heavily customized Agentic Retrieval Augmented Generation (RAG) pipeline, a visibility checker tool, a long-term light curve generator, and an autonomous agent to execute data processing tasks from the Scientific Analysis System (SAS). 

The RAG system is built from SAS technical documentation with contextual embeddings, naive knowledge graphs, a hybrid fine-tuned text and vector similarity search, as well as an agent that routes user queries to appropriate documentation types. 

The visibility checker tool transforms natural language queries into API calls to a visibility server which implements the IVOA ObjObsSAP protocol while the long-term light curve generator transforms natural language queries into API calls to gather flux data and generate a representative plot.

The autonomous SAS Agent takes natural language queries and goes through a series of sequential steps in order to send API calls to RISA (Remote Interface to SAS Analysis) or ULS (Upper Limit Server) and retrieve valid scientific products.

Data privacy and cost concerns have led to the project being developed on relatively small local hardware and a constant challenge has been to find state-of-the-art, smaller footprint, language models which give good performance.

Approaches to autonomous SAS code creation and execution are currently being explored while the project is steadily being updated to take advantage of evolving industry standards.

SciServer is a high-impact, highly successful Science Platform with a well-developed existing code base; an established user community; and demonstrated impact on scientific discovery, research, and education. SciServer has demonstrated a transformational impact in astronomy, providing collaborative features such as groups and file sharing, and free computational resources to access large datasets in both observational (e.g. SDSS, HEASARC) and theoretical (e.g. Millennium, Indra simulations) astrophysics.

Since its inception, SciServer has grown beyond astronomy to a platform that creates significant impact in multiple science domains including – but not limited to - materials science and engineering; turbulence; oceanography; and precision medicine and genomics. SciServer has also been used in a classroom setting across a number of these disciplines. The platform has also been successfully installed in other institutes, namely Brookhaven National Laboratory to provide simple access to GPU resources for computational research needs, and at the Max Planck Institute for Extraterrestrial Physics to provide access to eRosita data.

More recently, SciServer has been selected by an NFS sustainability grant to bring the platform code and governance to the open source community. We have collected the platform code in a single github repository that is now public, updating codes for style, documentation and consolidating the automated build and release processes. SciServer can now be installed by anyone with a Kubernetes cluster using Helm, either within their data center or in a public cloud environment.

In this talk we will provide an introduction to SciServer and its impact in a variety of science disciplines, focusing on astronomy. We will talk about the underlying architecture and deployment mechanisms, about our journey to open source release and the opportunities an Open SciServer provides to the astronomy community and beyond, and finally how you can contribute!