Speaker
Description
The emerging technology of Data Science platforms addresses the need to analyze Big Data directly where it is stored, because moving large volumes of data is nearly impossible. Typical platforms, like Pangeo, ESA DataLab, and SciServer are complex cloud-based systems, running on large clusters, providing hundreds of users access to petabyte-scale astronomical data archives. They use flexible orchestration of multiple containers to allow script-driven data processing and complex database queries, as well as interactive exploratory analysis via a web GUI.
Similar principles can be applied to smaller, task-specific infrastructures, particularly for machine learning experiments that demand continuous user interaction. A representative example is the active learning–based classification of millions of stellar spectra from large surveys. In such cases, iterative training requires annotators to label candidate spectra across multiple cycles while simultaneously accessing complementary data such as catalog metadata, images, and spectra from other surveys.
To address this need, we introduce the ml-job-manager, a dedicated multi-tier cloud platform designed for orchestrating parallel experiments with human-in-the-loop active deep learning on several million spectra from the LAMOST archives. The system adopts modern DevOps strategies, leveraging a microservices backend with RESTful asynchronous job control inspired by the IVOA UWS protocol. The entire environment in several containers can be rapidly deployed using Docker Compose, ensuring reproducibility and ease of installation.
| Affiliation of the submitter | Astronomical Institute of the Czech Academy of Sciences |
|---|---|
| Attendance | in-person |
