Speaker
Description
The Astro Data Lab science platform recently marked eight years of operational service, a significant milestone in the fast-evolving domains of big data research, software development, and computational infrastructure. Initially designed to host and analyze data from the Dark Energy Survey, Data Lab has expanded its scope far beyond these (modest) first goals. Now integral to the success of all-sky surveys such as Rubin's LSST, Euclid, and Roman, Data Lab supports a growing community of currently over 4,000 registered users.
This talk will briefly outline the evolution of Data Lab, emphasizing its ability to scale alongside both increasing data volumes and user demands, while navigating the challenges of uncertain and diminishing budgets. Key recent developments will be highlighted, including the launch of a new integrated Web Portal, VO Registry integration, and the introduction of Apache Airflow for managing data ingestion workflows. Additional advancements include leveraging Rubin’s Felis for metadata handling, incorporating Gemini's DRAGONS pipeline for data reduction, and conducting our first annual user survey to gauge interest in specific directions of developement.
Looking ahead, I will present plans for near-future enhancements, such as enabling GPU support for machine learning applications, improving cross-platform query capabilities, optimizing query performance to meet the demands of upcoming large datasets, and leveraging LINCC's HATS & LSDB to power large-scale cross-matching capabilities. By maintaining a focus on scalability, user needs, and innovation, Data Lab aims to remain a critical tool for the next decade of astronomical research and discovery.
| Affiliation of the submitter | NSF NOIRLab |
|---|---|
| Attendance | in-person |