% This is the ADASS_template.tex LaTeX file, 24 Oct 2025.
% It is based on the ASP general author template file, but modified to reflect the specific
% requirements of the ADASS proceedings.

% This is the aspauthor.tex LaTeX file
% Copyright 2021, Astronomical Society of the Pacific Conference Series
% Revision:  29 November 2021

% To compile, at the command line positioned at this folder, type:
% pdflatex ADASS_template
% pdflatex ADASS_template
% This will create a file called ADASS_template.pdf

\documentclass[11pt,twoside]{article}

% Do NOT use ANY packages other than asp2021. 
\usepackage{asp2021}

\aspSuppressVolSlug
\resetcounters

% References must all use BibTeX entries in a .bibfile.
% References must be cited in the text using \citet{} or \citep{}.
% Do not use \cite{}.
% See ManuscriptInstructions.pdf for more details
\bibliographystyle{asp2021}

% The ``markboth'' line sets up the running heads for the paper.
% 1 author: "Surname"
% 2 authors: "Surname1 and Surname2"
% 3 authors: "Surname1, Surname2, and Surname3"
% >3 authors: "Surname1 et al."
% Replace ``Short Title'' with the actual paper title, shortened if necessary.
% Use mixed case type for the shortened title
% Ensure shortened title does not cause an overfull hbox LaTeX error
% See ASPmanual2010.pdf 2.1.4  and ManuscriptInstructions.pdf for more details
\markboth{Moseley et al.}{New Plotly-Dash Based Query Infrastructure}

\begin{document}

\title{A New Plotly-Dash-based Query Infrastructure for the Keck Observatory Archive}

% Note the position of the comma between the author name and the 
% affiliation number.
% Authors surnames should come after first names or initials, eg John Smith, or J. Smith.
% Author names should be separated by commas.
% The final author should be preceded by "and".
% Affiliations should not be repeated across multiple \affil commands. If several
% authors share an affiliation this should be in a single \affil which can then
% be referenced for several author names. If only one affiliation, no footnotes are needed.
% See ManuscriptInstructions.pdf and ASP's manual2010.pdf 3.1.4 for more details
\author{R.~Moseley,$^1$ G.~Bruce~Berriman,$^1$ Christopher~R.~Gelino,$^1$ 
 John~C.~Good,$^1$ Meca~Lynn,$^1$, Melanie~Swain,$^1$ and Toba~Oluyide$^1$}
\affil{$^1$Caltech/IPAC-NExScI,~Pasadena,~CA~91125,~USA; \\
\email{moseley@ipac.caltech.edu}}

% This section is for ADS Processing.  There must be one line per author. paperauthor has 9 arguments.
\paperauthor{R.~Moseley}{moseley@ipac.caltech.edu}{0000-0002-0759-0475}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Moseley, ~R.}
\paperauthor{G.~Bruce~Berriman}{gbb@ipac.caltech.edu}{0000-0001-8388-534X}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Berriman, ~G. ~B.}
\paperauthor{C.~R.~Gelino}{cgelino@ipac.caltech.edu}{0000-0001-5072-4574}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Gelino, ~C. ~R.}
\paperauthor{J.~C.~Good}{jcg@ipac.caltech.edu}{0009-0003-3906-719X}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Good, ~J. ~C.}
\paperauthor{M.~Lynn}{mlynn@ipac.caltech.edu}{0009-0008-7417-3170}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Good, ~J. ~C.}
\paperauthor{M.~Swain}{mswain@ipac.caltech.edu}{0000-0003-4557-1192}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Good, ~J. ~C.}
\paperauthor{T.~Oluyide}{toluyide@ipac.caltech.edu}{}{Caltech}{IPAC/NExScI}{Pasadena}{CA}{91125}{USA}
%\aindex{Oluyide, ~T.}


% There should be one \aindex line (commented out) for each author. These are used to
% build up the author index for the Proceedings. The surname must come first, followed by
% initials. Note the use of ~ before each initial to control spacing.
% The \author entries (see above) have surname last. These \aindex entries have
% surname first.
% The Aindex.py command willl create them for you after you have constructed the \author
% The first entry should be the first author, for bold-facing the author index page-reference

%\aindex{Moseley,~R.}
%\aindex{Berriman,~G.~B.}
%\aindex{Gelino,~C.~R.}
%\aindex{Good,~J.~C.}
%\aindex{Lynn,~M.}
%\aindex{Swain,~M.}
%\aindex{Oluyide,~T.}


\begin{abstract}
The Keck Observatory Archive (KOA) curates all observational data acquired at the W. M. Keck Observatory. The archive is expected to grow rapidly as complex new instruments are commissioned and as the expectations of archive users have expanded. In response, KOA has implemented a new Python-based, VO-compliant query infrastructure. This work is a continuation of the architectural design and technology selection identified at ADASS 2024. We have deployed real-time ingestion of newly acquired data and a dedicated interface for observers to manage these data. Our ADASS 2024 poster identified the new technologies chosen: Plotly-Dash, a low-code framework that exploits event-driven callbacks to simplify the handling of user interactions; R-tree spatial indexing to speed up spatial searches by $\times 20$; a VO-compliant TAP middleware, already in use at the NASA Exoplanet Archive and NEID archive; and \texttt{mViewer}, a visualization engine in the Montage Image Mosaic toolkit that is optimized for astronomy images.

These technologies will underpin new services that can be hosted on web pages or in Jupyter notebooks, and when completed, will replace the current query infrastructure. We have completed two new services now in beta release. The first is the Data Discovery Service, a web-based dashboard that returns spatial and temporal queries of the entire archive in seconds. It supports filtering observations by keywords, previewing results in an interactive data grid, and visualizing images, and it offers data downloads. The second is a Jupyter notebook that performs interactive visualization of Keck observations of protostars in the Rho Oph Dark Cloud and uses data from CDS and IRSA, as well as KOA.

\end{abstract}

% These lines show examples of subject index entries. At this stage these have to commented
% out, and need to be on separate lines. Eventually, they will be automatically uncommented
% and used to generate entries in the Subject Index at the end of the Proceedings volume.
% Don't leave these in! - replace them with ones relevant to your paper.



% These lines show examples of ASCL index entries. At this stage these have to commented
% out, and need to be on separate lines. Eventually, they will be automatically uncommented
% and used to generate entries in the ASCL Index at the end of the Proceedings volume.
% The ascl.py command will scan your paper on possible code names.
% Don't leave these in! - replace them with ones relevant to your paper.
%\ooindex{Montage, ascl:1010.036}


\section{Introduction}
The Keck Observatory Archive (KOA) manages over 30 years of raw and reduced data, from all current and decommissioned W. M. Keck Observatory instruments. The archive is currently modernizing its operations and query infrastructure to respond to changing directions in research and to serve massive datasets from the next generation of instruments. Modernization prioritizes widely adopted, open-source, and sustainable Python technologies wherever possible.
This paper describes two services that use the core technologies, described in \citet{Moseley} and summarized below, which have been demonstrated in the following two YouTube videos:

\begin{itemize}
    \item A Data Discovery Service (DDS) that queries the archive in seconds\footnote{\url{https://www.youtube.com/watch?v=RMtqsMjeh-Y}}
    \item A Jupyter notebook that uses the DDS to discover Keck data and integrate them with remote datasets, accessed through Virtual Observatory (VO) protocols\footnote{\url{https://www.youtube.com/watch?v=UC6-tuWWYQo}}.
\end{itemize}

\section{Data Discovery Service}

The Data Discovery Service (DDS) is a major part of the modernization of the KOA query infrastructure. The DDS takes a "low-code," Python-centric approach to architecture through the Plotly-Dash framework. This framework uses event-driven callbacks to manage complex client-server interactions, including coordinate resolution and quick spatial searches, without the overhead of maintaining a decoupled JavaScript front end. Figure \ref{ex_fig1} shows a block diagram of the DDS functionality.

The video demonstration performs a spatial search of a 6\deg radius centered on M51, which has been observed with every instrument at the Keck Observatory since it began operating in 1994. The DDS queried the entire archive of six million records in approximately 15 seconds. The return is shown in Figure \ref{ex_fig2}. The left column shows an instrument-by-instrument summary; on the right is an interactive tabbed display of the metadata.

\articlefigure[width=0.5\textwidth]{F1_f1.png}{ex_fig1}{Overview of the Data Discovery Service (DDS) functional design}

\articlefiguretwo{F1_f2_a.png}{F1_f2_b.png}{ex_fig2}{The Data Discovery Service (DDS) return page. \emph{Left:} The main search and results dashboard showing the instrument-based observation summary and filtered data table \emph{Right:} The integrated \texttt{mViewer} component displaying a DEIMOS FITS image preview with interactive histogram and stretch controls}

The DDS uses the following technologies:

\begin{itemize}
    \item \texttt{nexsciTAP}, a Python Table Access Protocol (TAP) implementation that provides a VO-compliant mechanism for handling Astronomy Data Query Language (ADQL) requests.
    \item R-tree spatio-temporal indices that approximate complex instrument geometries using minimum bounding rectangles (MBRs). Grouping nearby objects into hierarchical MBRs \citep{10.1145/602259.602266} instantiated as memory-mapped files\footnote{\url{https://github.com/Caltech-IPAC/Montage}}. R-tree indices allow fast searches, $\times 20$ faster than current services.
    \item A Dash React component wraps the \texttt{mViewer} module of the Montage Image Mosaic Engine \citep{2017PASP..129e8006B} to provide an adaptive, histogram-based visualizer.
    \item \texttt{wget} shell scripts for bulk data retrieval.

\end{itemize}





\section{KOA DDS Notebook}
We have delivered an example Jupyter notebook that uses the DDS to interact with remote observations of Rho Ophiuchi (Rho Oph). Two screenshots from the video\footnote{\url{https://www.youtube.com/watch?v=UC6-tuWWYQo}} are shown in Figure \ref{ex_fig3}. This second demonstration details the following workflow: after resolving the coordinates of Rho Oph with SIMBAD and defining a search radius, the DDS discovers all Keck observations in Rho Oph, retrieves a catalog of protostars in Rho Oph from VizieR \citep{2017ApJ...835....3L}, and overlays these data on a Spitzer IRAC mosaic acquired from IRSA 

\articlefiguretwo{F1_f3_a.png}{F1_f3_b.png}{ex_fig3}{Jupyter Notebook integration with the DDS API. \emph{Left:} Notebook usage of Keck observations and external VO catalogs within a Jupyter environment \emph{Right:} The resulting interactive visualization overlaid on the Spitzer IRAC mosaic}

\section{Conclusion}
The Plotly-Dash framework provides a powerful mechanism for building a new query infrastructure for KOA. The DDS and the associated notebook are examples of powerful new services that can be built with the new query infrastructure, and will deliver new tools to serve the needs of the rapidly expanding archive. 

\acknowledgements The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by the NASA Exoplanet Exploration Program and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

The observatory was made possible by the generous financial support of the W. M. Keck Foundation. The authors wish to recognize and acknowledge the very significant cultural role and reverence that the summit of Mauna Kea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain.


\bibliography{F1}  % For BibTex



% if we have space left, we might add a conference photograph here. Leave commented for now.
% \bookpartphoto[width=1.0\textwidth]{foobar.jpg}{FooBar Photo (Photo: Any Photographer)}

\end{document}
