The 35th annual conference on Astronomical Data Analysis Software & Systems 2025 in Görlitz, Germany

Most traditional data reduction pipelines are on run an investigators local machine or remote machines requiring a manual touch to be executed. This approach leads to data discoverability and reproducibility issues. Additionally, observatory sites are also often remote and one of the major challenges is facilitating data transfer from site to site. Data Central's Apache NiFi system aims to combine data transfer, archival workflows and triggering pipeline execution to produce discoverable data products while managing the complexity of secure site to site data transfer. This talk will cover the implementation and design of this system for Data Central's archive of the ANU 2.3m Telescope at Siding Spring Observatory and how we have designed our system to enable scalable data transfers, data transformation and pipeline execution for incoming observations. The talk will also highlight how archival processes are integrated to improve data discoverability ensuring data products are easily located, shared and reused while also improving observability at every stage of the process. We will also discuss possible future directions including automatic notification systems (such as for transient events) and multi instrument or multi observatory abilities.