The 35th annual conference on Astronomical Data Analysis Software & Systems 2025 in Görlitz, Germany

How do you present and discuss the functionality of a science pipeline/workflow? Chances are very high, that you will draw some connected boxes with a bit of explanation around them on a whiteboard or in a publication figure. The DALiuGE framework enables you to do just that by dragging and connecting abstract component symbols or, far more useful, by using automatically generated component symbols from your code in a web-based workflow graph editor called EAGLEπ. The stand-alone auto-generation tool dlg-paletteGen can introspect any installed Python module, including PyBind11 based code, to extract the signature of functions, methods and classes. It creates so-called component palettes, which in turn can be loaded into EAGLEπ and used there to design workflow graphs. dlg-paletteGen also extracts any in-line documentation and argument descriptions, including types, which can then be viewed in EAGLEπ. If no in-line documentation can be found, a LLM is used to generate it from the (Python) code. This enables the user to inspect the functionality of every single component of an existing graph or palette. Users can also add graph descriptions and comments. Palettes and graphs are version controlled in GitHub or GitLab repositories and can be shared and co-developed by a team of people. The best of all of that is that the rest of the DALiuGE system then allows you to translate and schedule such a graph and deploy it to an engine running on a laptop, server, small cluster or the biggest HPC facilities in the world. Builtin, high-level components support scatter, gather, loop and branch constructs, enabling the development and execution of highly sophisticated, parallelised workflows.