Abuse from crawling services

PO
Not scheduled
15m
Wichernhaus

Wichernhaus

Board: C243
poster presentation Collaborating with other software ecosystems and disciplines Poster

Speaker

Anaïs Oberto (CDS - Observatoire astronomique de Strasbourg)

Description

The increasing integration of astronomical data services with machine learning workflows has led to unprecedented demand on web-based astronomical databases. The SIMBAD astronomical database, operated by the Centre de Données astronomiques de Strasbourg (CDS), has experienced a significant surge in API requests, particularly from automated systems and AI model training pipelines that span multiple software ecosystems.
We present an analysis of request patterns observed on the SIMBAD server, characterizing the emergence of untraceable automated queries that complicate load balancing and resource allocation.
Should we addresses strategies for sustainable collaboration between astronomical data providers and the broader computational science community, including rate limiting approaches, API design considerations, and infrastructure scaling solutions that accommodate diverse usage patterns across different software ecosystems and research disciplines.

Affiliation of the submitter Centre de Données astronomiques de Strasbourg
Attendance in-person

Primary author

Anaïs Oberto (CDS - Observatoire astronomique de Strasbourg)

Co-author

gilles Landais (CDS, Observatoire Astronomique de Strasbourg)

Presentation materials