YourFS:A read only file system for data non-landing scenarios in the astronomical cloud

PO
Not scheduled
15m
Wichernhaus

Wichernhaus

Board: O233
poster presentation Science platforms in the big data era Poster

Speaker

Jun Han (National Astronomical Observatories, Chinese Academy of Sciences)

Description

With astronomy entering an era of petabyte to exabyte scale data from next-generation telescopes and surveys, existing cloud platforms face critical data management challenges. Traditional approaches of mounting entire datasets or copying filtered subsets create trade-offs between access efficiency and storage costs, while conventional storage engines lack flexible permission control. To address this, this paper presents a POSIX-compliant read-only file system designed for non-landing data scenarios. The system integrates FUSE for POSIX interface implementation, libnfs for server-side data retrieval, and a database-driven metadata layer that enables file-level access isolation through SQL-defined policies. Performance testing indicates that maintaining under 5,000 files per directory ensures optimal user experience, while read operations achieve speeds of more than 50MB/s under a gigabit local area network – sufficient for most application scenarios in astronomical data workflows. Our results indicate that YourFS effectively balances accessibility and cost-efficiency for massive datasets, providing a viable paradigm for server-side data management in cloud platforms.

Affiliation of the submitter National Astronomical Observatories, Chinese Academy of Science
Attendance in-person

Primary author

Jun Han (National Astronomical Observatories, Chinese Academy of Sciences)

Presentation materials