Intelligent Data Delivery Service
If HL-LHC is going to process exabytes of data, it needs data access systems that can deliver. The HEP community has always focused on delivering bytes to applications - each application must have an innate understanding of the data format in order to interpret the byte stream and it must understand the file naming scheme of the underlying storage.
The intelligent data delivery system is working to raise these semantics to work on delivering events as opposed to delivering bytes. This allows an edge service to prepare data for production consumption (filtering out unnecessary events and objects), the on-disk data format to evolve independently of applications, and decrease the latency between the application and the storage.
The IDDS work is an ongoing project within IRIS-HEP in the DOMA and Analysis Systems area, as well as within the HEP Software Foundation event delivery group.
- Event Streaming Service for ATLAS Event Processing (Wen Guan, 07 Nov 2019) at 24th International Conference on Computing in High Energy Physics(CHEP2019)
- IDDS (Wen Guan, 30 Sep 2019) at HSF & ATLAS Joint Event Delivery Workshop
- Delivery of columnar data to analysis systems (Marc Weinberg, 24 Jun 2019) at ATLAS Software & Computing Week #62
- WLCG DOMA TPC Updates (Brian Bockelman, 20 Mar 2019) at 2019 Joint HSF/OSG/WLCG Workshop (HOW2019)
- IRIS-HEP DOMA (Brian Bockelman, 06 Feb 2019) at IRIS-HEP Steering Board Meeting