Scaling to HL-LHC Data Rates

During the High-Luminosity era the LHC experiments expect a significant increment in the data produced by the collider, this also supposes an serious increase of data to be distributed across the different Facilities that pledge resources to the collaborations.

Data Challenge In order to prepare and assess the readiness of the infrastructure and the systems in charge of transferring data, the WLCG has proposed a series of incremental “Data Challenges”. These challenges are carried out every 2 years, with many “Mini-challenges” happening in between.

The objective of this project is to support the participation of US sites in the “Data Challenges” and coordinate mini-challenges that help to not only identify possible shortcomings in the capacity of the sites but also test new technologies that could help us to sustain the expected data rates. One of such technologies is SENSE.

Data Challenge 2024

Overview of the data rates reached during the Data Challenge 2024.

SENSE/Rucio SENSE is a Network Services Orchestrator created by ESNet that allows, among other things, to create bandwidth allocations across Facilities. We have envisioned Rucio as the first use case of SENSE. The objective of this project is deploy support for SENSE in the majority of the USCMS and USATLAS sites and enable Rucio to make use of this technology in order to assign priorities to its more critical data flows.

SENSE Demonstration

Demonstration of SENSE’s bandwidth allocation over 2 competing data flows.

Team

• Brian Bockelman
• Diego Davila
• Aashay Arora

Publications

• SENSE in Practice: Quantifying the End-to-End Benefits of Intent-Based Bandwidth Reservation for Exascale Science Workflows, Monga, Inder, Mazahir Hussain, Justas Balcas, Aashay Arora, Diego Davila, Buseung Cho, Xi Yang, and Cees De Laat. SENSE in Practice: Quantifying the End-to-End Benefits of Intent-Based Bandwidth Reservation for Exascale Science Workflows. In Proceedings of the SC'25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 875-885. 2025. (16 Nov 2025).
• Benchmarking XRootD-HTTPS on 400Gbps Links with Variable Latencies, A. Arora, D. Davila, F. Würthwein, J. Graham, D. Mishin, J. Balcas, T. Lehman, X. Yang, C. Guok and H. Newman, EPJ Web Conf. 337 01212 (2025) (01 Aug 2025).
• Data Movement Manager (DMM) for the SENSE-Rucio Interoperation Prototype, A. Arora, D. Davila, J. Guiang, F. Würthwein, H. Newman, J. Balcas, T. Lehman and X. Yang, EPJ Web Conf. 337 01208 (2025) (01 Jan 2025).