Third Party Copy
LHC data is constantly beign moved between computing and storage sites to support analysis, processing, and simluation; this is done at a scale that is currently unique within the science community. For example, the CMS experiment on the LHC manages approximately 200PB of data and, on a daily basis, moves 1PB between sites.
Historically, this has been done with the GridFTP protocol; as we look to the increased data volumes of HL-LHC and GridFTP becomes increasingly niche, the LHC community is looking for alternate mechanisms and protocols to move data. The IRIS-HEP DOMA area - in collaboration with the WLCG DOMA activity - is investigating the use of HTTP for bulk data transfer.
Bandwidth achieved from standalone testing
The above graph shows data movement rates (up to 24Gbps) for a single host, achieved during standalone tests; a typical LHC site will load-balance across multiple hosts in order to saturate available network links.
- Brian Bockelman
- Edgar Fajardo
- Third-party transfers in WLCG using HTTP (Brian Bockelman, 05 Nov 2019) at 24th International Conference on Computing in High Energy & Nuclear Physics
- WLCG DOMA TPC Working Group (Brian Bockelman, 28 May 2019) at US CMS Tier-2 Facilities May 2019 Meeting
- IRIS-HEP DOMA (Brian Bockelman, 06 Feb 2019) at IRIS-HEP Steering Board Meeting