Third Party Copy
LHC data is constantly beign moved between computing and storage sites to support analysis, processing, and simluation; this is done at a scale that is currently unique within the science community. For example, the CMS experiment on the LHC manages approximately 200PB of data and, on a daily basis, moves 1PB between sites.
Historically, this has been done with the GridFTP protocol; as we look to the increased data volumes of HL-LHC and GridFTP becomes increasingly niche, the LHC community is looking for alternate mechanisms and protocols to move data. The IRIS-HEP DOMA area - in collaboration with the WLCG DOMA activity - is investigating the use of HTTP for bulk data transfer.
Bandwidth achieved from standalone testing
The above graph shows data movement rates (up to 24Gbps) for a single host, achieved during standalone tests; a typical LHC site will load-balance across multiple hosts in order to saturate available network links.
- Brian Bockelman
- Edgar Fajardo
- 5 Nov 2019 - "Third-party transfers in WLCG using HTTP", Brian Bockelman, 24th International Conference on Computing in High Energy & Nuclear Physics
- 28 May 2019 - "WLCG DOMA TPC Working Group", Brian Bockelman, US CMS Tier-2 Facilities May 2019 Meeting
- 6 Feb 2019 - "IRIS-HEP DOMA", Brian Bockelman, IRIS-HEP Steering Board Meeting