Caching Analysis Data
Significant portions of LHC analysis use the same datasets, running over each dataset several times. Hence, we can utilize cache-based approaches as an opportunity to efficiency of CPU use (via reduced latency) and network (reduce WAN traffic). We are investigating the use of regional caches to store, on-demand, certain datasets. For example, the UCSD CMS Tier-2 and Caltech CMS Tier-2 joined forces to create and mantain a regional cache that benefits all southern California CMS researchers.
These in-production caches have shown to save up to a factor of three of WAN bandwidth compared with traditional data management techniques.
Repositories
Currently XCache is distributed by the OSG both in the form of RPM and docker images. The following are the corresponding repositories where the base code can be found:
Reports
- Report on cache usage on the WLCG and potential use cases and deployment scenarios for the US LHC facilities
- Report on LHC data access patterns, data uses, and intelligent caching approaches for the HL-LHC (draft)
Team
- Brian Bockelman
- Edgar Fajardo
- Igor Sfiligoi
- Frank Wuerthwein
- Matevz Tadel
- Diego Davila
Presentations
- 15 Sep 2020 - "Data lake prototyping for US CMS", Edgar Fajardo, DOMA / ACCESS Meeting
- 23 Apr 2020 - "How CMS user jobs use the caches", Edgar Fajardo, XCache DevOps SPECIAL
- 22 Apr 2020 - "XRootD Transfer Accounting Validation Plan", Diego Davila, S&C Blueprint Meeting
- 27 Feb 2020 - "XCache", Edgar Fajardo, IRIS-HEP Poster Session
- 5 Nov 2019 - "Creating a content delivery network for general science on the backbone of the Internet using xcaches.", Edgar Fajardo, CHEP 2019
- 5 Nov 2019 - "Moving the California distributed CMS xcache from bare metal into containers using Kubernetes", Edgar Fajardo, CHEP 2019
- 12 Sep 2019 - "OSG XCache Discussion", Frank Wuerthwein, IRIS-HEP retreat
- 31 Jul 2019 - "CMS XCache Monitoring Dashboard", Diego Davila, OSG Area Coordination
- 8 Jul 2019 - "XCache Initiatives and Experiences", Frank Wuerthwein, pre-GDB meeting on XCache
- 20 Mar 2019 - "Data Access in DOMA", Frank Wuerthwein, HOW2019 (Joint HSF/OSG/WLCG Workshop)
- 7 Mar 2019 - "The OSG Data Federation", Frank Wuerthwein, Internet2 Global Summit 2019
- 16 Jan 2019 - "OSG Cache on Internet Backbone developments", Edgar Fajardo, GDB Jan 2019
- 2 Oct 2018 - "Current production use of caching for CMS in Southern California", Edgar Fajardo, DOMA / ACCESS Meeting