Data Organization, Management and Access (DOMA)

The HL-LHC era will provide enormous challenges in the area of Data Organization, Management and Access (DOMA). The LHC will provide a significantly increased number of events and increased event complexity, both of which will drive much larger data sizes - with no changes in how the LHC community functions, the total increase in data volume may be a factor of 30.

Given the LHC experiments are, combined, managing nearly an exabyte of data, such a significant increase in volume is unmanageable. New mechanisms and techniques are necessary to more efficiently manage storage resources and deliver data to processing endpoints; the DOMA area in IRIS-HEP is working on the R&D necessary to affect such change.

The bulk data transfer technologies were designed almost 15 years ago; the DOMA team is taking a fresh look at the transfer protocols and the authentication and authorization infrastructure used by the LHC community. This is resulting in a worldwide transition to new protocols and authorization approaches. The first phase of this work is done: the LHC community has successfully transitioned to HTTP as a foundation for bulk data transfers.

It is not only data volumes that are potentially disruptive to the HL-LHC physics program; the extraordinarily large number of events (potentially 150 billion similated and recorded events per year per experiment) presents a challenge in data management for users. Along with the analysis systems team within IRIS, DOMA is working on improved techniques for delivering events to users. Not only is the team researching new approaches for data delivery and implementing services but also working to integrate them together with into a coherent analysis facility, Coffea-Casa, for users.

Contact us: doma-team@iris-hep.org