Jump to: Fellows - Projects - Presentations - Publications

Data Organization, Management and Access (DOMA)

Diagram of conceptual components in the DOMA ecosystem.
An overview of the conceptual components of the data management ecosystem for HEP. The technologies that the DOMA team contributes to are highlighted in red.

The HL-LHC era will provide enormous challenges in the area of Data Organization, Management and Access (DOMA). The LHC will provide a significantly increased number of events and increased event complexity, both of which will drive much larger data sizes - with no changes in how the LHC community functions, the total increase in data volume may be a factor of 30.

Given the LHC experiments are, combined, managing nearly an exabyte of data, such a significant increase in volume is unmanageable. New mechanisms and techniques are necessary to more efficiently manage storage resources and deliver data to processing endpoints; the DOMA area in IRIS-HEP is working on the R&D necessary to affect such change.

The bulk data transfer technologies were designed almost 15 years ago; the DOMA team is taking a fresh look at the transfer protocols and the authentication and authorization infrastructure used by the LHC community. This is resulting in a worldwide transition to new protocols and authorization approaches. The first phase of this work is done: the LHC community has successfully transitioned to HTTP as a foundation for bulk data transfers.

It is not only data volumes that are potentially disruptive to the HL-LHC physics program; the extraordinarily large number of events (potentially 150 billion similated and recorded events per year per experiment) presents a challenge in data management for users. Along with the analysis systems team within IRIS, DOMA is working on improved techniques for delivering events to users. Not only is the team researching new approaches for data delivery and implementing services but also working to integrate them together with into a coherent analysis facility, Coffea-Casa, for users.

Contact us: doma-team@iris-hep.org

Current and Previous DOMA Fellows

DOMA Projects


Card image cap

Analysis Grand Challenge

Analysis Grand Challenge
More information
Development
Card image cap

Caching Data for LHC Analysis

Cached-based placement of analysis datasets.
More information
Deployed
Card image cap

Coffea-Casa

A Prototype of Analysis Facility
More information
Testing
Card image cap

Intelligent Data Delivery Service

Delivering Data. Better.
More information
Deployed
Card image cap

Modeling Data Workflows

Modeling HL-LHC Data flows
More information
Archived
Card image cap

ServiceX

Delivering columnar data on demand
More information
Testing
Card image cap

SkyhookDM

Programmable Storage for Databases and Datasets
More information
Development
Card image cap

Third Party Copy

Envisioning a new way to move LHC data
More information
Deployed

DOMA Presentations

DOMA Publications

Join us

We collaborate with groups around the world on code, data, and more. See our project pages for more.

gantt title Data Organization, Management and Access (DOMA) project lifecycles dateFormat YYYY-MM-DD axisFormat %Y Caching Data for LHC Analysis :active , 2015-02-08, 2024-11-10 Coffea-Casa :done , 2019-01-01, 2024-11-10 Intelligent Data Delivery Service :active , 2019-09-01, 2024-11-10 ServiceX :done , 2019-04-17, 2024-11-10 SkyhookDM :done , 2021-02-01, 2024-11-10 Third Party Copy :active , 2017-09-01, 2024-11-10