The Analysis Grand Challenge

Analysis Grand Challenge

The “Analysis Grand Challenge” organized by IRIS-HEP includes the binned analysis, reinterpretation and end-to-end optimization of physics analysis use cases. It also includes the development of the required cyber infrastructure to execute them in order to demonstrate technologies envisioned for HL-LHC. To enable these use cases and more, the expected capabilities include:

  • New user interfaces: complementary services that present the analyst with a notebook-based interface. Example software: Jupyter.
  • Data access: services that provide quick access to the experiment’s official data sets, often allowing simple derivations and local caching for efficient access. Example software and services: Rucio, ServiceX, SkyHook, RNTuple.
  • Event selection: systems/frameworks allowing analysts to process entire datasets, select desired events, and calculate derived quantities. Example software and services: Coffea, awkward-array, awkward-dask, func_adl, RDataFrame.
  • Histogramming and summary statistics: closely tied to the event selection, histogramming tools provide physicists with the ability to summarize the observed quantities in a dataset. Example software and services: Coffea, func_adl, cabinetry, hist.
  • Statistical model building and fitting: tools that translate specifications for event selection, summary statistics, and histogramming quantities into statistical models, leveraging the capabilities above, and perform fits and statistical analysis with the resulting models. Example software and services: cabinetry, pyhf, FuncX+pyhf fitting service
  • Reinterpretation / analysis preservation: standards for capturing the entire analysis workflow, and services to reuse the workflow which enables reinterpretation. Example software and services: REANA, RECAST.

Generic schema of AGC components

The Analysis Grand Challenge is being conducted during 2021‒2023, leaving enough time for tuning software tools and services developed as a part of the IRIS-HEP ecosystem before the start-up of the HL-LHC.

AGC has a dedicated webpage for documentation: https://agc.readthedocs.io/en/latest/

Documentation Status GitHub Project

Recent accomplishments and plans

Recent accomplishments:

Future plans for 2023:

  • Improve experiment-related coffea-casa setups (e.g. improve experiment specific data access and other features)
  • Performance tests of ServiceX integrated in Coffea-Casa analysis facility at University Nebraska-Lincoln and ATLAS analysis facility instance at the University of Chicago
  • Benchmark performance of prototype of other system components for Analysis Grand Challenge
  • Improve and increase complexity of developed analysis example used for next round of demonstration (the first example based on CMS Open Data was shown at the AGC Tools 2022 Workshop)
  • Incorporate ML into Analysis Grand Challenge analysis workflow and execute on analysis facilities
  • Organise IRIS-HEP AGC community workshop at University Wisconsin-Madison - AGC 2023 Workshop
  • Prepare for Analysis Grand Challenge execution event (September 2023)

Videos and tutorials

AGC Workshop 2022

AGC Workshop 2021

Fellows

Team

Presentations

Publications