Functional ADL
func-adl
is an implementation based upon the C#’s Language Integrated Query (LINQ) feature. func-adl
allows the physicist to declare, using a functional syntax, a query against particle physics data (or any structured data). It is based around streams of objects, be they events, jets, tracks, or electrons. A single event is expressed as a database, of sorts, and a stream of the events is processed.
The language is built to be data-format agnostic. There are projects to translate func-adl
into C++ to run against ATLAS xAOD’s and CMS Run 1 AOD’s, and any flat ROOT TTree.
Status
This project is released in production for ServiceX. It works with the ATLAS xAOD
and CMS Run 1 AOD (experiment-specific custom formats) and flat ROOT TTree files (using the uproot
and awkward
ecosystem).
GitHub Repositories
Repositories connected with this work:
- func_adl - The base package for the
func_adl
SQL-like data query language. - func_adl_xAOD - Backend that works with ATLAS xAOD files to extract columnar data.
- func_adl_uproot - Backend that works with
ROOT
TFile
’s that containTTree
’s. - func_adl_servicex - Front end that seamlessly connects
func-adl
and theServiceX
backend. - qastle - Interchange language that converts
func_adl
to a text language suitable for transmission over the web. While list-like and plain-text, it is meant as an interchange format, not an end-user language.
This work is based on the original work done with LINQ in the C# programming language (prior to the start or IRIS-HEP):
- Developing a Declarative Analysis Language: LINQToROOT - Given CHEP 2018
- Experiments Toward a Modern Analysis Environment: Using TMVA and other tools in a functional world with continuous integration for analysis - Given CHEP 2017
Team
- Gordon Watts
- Emma Torro
- Mason Proffitt
Presentations
- 11 Mar 2024 - "From Amsterdam to ACAT 2024: The Evolution and Convergence of Declarative Analysis Language Tools and Imperative Analysis Tools", Gordon Watts, ACAT 2024
- 15 Sep 2022 - "End-to-end physics analysis with Open Data: the Analysis Grand Challenge", Alexander Held, PyHEP 2022 (virtual) Workshop
- 25 Apr 2022 - "From data delivery to statistical inference with CMS Open Data", Alexander Held, IRIS-HEP AGC Tools 2022 Workshop
- 1 Dec 2021 - "Declarative interfaces for HEP data analysis: FuncADL and ADL/CutLang", Mason Proffitt, ACAT 2021
- 30 Nov 2021 - "ServiceX - Making everything Columnar", Gordon Watts, ACAT 2021
- 3 Nov 2021 - "Queries with func_adl" , Mason Proffitt, IRIS-HEP AGC Tools 2021 Workshop
- 3 Nov 2021 - "From data delivery to statistical inference: ServiceX, coffea, cabinetry & pyhf", Alexander Held, IRIS-HEP AGC Tools 2021 Workshop
- 9 Jul 2021 - "FuncADL: Functional Analysis Description Language" , Mason Proffitt, PyHEP 2021 Workshop
- 9 Jul 2021 - "Using Python, coffea, and ServiceX to Rediscover the Higgs. Twice." , Gordon Watts, PyHEP 2021
- 18 May 2021 - "FuncADL: Functional Analysis Description Language", Mason Proffitt, vCHEP 2021
- 26 Oct 2020 - "Data Selection & Delivery for Analysis: ServiceX & funcADL", Mason Proffitt, Future Analysis Systems and Facilities
- 27 Feb 2020 - "Functional Analysis Description Language (FuncADL)", Mason Proffitt, IRIS-HEP Poster Session
- 5 Nov 2019 - "Using Analysis Declarative Languages for the HL-LHC", Gordon Watts, CHEP 2019
- 16 Oct 2019 - "Lightning Talk: A Living HEP Analysis", Gordon Watts, PyHEP
- 26 Sep 2019 - "Declarative programming: A paradigm shift in data analysis in preparation for the HL-LHC", Gordon Watts, eScience2019
- 13 Sep 2019 - "func-adl to C++/xAOD backend", Gordon Watts, IRIS-HEP Institute Retreat
- 12 Sep 2019 - "Prototype declarative analysis interface using uproot and awkward-array", Mason Proffitt, IRIS-HEP Institute Retreat
- 19 Jun 2019 - "Functional/Declarative Selection Languages", Gordon Watts, Analysis Systems Topical Workshop
- 8 May 2019 - "IRIS-HEP: A new software institute to prepare for the data from the High Luminosity Large Hadron Collider in the exabyte era", Mason Proffitt, Northwest Data Science Summit
- 7 May 2019 - "Thinking about Analysis Languages and Recent Progress", Gordon Watts, Analysis Description Languages
- 25 Feb 2019 - "The C# LINQ Analysis Language", Gordon Watts, IRIS-HEP Topical Meeting on Analysis Description Languages
- 13 Feb 2019 - "LINQ To ROOT", Gordon Watts, 1st DAWG Technology and Innovation Survey (HSF)
Publications
- Declarative interfaces for HEP data analysis: FuncADL and ADL/CutLang, C. Huh, M. Proffitt, H. Prosper, S. Sekmen, B. Sen, G. Unel and G. Watts, J.Phys.Conf.Ser. 2438 012075 (2023) (01 Jan 2023) [NSF PAR].
- FuncADL: Functional Analysis Description Language, M. Proffitt and G. Watts, EPJ Web Conf. 251 03068 (2021) (02 Mar 2021) [8 citations] [NSF PAR].
- Snowmass 2021 Letter of Interest: Analysis Ecosystem at the HL-LHC, G. Watts Snowmass 2021 Letter of Interest (10 Sep 2020).