awkward-array is a pure Python+NumPy library for manipulating complex data structures as you would NumPy arrays. Even if your data structures
- contain variable-length lists (jagged or ragged),
- are deeply nested (record structure),
- have different data types in the same list (heterogeneous),
- are masked, bit-masked, or index-mapped (nullable),
- contain cross-references or even cyclic references,
- need to be Python class instances on demand,
- are not defined at every point (sparse),
- are not contiguous in memory,
- should not be loaded into memory all at once (lazy),
this library can access them with the efficiency of NumPy arrays. They may be converted from JSON or Python data, loaded from “awkd” files, HDF5, Parquet, or ROOT files, or they may be views into memory buffers like Arrow.
Team
Presentations
- 8 Nov 2023 - "An Awkward module for round-tripping data structures between Python and Julia", Ianna Osborne, JuliaHEP 2023 Workshop
- 11 Oct 2023 - "Awkward Target for Kaitai Struct", Manasvi Goyal, PyHEP 2023 - "Python in HEP" Users Workshop (online)
- 9 Oct 2023 - "Python and Fast Imperative Code: Lowering the Barriers", Ianna Osborne, PyHEP 2023 - "Python in HEP" Users Workshop (online)
- 11 May 2023 - "Awkward Just-In-Time (JIT) Compilation: A Developer’s Experience", Ianna Osborne, 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP2023)
- 5 May 2023 - "Scientific Python", Jim Pivarski, HSF-India Training event at TIFR
- 6 Mar 2023 - "All about AwkwardForth", Jim Pivarski, ATLAS columnar analysis group
- 27 Oct 2022 - "The Awkward World of Python and C++", Manasvi Goyal, 21st International Workshop on Advanced Computing and Analysis Techniques in Physics Research - ACAT2022
- 25 Oct 2022 - "Awkward to RDataFrame and back", Ianna Osborne, 21st International Workshop on Advanced Computing and Analysis Techniques in Physics Research - ACAT2022
- 6 Oct 2022 - "Awkward <--> RDataFrame Status", Ianna Osborne, 143rd ROOT Parallelism, Performance and Programming Model Meeting
- 28 Sep 2022 - "Status of Analysis - The Python Perspective", Jim Pivarski, Future Trends in Nuclear Physics Computing
- 15 Sep 2022 - "Awkward RDataFrame Tutorial", Ianna Osborne, PyHEP 2022 (virtual) Workshop
- 15 Sep 2022 - "End-to-end physics analysis with Open Data: the Analysis Grand Challenge", Alexander Held, PyHEP 2022 (virtual) Workshop
- 12 Sep 2022 - "Uproot, Awkward Array, hist, Vector: from basics to combinatorics", Jim Pivarski, PyHEP 2022 Workshop
- 8 Sep 2022 - "Adoption of Python and modern software practices in high energy physics", Jim Pivarski, CHESS Group Meeting
- 3 Aug 2022 - "Columnar data analysis (tutorial)", Jim Pivarski, Fourth Computational and Data Science for High Energy Physics (CoDaS-HEP) School
- 25 Jul 2022 - "Awkward Arrays: Accelerating scientific data analysis on irregularly shaped data (poster)", Jim Pivarski, NSF Cyberinfrastructure for Sustained Scientific Innovation (CSSI) Principal Investigator Meeting
- 20 Jul 2022 - "Analysis Grand Challenge / HEP Scientific Python Ecosystem", Mason Proffitt, DANCE/CoDaS@Snowmass 2022 computational and data science software training
- 11 Jul 2022 - "Loopy and Unloopy programming techniques (tutorial)", Jim Pivarski, SciPy 2022 Conference
- 23 May 2022 - "Analysis user experience with the Python HEP ecosystem", Jim Pivarski, Analysis Ecosystems Workshop II
- 10 May 2022 - "Awkward Array to RDataFrame and back", Ianna Osborne, ROOT Users Workshop
- 25 Apr 2022 - "Foundation libraries (uproot, awkward, hist, mplhep)", Mason Proffitt, IRIS-HEP AGC Tools 2022 Workshop
- 6 Apr 2022 - "Awkward Array Updates", Jim Pivarski, PyHEP Topical Meeting
- 1 Dec 2021 - "Lessons learned in Python-C++ integration", Jim Pivarski, ACAT 2021
- 3 Nov 2021 - "Data handling: uproot, awkward & vector" , Mason Proffitt, IRIS-HEP AGC Tools 2021 Workshop
- 27 Sep 2021 - "Awkward suggestions for the Julia Workshop", Jim Pivarski, Julia for HEP mini-workshop
- 13 Sep 2021 - "Uproot/Awkward Array tutorial for STAR", Jim Pivarski, STAR Junior's meeting
- 14 Jul 2021 - "Awkward Array 3-minute update", Jim Pivarski, SciPy 2021 (Tools plenary)
- 6 Jul 2021 - "Uproot/Awkward Array tutorial", Jim Pivarski, PyHEP 2021
- 14 Jun 2021 - "Uproot Awkward Columnar HATS", Jim Pivarski, Hands-on Advanced Tutorial Sessions at the LPC
- 26 May 2021 - "Refactoring Awkward Array", Jim Pivarski, Analysis Systems biweekly meeting
- 21 May 2021 - "New Dask collection type for Awkward Arrays", Jim Pivarski, Dask Distributed Summit
- 19 May 2021 - "AwkwardForth: accelerating Uproot with an internal DSL", Jim Pivarski, vCHEP 2021
- 26 Oct 2020 - "Access & Manipulation of Complex Data Structures: Uproot & Awkward Array", Jim Pivarski, Future Analysis Systems and Facilities
- 1 Oct 2020 - "Future of User Analysis", Jim Pivarski, LHCb Computing Workshop
- 15 Jul 2020 - "Uproot and Awkward Array tutorial", Jim Pivarski, PyHEP 2020
- 5 Jul 2020 - "Awkward Array: Manipulating JSON like Data with NumPy like Idioms", Jim Pivarski, SciPy 2020
- 8 Jun 2020 - "Uproot Awkward columnar HATS", Jim Pivarski, LPC HATS: Hands-on Training for CMS
- 8 Apr 2020 - "Uproot and Awkward Array tutorials for the Electron Ion Collider", Jim Pivarski, Electron Ion Collider User's meeting
- 27 Feb 2020 - "Awkward Arrays for Analysis Systems", Jim Pivarski, IRIS-HEP Review
- 7 Nov 2019 - "Ragged, jagged, nested, indirected, very awkward arrays", Jim Pivarski, CHEP 2019
- 17 Oct 2019 - "Awkward 1.0", Jim Pivarski, PyHEP Workshop
- 14 Sep 2019 - "Jagged, ragged, awkward arrays", Jim Pivarski, Strange Loop 2019
- 29 Jul 2019 - "IRIS-HEP Tutorial: Fast columnar data analysis with data science tools", Jim Pivarski, Division of Particles and Fields (DPF) of the American Physical Society (APS)
- 10 Jul 2019 - "Motivation and requirements for awkward 1.0", Jim Pivarski, Analysis Systems Biweekly Meeting
- 19 Jun 2019 - "Update on awkward-array, uproot, and related projects", Jim Pivarski, Analysis Systems Topical Workshop
- 29 May 2019 - "Columnar Analysis Tools HATS", Jim Pivarski, LPC HATS: Hands-on Training for CMS
- 17 Apr 2019 - "Awkward Array: Numba", Jim Pivarski, IRIS-HEP Topical Meetings
- 1 Apr 2019 - "PyROOT, uproot, and awkward-arrays", Jim Pivarski, Software Carpentry at Fermilab
- 14 Mar 2019 - "Nested data structures in array and SIMD frameworks", Jim Pivarski, ACAT 2019
Publications
- Awkward Just-In-Time (JIT) Compilation: A Developer's Experience, Ianna Osborne, Jim Pivarski, Ioana Ifrim, Angus Hollands and Henry Schreiner, arXiv:2310.01461 [cs.PL] (Submitted to CHEP 2023) (03 Oct 2023).
- Using a DSL to read ROOT TTrees faster in Uproot, A. Roy and J. Pivarski, arXiv 2303.02202 (Submitted to ACAT 2022) (03 Mar 2023).
- The Awkward World of Python and C++, M. Goyal, I. Osborne and J. Pivarski, arXiv 2303.02205 (Submitted to ACAT 2022) (03 Mar 2023).
- Awkward to RDataFrame and back, I. Osborne and J. Pivarski, arXiv 2302.09860 (Submitted to ACAT 2022) (20 Feb 2023) [1 citation].
- An array-oriented Python interface for FastJet, A. Roy, J. Pivarski and C. Freer, J.Phys.Conf.Ser. 2438 012011 (2023) (08 Feb 2022) [NSF PAR].
- AwkwardForth: accelerating Uproot with an internal DSL, J. Pivarski, I. Osborne, P. Das, D. Lange and P. Elmer, EPJ Web Conf. 251 03002 (2021) (24 Feb 2021) [NSF PAR].
- Awkward Array: JSON-like data, NumPy-like idioms, Jim Pivarski, Ianna Osborne, Pratyush Das, Anish Biswas, Peter Elmer, SciPy 2020 (07 Jul 2020).
- Nested data structures in array frameworks, J. Pivarski, D. Lange and P. Elmer, J.Phys.Conf.Ser. 1525 012053 (2020) (07 Jul 2020) [1 citation] [NSF PAR].
- Awkward Arrays in Python, C++, and Numba, J. Pivarski, P. Elmer and D. Lange, EPJ Web Conf. 245 05023 (2020) (15 Jan 2020) [10 citations].
- Columnar data processing for HEP analysis, J. Pivarsk, J. Nandi, D. Lange and P. Elmer, EPJ Web Conf. 214 06026 (2019) (17 Sep 2019) [3 citations].