OSG Software and Release
The OSG Software and Release team produces and supports the integrated software systems that comprise the OSG fabric of production services.
Software curation: the team works with external software providers, ensures the continued functionality of abandoned software, and develops tools to fill any gaps to provide a software stack suitable for WLCG sites. This software stack is distributed through packages used widely by the community, such as RPMs and container images.
Integration testing: the team coordinates tests of the integrated software stack among internal staff and external stakeholders. Additionally, the team maintains and monitors nightly automated tests that exercise different integrated software installations, varying software versions, installation methods, and hosting operating systems (OS).
Site documentation: the team maintains documentation that outline how to install, configure, and update software; how to address common problems; detailed notes about new software releases and any known issues; where to find help; and policy information regarding the OSG Software Stack.
Registration: the team provides tools for sites to register site contacts, services, and downtimes with the WLCG.
Community enagement: members of the team attend and run regular meetings with members of U.S. LHC, external software providers, and other IRIS-HEP teams to ensure that the software stack is responsive to all stakeholder needs.
Authentication and transfer protocol transitions: the OSG Software and Release team has been working with the DOMA group to lead the transition to a bearer token-based authentication and authorization infrastructure and HTTP/S-based transfer protocol.
Retirement of OSG 3.5: OSG 3.5 was the last version of the OSG Software Stack with Globus dependencies and its retirement marks the completion of the years-long transition from X.509 to token-based pilot job authentication. In particular, this meant the end of OSG Software support for GridFTP, GSI, Hadoop, and LCMAPS. This version of the software stack provided the bridge between the two different authentication methods, supporting both concurrently, and allowed for sites to become more familiar with the new technologies.
Release of OSG 3.6 OSG 3.6 is the first version of the OSG Software Stack without any Globus dependencies. Triggered by the 2017 announcement that Globus was ending support for their open-source Globus Toolkit, the release of OSG 3.6 is the culmination of years of planning and development, and includes versions of HTCondor, HTCondor-CE, and XRootD that rely on token-based authentication instead of X.509 proxy authentication. In addition to the changes in software, the team made a number of release policy changes to ease the transition to token-authentication and to reflect the increasingly agile operational models being explored by the LHC.
Enterprise Linux 8 support: the OSG Software Stack is built upon RedHat Enterprise Linux (EL) compatible operating systems and new operating system versions require rebuilding and verification of the entire software stack. With the release of EL 8, RedHat announced a change in support model for CentOS, a popular open source EL-compatible OS, promising quicker updates but shorter support lifecycles. The OSG Software Team worked with the LHC community to test this new version of CentOS as well as other open-source alternatives and provided guidance for US LHC sites. EL 8 also included a major version upgrade of Python from 2 to 3, resulting in major development efforts by the team and coordination of external software providers.
Streamlining the release process: to get new software into the hands of site administrators more quickly, the team has implemented rolling-style releases along with a policy to make release candidates available more quickly.
The OSG Software and Release team has also supported the LHC community’s exploration into DevOps deployment models by distributing select production services as container images. In particular, the team developed and maintains the caching container image used by multiple LHC experiments. The team follows a strategy of providing a common base container across all communities, plus community-specific refinements for U.S. ATLAS and U.S. CMS.
Retirement of OSG 3.4: Retirement of older software is the other side of the coin of maintaining the OSG Software Stack that gives our community timelines to transition to new technologies while ensuring production-readiness of the distributed computing infrastructure. In particular, the retirement of OSG 3.4 meant the end of support for software such as edg-mkgridmap, GUMS, BeStMan, and VOMS Admin Server packages as well as the end-of-life of RedHat Enterprise Linux 6 based operating systems.
Provide regular software updates: to ensure the security and continued functionality of the integrated OSG Software Stack, the Software Team provides periodic updates to its constituent software components. Over the calendar year 2021, the Software Team published 54 releases containing 200 software version updates.
- Brian Lin
- Carl Edquist
- Mátyás (Mat) Selmeci
- Tim Theisen
- 17 Nov 2022 - "OSG Container Experiences", Brian Lin, WBS 2.3/5 AF Container meeting
- 25 Mar 2022 - "Distributing Production-Ready Software", Brian Lin, Analysis Facilities Forum Kick-off Meeting
- 16 Mar 2022 - "OSG 3.6 and Token Transition Update", Brian Lin, OSG All-Hands Meeting 2022
- 10 Nov 2021 - "Token Transition Workshop Summary", Brian Lin, 10 November 2021 Grid Deployment Board
- 25 Oct 2021 - "Token Transition Workshop Summary", Brian Lin, ADC Technical Coordination Board
- 14 Oct 2021 - "OSG Software Bearer Token Transition Plans", Brian Lin, OSG Token Transition Workshop
- 24 Sep 2021 - "HTCondor-CE: What’s new and upcoming?", Brian Lin, European HTCondor Workshop 2021
- 27 May 2021 - "HTCondor CE 5 and Job Router Transform Language", Brian Lin, HTCondor Week 2021
- 12 May 2021 - "OSG/GlideinWMS Token Auth Update", Brian Lin, 12 May 2021 GDB
- 11 Feb 2021 - "OSG Services and Kubernetes", Brian Lin, GP-ARGO Technical Meeting
- 23 Sep 2020 - "HTCondor-CE Overview", Brian Lin, HTCondor Workshop Autumn 2020
- 23 Sep 2020 - "HTCondor-CE Configuration", Brian Lin, HTCondor Workshop Autumn 2020
- 23 Sep 2020 - "HTCondor-CE Live Installation", Brian Lin, HTCondor Workshop Autumn 2020
- 23 Sep 2020 - "HTCondor-CE Troubleshooting", Brian Lin, HTCondor Workshop Autumn 2020
- 23 Sep 2020 - "What is next for the HTCondor-CE?", Brian Lin, HTCondor Workshop Autumn 2020
- 17 Jun 2020 - "Introduction to HTCondor-CE", Brian Lin, EGI Community Webinar Programme
- 27 Feb 2020 - "Infrastructure Software Delivery (Poster)", Brian Lin, IRIS-HEP Poster Session
- 27 Feb 2020 - "GridFTP and GSI Migration Plan (Poster)", Brian Lin, IRIS-HEP Poster Session
- 26 Sep 2019 - "HTCondor-CE Basics and Architecture", Brian Lin, European HTCondor Week
- 26 Sep 2019 - "HTCondor-CE Configuration", Brian Lin, European HTCondor Week
- 26 Sep 2019 - "HTCondor-CE Integration with APEL and BDII", Brian Lin, European HTCondor Week
- 26 Sep 2019 - "HTCondor-CE Troubleshooting", Brian Lin, European HTCondor Week
- 26 Sep 2019 - "HTCondor-CE What's Next?", Brian Lin, European HTCondor Week
- 27 Jun 2019 - "Security Model for Containerized Grid Services", Brian Lin, Atlas Software & Computing Week #62
- 21 Mar 2019 - "OSG Software Containers", Brian Lin, 2019 Joint HSF/OSG/WLCG Workshop
- 21 Mar 2019 - "Container Basics", Brian Lin, 2019 Joint HSF/OSG/WLCG Workshop
- 21 Mar 2019 - "HTCondor-CE Overview", Brian Lin, 2019 Joint HSF/OSG/WLCG Workshop
- 21 Mar 2019 - "OSG Site Installation Overview", Brian Lin, 2019 Joint HSF/OSG/WLCG Workshop
- 19 Mar 2019 - "OSG Software: The Year in Review", Brian Lin, 2019 Joint HSF/OSG/WLCG Workshop
- 3 Dec 2018 - "OSG-LHC and ATLAS", Brian Lin, US-ATLAS Facilities Meeting