Package release process

The package release process has four stages. The last two steps are automated:

  • Merge changes from the develop branch onto main that will form the new release version.

  • Publish a release on GitHub - this is basically just a specific tagged commit on main that has some associated release notes.

  • Publish the code on Zenodo.

  • Publish the built code packages to PyPI - this is the packaged version of the code that users will install and use. The pyrealm package uses the trusted publishing mechanism to make it easy to add new release to PyPI.

Generate the code commit to be released

The release process for new versions of the pyrealm package is managed using pull requests to the main branch to create a specific commit that will be released. The steps of the process are:

  1. Create a new release branch from the develop branch called release/X.Y.Z , where X.Y.Z is the expected release version number.

  2. Update the pyproject.toml file to use the expected release versions number and commit that change. You can use poetry version command to increment the major, minor and patch version but it is almost as easy to edit the file by hand.

  3. That commit should set the standard pyrealm_ci.yaml actions going, which includes code QA, testing and docs building. However, you should also check the documentation builds on Read The Docs.

    Log in to https://readthedocs.org which is the admin site controlling the build process. From the Versions tab, activate the release/X.Y.Z branch and wait for it to build. Check the Builds tab to see that it has built successfully! If it has built successfully, do check pages to make sure that page code has executed successfully, and then go back to the Versions tab and deactivate and hide the branch. If the release branch needs any changes, do come back and check that those changes have also built successfully.

  4. Start a pull request against the main branch. The PR will transfer all of the changes to the develop branch since the last release on to the main branch. The PR description should provide a good explanation of the functionality that is being changed or added in this version, and an explanation of the suggested version number increment. For example, “This PR fixes a bug in calculating plant growth and so is a patch release from v.0.1.8 to v0.1.9”.

  5. The CI testing obviously now needs to pass. Any issues need to be resolved by commits or PRs onto the release/x.y.x branch.

  6. The PR also must be reviewed. The code itself has already gone through the review process to be merged into develop, so this is not a code review so much as a review of the justification for a release.

  7. The branch can then be merged into main. Do not delete the release branch at this point.

  8. Create a second PR to merge the release branch into develop. This is to synchronise any release changes (including the version number change) between the main and develop branches.

Create the GitHub release

The head of the main branch is now at the commit that will be released as version X.Y.Z. The starting point is to go to the draft new release page. The creation of a new release is basically attaching notes and files to a specific commit on a target branch. The steps are:

  1. On that release page, the release target dropdown should essentially always be set to main: the whole point in this branch is to act as a release branch.

  2. You need to provide a tag for the commit to be released - so you need to tag the commit on the main branch using the format vX.Y.Z. You can:

    • Create the tag locally using git tag vX.Y.Z and then push the tag using git push --tags. You can then select the existing tag from the drop down on the release page.

    • Alternatively, you can simply type the tag name into that drop down and the tag will be created alongside the draft release.

  3. You will need to choose a title for the release: basically Release vX.Y.Z is fine. However, the title text also provides a mechanism for suppressing automatic trusted publication to the main PyPI server by using Release vX.Y.Z test-pypi-only. See below for details.

  4. You can create release notes automatically - this is basically a list of the commits being added since the last release - and can also set the version as a pre-release. This is different from having an explicit release version number (e.g. X.Y.Za1) - it is just a marker used on GitHub.

    At this point, you can either save the draft or simply publish it. It is probably good practice to save the draft and then have a discussion with the other developers about whether to publish it.

  5. Once everyone is agreed publish the release.

Publish the package on Zenodo

The pyrealm package is set up to publish on Zenodo automatically when a release is published. This saves a zip file of the repository as a new Zenodo record, under the pyrealm concept ID.

This process can be turned off for test releases but only through the owning Zenodo account, which belongs to David Orme.

Publish the package on PyPI

We publish to two package servers:

  • The TestPyPI server is a final check to make sure that the package build and publication process is working as expected.

  • The package builds are then published to the main PyPI server for public use.

The GitHub action automates the publication process but the process can also be carried out manually.

Manual publication

The publication process can be carried out from the command line. The manual process looks like this:

# Use poetry to create package builds in the dist directory
poetry build
# Building pyrealm (x.y.z)
#  - Building sdist
#  - Built pyrealm_x.y.z.tar.gz
#  - Building wheel
#  - Built pyrealm_x.y.z-py3-none-any.whl

# Use twine to validate publication to TestPyPI
twine upload --repository testpypi --config-file .pypirc dist/*

# Use twine to publish to PyPI
twine upload --repository pypi --config-file .pypirc dist/*

The tricky bit is that you need to provide a config file containing authentication tokens to permit publication. Those tokens must not be included in the repository and so need to be carefully shared with developers who make releases. If this is being used the .pypirc file contents will look something like:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-longAuthToken


[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-anotherLongAuthToken

At this point, you should also upload the built package wheel and source files to the assets section of the GitHub release.

Trusted publishing

The pyrealm repository is set up to use trusted publishing through a Github Actions workflow. The workflow details are shown below, along with comments, but the basic flow is:

  1. When a GitHub release is published, the PyPI publication workflow is triggered.

  2. The standard continuous integration tests are run again, just to be sure!

  3. If the tests pass, the package is built and the wheel and source code are stored as job artefacts.

  4. The built files are automatically added to the release assets.

  5. The job artefacts are published to the Test PyPI server, which is configured to automatically trust publications from this GitHub repository.

  6. As long as all the steps above succeed, the job artefacts are now published to the main PyPI site, which is also configured to trust publications from the repository.

    The last step of publication to the main PyPI site can be skipped by including the text test-pypi-only in the title text for the release. This allows pre-release tests and experimentation to be tested without automatically adding them to the official released versions.

name: Publishing

on:
  release:
    types: [published]

jobs:
  # First, run the standard test suite - for this to work correctly, the workflow needs
  # to inherit the organisation secrets used to authenticate to CodeCov.
  # https://github.com/actions/runner/issues/1413
  test:
    uses: ./.github/workflows/pyrealm_ci.yaml
    secrets: inherit

  # Next, build the package wheel and source releases and add them to the release assets
  build-wheel:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Build the package - this could use `poetry build` directly but pyproject.toml
      # already has the build-system configured to use poetry so `pip` should pick that
      # up automatically.
      - name: Build sdist
        run: |
          python -m pip install --upgrade build
          python -m build

      # Upload the build outputs as job artifacts - these will be two files with x.y.z
      # version numbers:
      # - pyrealm-x.y.z-py3-none-any.whl
      # - pyrealm-x.y.z.tar.gz
      - uses: actions/upload-artifact@v4
        with:
          path: dist/pyrealm*

      # Add the built files to the release assets, alongside the repo archives
      # automatically added by GitHub. These files should then match exactly to the
      # published files on PyPI.
      - uses: softprops/action-gh-release@v1
        with:
          files: dist/pyrealm*

  # Now attempt to publish the package to the TestPyPI site, where the pyrealm project
  # has been configured to allow trusted publishing from this repo and workflow.
  #
  # The skip-existing option allows the publication step to pass even when the release
  # files already exists on PyPI. That suggests something has gone wrong with the
  # release or the build file staging and the release should not be allowed to continue
  # to publish on PyPI.

  publish-TestPyPI:
    needs: build-wheel
    name: Publish pyrealm to TestPyPI
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      # Download the built package files from the job artifacts
      - name: Download sdist artifact
        uses: actions/download-artifact@v4
        with:
          name: artifact
          path: dist

      # Information step to show the contents of the job artifacts
      - name: Display structure of downloaded files
        run: ls -R dist

      # Use trusted publishing to release the files downloaded into dist to TestPyPI
      - name: Publish package distributions to TestPyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: https://test.pypi.org/legacy/
          # skip-existing: true

  # The final job in the workflow is to publish to the real PyPI as long as the release
  # name does not contain the tag 'test-pypi-only'
  publish-PyPI:
    if: ${{ ! contains(github.event.release.name, 'test-pypi-only')}}
    needs: publish-TestPyPI
    name: Publish pyrealm to PyPI
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      # Download the built package files from the job artifacts
      - name: Download sdist artifact
        uses: actions/download-artifact@v4
        with:
          name: artifact
          path: dist

      # Information step to show the contents of the job artifacts
      - name: Display structure of downloaded files
        run: ls -R dist

      # Use trusted publishing to release the files downloaded into dist to PyPI
      - name: Publish package distributions to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1