How to keep track of docstring coverage of Python packages on CI

Months ago, a collaborator asked for a way to keep track of the docstring coverage of our Python package. We wanted to measure the current coverage of the docstrings in our code and prevent pull requests from decreasing that value.

To solve this problem, a workflow needed to fulfill the following requirements:

The workflow should fail if the coverage is lower than the score of the base branch (on pull requests) or the previous commit (on push), and files that made the job fail should be blamed.

Pushing to main branch should update the current coverage of the project and display the result on a nice badge.

Then, I found the excellent docstr-coverage package that suited most of our needs, and in combination with GitHub Actions, shields.io and jsonbin.org I came up with a solution.

jsonbin

In this case, we use this service to create dynamic badges with the shields.io endpoint. A small JSON dictionary is passed to the endpoint and the badge is created on-the-go. Thus, we avoid to push a new badge to the repository each time the coverage changes.

{
  "schemaVersion": 1,
  "label": "docstr-cov",
  "message": "75%",
  "color": "green"
}

You will need to sign in to jsonbin with your GitHub account and store the token as a repository secret named JSONBIN_APIKEY.

Configuration file

Place a file named .docstr.yml at the root of your repository.

paths:
  - your_package
verbose: 2 # int (0-3)
skip_magic: True
skip_file_doc: True
skip_init: True
skip_class_def: False
skip_private: True
follow_links: True
percentage_only: False

For more information, see the package documentation.

The workflow

This workflow should work out the box. It's not necessary to tweak any environment variable, but you can choose different values for RANGE.

For example, a RANGE of 50..75 means that coverage below 50 will display a red badge, and above 75 a green one. It works exactly like Codecov!

name: docstr-cov

on:
  push:
    branches:
    - main

  pull_request:
    branches:
    - main

env:
  RANGE: 50..75
  ENDPOINT: https://jsonbin.org/${{ github.repository_owner }}/${{ github.event.repository.name }}
  TOKEN: ${{ secrets.JSONBIN_APIKEY }}

At the job level, the steps are easy to follow:

Commits to compare (HEAD and BASE) are chosen depending on the event trigger: HEAD of the branch vs. previous commit on push events, or HEAD of the PR vs. base of the branch on pull_request.

Get coverage score on BASE and HEAD. Fails if HEAD score is lower.

Blame files (if failed).

jobs:
  check:
    runs-on: ubuntu-latest
    steps:

      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.x

      - name: Install docstr-coverage
        run: pip install docstr-coverage

      - name: Get SHAs
        run: |
          if [[ ${{ github.event_name }} == 'push' ]]; then
            echo "BASE=$(git rev-parse HEAD^)" >> $GITHUB_ENV
            echo "HEAD=$(git rev-parse HEAD)" >> $GITHUB_ENV

          elif [[ ${{ github.event_name }} == 'pull_request' ]]; then
            echo "BASE=${{ github.event.pull_request.base.sha }}" >> $GITHUB_ENV
            echo "HEAD=${{ github.event.pull_request.head.sha }}" >> $GITHUB_ENV

          else
            echo "Unexpected event trigger"
            exit 1
          fi

      - name: Get base coverage
        run: |
          git checkout $BASE
          echo "BASE_COV=$(docstr-coverage -p)" >> $GITHUB_ENV

      - name: Test head coverage
        run: |
          git checkout $HEAD
          docstr-coverage --fail-under=$BASE_COV

      - name: Blame
        run: |
          git diff --name-only $(git merge-base $BASE $HEAD) | \
          xargs docstr-coverage --accept-empty
        if: failure()

Only on push events:

Get coverage (rounded)

Set the color label according to RANGE.

Post results to jsonbin.

Make the endpoint public.

Print the badge URL.

- name: Get new coverage
        run: echo "NEW_COV=$(printf "%.f" $(docstr-coverage -p))" >> $GITHUB_ENV
        if: always() && github.event_name == 'push'

      - name: Set label color
        run: |
          if [[ $NEW_COV -ge $(echo {${{ env.RANGE }}} | awk '{print $NF;}') ]]; then
            echo "COLOR=green" >> $GITHUB_ENV
          elif [[ $NEW_COV -lt $(echo {${{ env.RANGE }}} | awk '{print $1;}') ]]; then
            echo "COLOR=red" >> $GITHUB_ENV
          else
            echo "COLOR=orange" >> $GITHUB_ENV
          fi
        if: always() && github.event_name == 'push'

      - name: Post results
        run: |
          curl -X POST $ENDPOINT/badges/docstr-cov \
          -H "authorization: token $TOKEN" \
          -d "{ \"schemaVersion\": 1, \"label\": \"docstr-cov\", \
                \"message\": \"$NEW_COV%\", \"color\": \"$COLOR\" }"
        if: always() && github.event_name == 'push'

      - name: Set public endpoint
        run: |
          curl -X PUT $ENDPOINT/_perms -H "authorization: token $TOKEN"
        if: always() && github.event_name == 'push'

      - name: Show badge URL
        run: echo "https://img.shields.io/endpoint?url=$ENDPOINT/badges/docstr-cov"
        if: always() && github.event_name == 'push'

The badge should be updated dynamically on every push to main. Remember to add it to your README.md.

Get the code

epassaro / docstr-cov-workflow

Measure docstring coverage of Python packages with GitHub Actions

docstr-cov-workflow

Measure docstring coverage of Python packages with GitHub Actions

Usage

Copy .github/workflows/docstr-cov.yml and .docstr.yaml to your repository.
Tweak the configuration file following the package documentation.
Login to https://jsonbin.org and store the API key as a repository secret named JSONBIN_APIKEY.
The workflow will fail if the coverage is lower than the score of the base branch (on pull_request) or the previous commit (on push), and files that made the job fail will be blamed.
Pushing to main branch updates the current coverage of the project by updating a nice badge.
You can change the color range of the badge by tweaking the RANGE variable at the top of the workflow. For example, 50..75 means that coverage below 50 will display a red badge, and above 75 a green one.
Remember to add the badge to your README.md