Modern Python setup for quality development

In this article, i'll show you what is my common python setup.

Goals are:

High quality code standard
Respect of the python standard
Dev experience quality
Security
repeatability

We'll see:

The IDE
Python versions
Package and virtual environments
Code linter/formatter
Tests tools
Documentation
Pre-commit
Security

This is entirely subjective :)

IDE: VSCode

VSCode is a great IDE for python programming.

One of his best feature is his extensibility though extensions.

Here are some extensions i used:

For the general development environment

Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.
Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:
vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.
Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.
GitLens: Now you can see who wrote that directly into the code

For data parsing/formatting/transformation...

There's lots of extensions for that depending on your use case (docker, helm, etc...)

I think these one are required by pretty much every project:

Better TOML
change-case
Code Spell checker
JSON Tools
XML Tools
YAML
vscode-base64
Sort JSON objects

Python versions: Pyenv

Python package and venv: Poetry

Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.

Code linter/formatter

flake8: Flake8 is a wrapper around these tools: PyFlakes pycodestyle Ned Batchelder's McCabe script

Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.

black: Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.

Black makes code review faster by producing the smallest diffs possible.

isort: isort your imports, so you don't have to.

For the tests

pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)
tox: The first point of tox is to run your tests into multiple environment (multiple python version). I'm not a big fan of it however it does the job. the nox is a modern alternative but i never took the time to try it.

For the documentation: Sphinx

Sphinx allow you to write documentation using rst or markdown format. Theme management and automatic code introspection are some great features of it. You can easily integrate your documentation to your CI/CD pipeline for automatically test (using doc8) then deploy it.

Pre-commit

This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically You can perform a TONS of tests/validation before committing anything.

This is basically a local CI pipeline run before the real one.

Here is an example of one .pre-commit-config.yaml config file i use, among others it will:

sort your import automatically
black will format your code
flake8 will warn you about style issue
Validate .gitlab-ci.yaml against my enterprise gitlab server.
check style for bash script
and so much more...

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: check-added-large-files
      - id: check-ast
      - id: check-builtin-literals
      - id: check-case-conflict
      - id: check-docstring-first
      - id: check-executables-have-shebangs
      - id: check-json
      - id: check-merge-conflict
      - id: check-symlinks
      - id: check-toml
      - id: check-vcs-permalinks
      - id: check-xml
      - id: check-yaml
        args: [--allow-multiple-documents]
      - id: debug-statements
      - id: detect-aws-credentials
        args: [--allow-missing-credentials]
      - id: destroyed-symlinks
      - id: end-of-file-fixer
      - id: fix-byte-order-marker
      - id: fix-encoding-pragma
        args: [--remove]
      - id: forbid-new-submodules
      - id: mixed-line-ending
        args: [--fix=auto]
      - id: name-tests-test
        args: [--django]
      - id: requirements-txt-fixer
      - id: trailing-whitespace
  - repo: local
    hooks:
      - id: black
        name: black
        entry: poetry run black
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]
  - repo: https://github.com/pycqa/isort
    rev: "5.9.1"
    hooks:
      - id: isort
        args:
          - --profile
          - black
          - --filter-files
  - repo: https://github.com/adrienverge/yamllint.git
    rev: v1.26.1
    hooks:
      - id: yamllint
        args: [-c=.yamllint.yaml]
  - repo: https://gitlab.com/devopshq/gitlab-ci-linter
    rev: v1.0.2
    hooks:
      - id: gitlab-ci-linter
        args:
          - "--server"
          - "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v2.17.11
    hooks:
      - id: commitizen
        stages: [commit-msg]
  - repo: https://github.com/jumanjihouse/pre-commit-hooks
    rev: 2.1.5 # or specific git tag
    hooks:
      - id: forbid-binary
      - id: shellcheck
      - id: shfmt

Security

bandit: Bandit is a tool designed to find common security issues in Python code.

To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

safety: Safety checks your installed dependencies for known security vulnerabilities.

By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.

gitleaks: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

What would i change

I'm not a big fan of:

tox
sphinx

The job is one but if someone have better alternative, i would try them.

Conclusion

I hope you leaned a thing or two in this article.

Using these tools can help you falling in some common pitfall.

Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.