71
Modern Python setup for quality development
In this article, i'll show you what is my common python setup.
Goals are:
We'll see:
This is entirely subjective :)
VSCode is a great IDE for python programming.
One of his best feature is his extensibility though extensions.
Here are some extensions i used:
For the general development environment
Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.
Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:
vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.
Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.
GitLens: Now you can see who wrote that directly into the code
For data parsing/formatting/transformation...
There's lots of extensions for that depending on your use case (docker, helm, etc...)
I think these one are required by pretty much every project:
Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.
Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.
Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.
Black makes code review faster by producing the smallest diffs possible.
pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)
tox: The first point of tox is to run your tests into multiple environment (multiple python version). I'm not a big fan of it however it does the job. the nox is a modern alternative but i never took the time to try it.
Sphinx allow you to write documentation using rst or markdown format. Theme management and automatic code introspection are some great features of it. You can easily integrate your documentation to your CI/CD pipeline for automatically test (using doc8) then deploy it.
This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically You can perform a TONS of tests/validation before committing anything.
This is basically a local CI pipeline run before the real one.
Here is an example of one .pre-commit-config.yaml config file i use, among others it will:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: check-added-large-files
- id: check-ast
- id: check-builtin-literals
- id: check-case-conflict
- id: check-docstring-first
- id: check-executables-have-shebangs
- id: check-json
- id: check-merge-conflict
- id: check-symlinks
- id: check-toml
- id: check-vcs-permalinks
- id: check-xml
- id: check-yaml
args: [--allow-multiple-documents]
- id: debug-statements
- id: detect-aws-credentials
args: [--allow-missing-credentials]
- id: destroyed-symlinks
- id: end-of-file-fixer
- id: fix-byte-order-marker
- id: fix-encoding-pragma
args: [--remove]
- id: forbid-new-submodules
- id: mixed-line-ending
args: [--fix=auto]
- id: name-tests-test
args: [--django]
- id: requirements-txt-fixer
- id: trailing-whitespace
- repo: local
hooks:
- id: black
name: black
entry: poetry run black
language: system
types: [python]
- id: flake8
name: flake8
entry: poetry run flake8
language: system
types: [python]
- repo: https://github.com/pycqa/isort
rev: "5.9.1"
hooks:
- id: isort
args:
- --profile
- black
- --filter-files
- repo: https://github.com/adrienverge/yamllint.git
rev: v1.26.1
hooks:
- id: yamllint
args: [-c=.yamllint.yaml]
- repo: https://gitlab.com/devopshq/gitlab-ci-linter
rev: v1.0.2
hooks:
- id: gitlab-ci-linter
args:
- "--server"
- "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token
- repo: https://github.com/commitizen-tools/commitizen
rev: v2.17.11
hooks:
- id: commitizen
stages: [commit-msg]
- repo: https://github.com/jumanjihouse/pre-commit-hooks
rev: 2.1.5 # or specific git tag
hooks:
- id: forbid-binary
- id: shellcheck
- id: shfmt
To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.
By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.
I'm not a big fan of:
The job is one but if someone have better alternative, i would try them.
I hope you leaned a thing or two in this article.
Using these tools can help you falling in some common pitfall.
Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.
71