22
Ease your development hurdles with these [M]ake recipes
- Introduction
-
Some Recipes/Tasks for your backend projects
- Build the project
- Clean up the workspace
- Run the project
- Prepare the local environment for development
- Reset the database
- Freeze and unfreeze the state of the database
- Update non-local environments
- File generation scripts
- Run arbitrary command
- Migrate database
- Check code
- Bump version
- Run unit/acceptance tests
- Open the profiling tool
- Conclusions
Make and other similar expert/automation systems like PyInvoke (I highly love this one over other build systems, even over Makefile), Rake, Cake, etc... avoid the necessity of having a bunch of scripts (bash, python...) to perform certain repetitive tasks. Even then, you often chain them in succession to achieve a certain goal (e.g. you know you have to run script_1.sh before script_2.sh, but that dependency is probably not made explicit in the project file structure). These dependencies are difficult to remember as they become more and more complex.
Or you might install a whole bunch of other tools for your project: poetry, gulp or docker just to name a few. Eventually recalling the arguments required to perform a certain action becomes difficult. But it's never difficult to recall the action you need to perform.
Dumb metaphor: It's easier to remember that you need an spanish omelette than remember that you need eggs, potatoes, onions and salt to build such delicious dish. Wouldn't it be easier if you had the power to imagine an omelette and PUFF!, it materializes in front of you, than building the omelette by yourself each and every time? That's where make and other derivatives can help you.
Here I present you with several suggested recipes that I use on a daily basis on the backend projects I have been working. The example are expressed in Makefile syntax, but the very idea can be applied to every build system.
Pretty self explanatory. Every project of a reasonable size and maturity needs to be built, either by compilation, building the docker image, you name it. Therefore, the responsibility of this recipe should be to build/update the project and leave it in a runnable state. Optionally it could run migrations as well to keep the database schema up to date.
Make recipe:
build:
# Put here the commands to compile, install dependencies, build docker images...Invocation:
make buildThis one is the inverse of build. Its responsibility is to clean up the docker images, volumes, generated files... created by the build command.
Make recipe:
clean:
# Clean docker images, dist files, etc...Invocation:
make cleanThis one runs the project. Prior to spinning up the application (running docker-compose up, executing the main script, etc.), it could prepare the environment to ensure that the project starts up successfully: invalidating caches, creating an AWS session, running database migrations, etc...
This is the command we tell the frontend team to run after building the application with make init to spin up the web server for theirs to run their HTTP requests.
Make recipe:
run:
# Run your project here
stop:
# In the same sense that there is a run, it might be interesting for you to have a stop command that kills the process/es that starts up the `run` command.Invocation:
make run
make stopThis one is intended for those that are going to do development over the project. It installs dependencies locally, set up pre-commit hooks, etc... basically setup and install anything required for local development.
Make recipe:
setup-dev:
# Run pre-commit scripts, install dependencies locally...Invocation:
make setup-devEventually and through your testing or the testing of your colleague developers, the database is going to be heavily polluted, and this command should reset the database to factory settings (destroying and creating the local database, re-running the initial data migrations, etc.)
Make recipe:
reset: down
# Run db scripts to purge the DB/s, remove docker volumes...Invocation:
make resetI came up with these after some people of my team suggested that it would be cool, from a testing perspective, if we could save a snapshot of the database and then restore it. It proved very useful.
Make recipe:
# Example with postgres, leveraging docker
freeze: down
docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup backup postgres
unfreeze: down
docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup restore postgresInvocation:
make freeze
make unfreezeOften non-local environments have different characteristics than your machine (they might not run via docker, for example). Having a recipe to handle migrations and updates to those environments is very useful to avoid forgetting to run something.
Make recipe:
update-remote:
# call here the commands that are intended to be called when you deploy to your remote environment: Database migrations, cache invalidations, scripts to fix something...Invocation:
make update-remoteSometimes you might need to generate files in your code (documentation, protobuf code, etc...). These kind of commands are intended for such tasks.
Make recipe:
generate-docs:
# call your doc generation tool (doxygen, sphynx...)Invocation:
make generate-docsI work with Django (but this probably applies to other frameworks as well). And wrapping these commands into a recipe is often a good idea to keep the execution centralized.
Make recipe:
run-command:
# e.g. python src/manage.py ($cmd)Invocation:
make run-command cmd="XXXXX"This recipe usually is a dependency of another of the previously mentioned recipes like init and up. As the name implies, it applies the migrations to the database.
Make recipe:
migrate:
# Run here the command to execute the migrationsInvocation:
make checkAny serious project run some kind of code formatting tool to comply with your company coding guidelines and best practices. This command performs these checks on the code.
Make recipe:
check:
# Example leveraging pre-commit
pre-commit run --all-filesInvocation:
make checkThis recipe updates the version of your project. Each time I commit code the pre-commit hook calls this command with the patch argument, and each time I deploy to production I manually call it with the minor|major argument depending on the type of code change I'm conducting.
Make recipe:
bump-patch:
# Example with bump2version
poetry run bump2version $(LEVEL)
git add _version.py
git add .bumpversion.cfgInvocation:
make bump-patch LEVEL=patchThis recipe runs the tests of my application with some default parameters I like to establish. Sometimes I create other similar recipes to run these very tests but with some quirks, like generating coverage and result reports when running.
Make recipe:
run-tests:
# Call here the command to execute your tests: py.test, NUnit... just make sure that the environment variables needed by your tests are loaded. This is why I like to run my tests with docker compose!Invocation:
make run-testsYou might have been profiling some code to find bottlenecks in your application and generated a report you'd like to visualize with tools like snakeviz or pyprof2calltree. This command will open the latest report generated, ready for visualization.
Make recipe:
open-profiling:
# Example of command, adjust to your preferred visual tool
pyprof2calltree -i profiling/results/$(FILE).prof -kInvocation:
make open-profiling FILE=resultsI exposed several recipes that I found useful when developing. Even though I gave some clues of the tooling you could potentially use, it's impossible to provide a detailed implementation since it wildly depends on the programming language you use, the environment, etc... However, what's really important about each recipe is the main objective it tries to achieve, and that's a fact (building the project, running tests...).
I'd like to hear from other fellow developers what other recipes you have come up with that could make our lives a little bit easier. If you have any to share please, feel free to contribute in the comments section!
22