Deploying Django Rest Framework with Postgres on fly.io

Background

In this post, I will be demonstrating how to deploy a Django Rest Framework (DRF) application on fly.io. DRF is built on top of Django and is my choice when it comes to building small to massive APIs. I have used it extensively professionally and personally, and as a cybersecurity researcher, it's been my favorite to build as a backend for some complex applications that track malicious actors and their infrastructure. I noticed the fly website did not have a Django example, so I wanted to provide the community with a template to get started to avoid the headaches that I endured building this :)

You should have an understanding of Docker, docker compose, DRF and postgres for this post. Although it's possible to use this template without too much knowledge of these concepts, you'd get a ton of knowledge going through the following tutorials:

Why DRF?

From the DRF website:

Some reasons you might want to use REST framework:

-  The Web browsable API is a huge usability win for your developers.
-  Authentication policies including packages for OAuth1a and OAuth2.
-  Serialization that supports both ORM and non-ORM data sources.
-  Customizable all the way down - just use regular function-based views if you don't need the more powerful features.
-  Extensive documentation, and great community support.
-  Used and trusted by internationally recognised companies including Mozilla, Red Hat, Heroku, and Eventbrite.

The biggest issue with DRF is that it can be somewhat challenging to get up and running (can be lots of up front work). My experience has been that once you get a good template for an API up, it's the fastest to build, most scalable and most intuitive ORM.

Why fly.io?

I've used Heroku for one-off apps for a few years. Fly came across my Twitter feed and I've been following it closely. The company has a great section on why you should use fly, and this section caught my eye and has kept me interested ever since I read it:

Despite the benefits of location-smart, time-agile and cloud-clever applications, there’s been no good platform for building applications that work like this. This is what Fly has set out to fix. In the process we want to make application distribution platforms as ubiquitous as CDNs.

You can think of fly as a Heroku competitor, although some folks might disagree with me. I like it because it does what it says it does well, is focused, and isn't as bloated as the Heroku stack.

Getting started

Bookkeeping

We'll be making a DRF app, the Silly Simple API, or ss-api for short, on fly. This will have the following features:

  • Postgres backend, courtesy of fly
  • No session authentication, only using Tokens (you can use some tricks from fly to manage this). This is especially nice for APIs and simplifies authentication to a token, which I prefer for microservices
  • Swagger and OpenAPI capabilities using drf-yasg, where you can only see endpoints and Swagger docs if you have a valid Token
  • TCP & HTTP health checks using fly. The HTTP health check will be somewhat useful by issuing a query to our DRF app under /ping/, which connects to the DB and issues an innocuous select 1 statement to make sure things are working
  • Using docker with a Dockerfile and gunicorn to launch the app. The cool thing about fly is that you can give it a Dockerfile and a fly.toml and you have a full-fledged app running on their infrastructure

This app will only have 1 endpoint, users, that you can use to manage your users. You must be authenticated to see it.

I will leave the following for later blog posts:

  • Metrics exposure and app tracing via Datadog (p.s., we're hiring https://www.datadoghq.com/careers/)
  • Tables for an app (this isn't a tutorial on building DRF apps, rather, to get you a template to get started)
  • Scaling primitives in DRF (avoiding n+1, indexing, replicas for postgres)

If this all sounds interesting for you still, let's get started :D

Clone and run locally

Make sure to have the following installed locally:

  • Latest Docker (with docker compose)
  • git
  • fly via here

then clone from the repo here:

git clone [email protected]:zmallen/ss-api.git

Run docker compose up and connect locally by navigating to:

http://localhost:8000

The docker-compose.yml file overrides the RUN command in the Dockerfile by issuing the following command on every docker compose up:

command: bash -c "python manage.py migrate && python manage.py runserver 0.0.0.0:8000"

This will differ from when we deploy on fly.io, where we use gunicorn to serve the app and we run migrations manually.

Token Auth & Authentication in DRF

Unauthenticated users can only see the /ping/ endpoint on Swagger. This is by design - the app will render endpoints based on permission, and under /ping/views.py on Line 16, the permission for this endpoint is:

permission_classes = (AllowAny,)

Compare this to ssapi/views.py, under the UserViewSet on Line 17:

authentication_classes = (authentication.TokenAuthentication,)

This is achieved via some magic in settings.py Lines 88-111:

# DRF settings
REST_FRAMEWORK = {
    "DEFAULT_PERMISSION_CLASSES": ("rest_framework.permissions.IsAuthenticated",),
    "DEFAULT_AUTHENTICATION_CLASSES": (
        "rest_framework.authentication.TokenAuthentication",
    ),
    "DEFAULT_RENDERER_CLASSES": ("rest_framework.renderers.JSONRenderer",),
}

# SWAGGER_SETTINGS
SWAGGER_SETTINGS = {
    "USE_SESSION_AUTH": False,
    "LOGIN_URL": "rest_framework:login",
    "LOGOUT_URL": "rest_framework:logout",
    "VALIDATOR_URL": None,
    "SECURITY_DEFINITIONS": {
        "api_key": {
            "type": "apiKey",
            "name": "Authorization",
            "in": "header",
        },
    },
    "REFETCH_SCHEMA_WITH_AUTH": True,
}

Under DRF_SETTINGS, I forced TokenAuthentication and isAuthenticated for viewing endpoints, so no more sessions! 🎉🎉

So how do you get an API token if you can't authenticate? This is where some magic with manage.py comes into play.

Authenticating - getdevtoken and /users/

To get a local API key, run the following command in a separate tab, in the same directory as docker-compose.yml:

â””> docker compose run api python manage.py getdevtoken
[+] Running 1/0
 â ¿ Container ss-api-db-1  Running                                          0.0s
Looking for superuser..
superuser doesn't exist, creating!
superuser created!
Use the following key for dev:
(｡◕‿‿◕｡)☞☞  Token 9884a551be31b80a61b49becf7c3640224a9ec42  ☜☜(｡◕‿‿◕｡)
  • Note, this Token is for my local deployment :)

Copy the Token abc and navigate over to your browser, then click 'Authorize', paste and press submit. You should get the /users/ endpoint to return in the Swagger frontend, and issuing a GET request will list the superuser!

You can do a lot more with Swagger documentation than just the defaults, I suggest checking out these resources to learn about Swagger docs in Django:

Deploying to fly

Everything is running smoothly in your local environment, now let's get it to a prod environment!

First, we need to create a toml file for fly. This is a configuration file used by fly to deploy your app. I generated one within the Github repo, but you can explore how to generate and configure other toml files on fly here.

A few things need to happen to finish our "deploy to prod" for ss-api:

  1. Create a fly app
  2. Create a postgres db with fly, and retrieve the DATABASE_URL string
  3. Set a fly secret with the DATABASE_URL from Step 1 so our app can dynamically render the secret in a fly environment and use the db created in step 1
  4. Deploy the app on fly, make sure TCP & HTTP health checks pass (they wont on first pass :D)
  5. Create a database and run a migration using fly ssh
  6. Get a devtoken for prod

Step 1: Create your fly app

Simply run fly create and name your app!

â””> fly create
? App Name: ssapiblog
automatically selected personal organization: Zack Allen
New app created: ssapiblog

Step 2: postgres

Launch a new postgres instance via fly with fly postgres create. Accept all the defaults (minimal DB settings, aka the cheapest!) and wait for fly to give you the DATABASE_URL.

â””> fly postgres create
? App Name: ssapidb
Automatically selected personal organization: Zack Allen
? Select region: iad (Ashburn, Virginia (US))
? Select VM size: shared-cpu-1x - 256
? Volume size (GB): 10
Creating postgres cluster ssapidb in organization personal
Postgres cluster ssapidb created
  Username:    postgres
  Password:   SECRETPASSWORD
  Hostname:    ssapidb.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Monitoring Deployment
...
...
2 desired, 2 placed, 2 healthy, 0 unhealthy [health checks: 6 total, 6 passing]
--> v0 deployed successfully

Connect to postgres
Any app within the personal organization can connect to postgres using the above credentials and the hostname "ssapidb.internal."
For example: postgres://postgres:[email protected]:5432

See the postgres docs for more information on next steps, managing postgres, connecting from outside fly:  https://fly.io/docs/reference/postgres/

You want the 12factor string after For example:, which in this example is:
postgres://postgres:[email protected]:5432

Step 3: Set DATABASE_URL as a fly secret

You want to set the DATABASE_URL with a 12factor string from before, as well as a database name (which we will create).

Note the /ssapidb at the end of the DATABASE_URL

fly secrets set DATABASE_URL="postgres://postgres:[email protected]:5432/ssapidb"

Step 4: Deploy your app

Change the following line in fly.toml to whatever you want:

app = "ssapiblog"

Run fly deploy:

â””> fly deploy 
Deploying ssapiblog
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8000
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.8 linux x86_64
Sending build context to Docker daemon  153.1kB
...
==> A bunch of Docker output

You can detach the terminal anytime without stopping the deployment
Monitoring Deployment

v0 is being deployed
2021-12-30T22:26:21.000 [info] 145.40.89.203 - - [30/Dec/2021:22:26:21 +0000] "GET /ping/ HTTP/1.1" 500 114326 "http://172.19.10.66:8000/ping" "Consul Health Check"

1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 2 total, 1 passing, 1 critical]

Step 5: Build a DB then migrate

Notice how 1 health check is passing (tcp), and 1 is critical. This is because we did not do a database migration and our /ping healthcheck is failing. This will most likely fail after a certain amount of time, so in a separate tab navigate to the project directory to run a few fly ssh commands.

First, make the ssapidb database by running a handy dandy bash script I added into the repo. We can use this via fly ssh console -C command:

â””> fly ssh console -C 'bash /app/provision_db.sh'
Connecting to ssapiblog.internal... complete
Database does not exist. Creating now..
CREATE DATABASE

If you run fly logs in a separate tab, you should see Consul health checks returning 200, which is healthy \o/:

2021-12-30T22:31:27.116 app[3d7bc4b5] iad [info] 145.40.89.203 - - [30/Dec/2021:22:31:27 +0000] "GET /ping HTTP/1.1" 301 0 "-" "Consul Health Check"
2021-12-30T22:31:27.119 app[3d7bc4b5] iad [info] 145.40.89.203 - - [30/Dec/2021:22:31:27 +0000] "GET /ping/ HTTP/1.1" 200 2 "http://172.19.10.66:8000/ping" "Consul Health Check"

Let's migrate and get a devtoken:

â””> fly ssh console -C 'python /app/manage.py migrate'
Connecting to ssapiblog.internal... complete
Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying authtoken.0001_initial... OK
  Applying authtoken.0002_auto_20160226_1747... OK
  Applying authtoken.0003_tokenproxy... OK
  Applying sessions.0001_initial... OK

Step 6: Get your devtoken and open the app 

â””> fly ssh console -C 'python /app/manage.py getdevtoken'
Connecting to ssapiblog.internal... complete
Looking for superuser..
superuser doesn't exist, creating!
superuser created!
Use the following key for dev:
(｡◕‿‿◕｡)☞☞  Token TOKEN  ☜☜(｡◕‿‿◕｡)

Woot! Run fly open and go through the same workflow as your local deployment: put Token TOKEN into the Authorize panel, and you can now see the authenticated users endpoint, and issue a GET request to get your token!

If you want to add more users, just use the POST request endpoint here to create a new user.

Conclusion

I enjoyed writing this app and this blog post! Fly is a cool concept and I will definitely play with it more. There are some sharp edges with DRF, so I tried to simplify it, but please study the DRF tutorials and the api/settings.py file for other configuration options I used.

For my next posts, Im looking to develop an app to do some basic cybersecurity threat intelligence tracking and correlation. If you have ideas for other apps, or have a question on this app, please leave a comment or open an issue on the ss-api repo here:

29

Categories:

DjangoPython

Tags:

DjangoFlyioProgrammingPython