28
How To Manage Open edX Environment Variables Using Doppler and Automating The Deployment
Secrets are a pain to manage and If you are using Ansible to maintain your application and its components there is a better way to do it. In this article, we see how Doppler can help us to simply and quickly manage environment variables for a complex system called Open edX.
By default, to install and maintain the Open edX we use Ansible. For example, in order to change SMTP credentials, you need to modify an Ansible variables file, Then you should deploy it to your target machine and manually restart services in your server to apply the change.
The automation of this process will allow us to speed up the tedious work and save time. By hosting our variables in Doppler, reading them via API in our codebase, and automating the deployment using Doppler webhook.
By default, to install and maintain the Open edX we use Ansible. For example, in order to change SMTP credentials, you need to modify an Ansible variables file, Then you should deploy it to your target machine and manually restart services in your server to apply the change.
The automation of this process will allow us to speed up the tedious work and save time. By hosting our variables in Doppler, reading them via API in our codebase, and automating the deployment using Doppler webhook.
We use Ansible to provision and maintain our platform. If you are not familiar with Ansible, it's an open source Devops tool that automates the software provisioning and configuration. It building blocks are:
We have our playbooks, roles and their variables in a GitHub repo and we used Ansible vault to encrypt the variables, to prevent exposing our credentials in GitHub.
After encrypting it using vault it looks like the following:
After encrypting it using vault it looks like the following:
$ANSIBLE_VAULT;1.1;AES256
61353035366436396262643237303063643839653630393261663234666461653566626130613562
.
.
.
After Open edX got provisioned by Ansible all of it's environment variables live as a plain text in the file system on Ubuntu server. The file called
lms.yml
and it contains all the environment variables for our Learning Management System, for example:EMAIL_BACKEND: django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST: smtp.elasticemail.com
EMAIL_HOST_PASSWORD: ********
EMAIL_HOST_USER: ******
EMAIL_PORT: 2525
EMAIL_USE_TLS: true
To change a variable we need to, Decrypt the variables file in our repo, Change a variable there and Encrypt the file again and deploy it using Ansible, after a successful deployment the equivalent variable get changed in the
lms.yml
on the server . One extra step to apply this change is to SSH to the server and restart all the services manually.There are multiple issues with this implementation:
Doppler can help us to address all these 3 issues.
Before using Doppler this is how the environment management looks like

After integrating Doppler with our platform, this is how the secrets management look like.

In this article we create one environment variable in Doppler but the process is the same for all of our environment variables that we want to manage in from Doppler.
In my account I created a project called
openedx
with 3 environments
I created one environment variable called
EMAIL_HOST_PASSWORD
to manage the SMPT password. The value starts with aohRj******
This is not my real SMTP password it's only an example
Our application is written in Python/Django so I used Doppler API for this integration.
I integrated our codebase with Doppler, So instead of reading the
I integrated our codebase with Doppler, So instead of reading the
EMAIL_HOST_PASSWORD
from the local environment variables file it makes a call to Doppler API and gets the value for EMAIL_HOST_PASSWORD
. I added the following to our codebase in the edx-platform/lms/envs/production.py
file################################### Getting Environment Variables From Doppler ###################################
url = "https://api.doppler.com/v3/configs/config/secrets"
querystring = {"project": "openedx", "config": "dev"}
DOPPLER_TOKEN = AUTH_TOKENS.get('DOPPLER_TOKEN', '')
Authorization = "Basic {DOPPLER_TOKEN}".format(DOPPLER_TOKEN=DOPPLER_TOKEN)
headers = {
"Accept": "application/json",
"accepts": "application/json",
"Authorization": Authorization
}
doppler_response = requests.request("GET", url, headers=headers, params=querystring)
EMAIL_HOST_PASSWORD = doppler_response.json()['secrets']['EMAIL_HOST_PASSWORD']['raw']
and removed
EMAIL_HOST_PASSWORD = AUTH_TOKENS.get('EMAIL_HOST_PASSWORD', '')
from our code to prevent reading the value from local environment variables. You can see the git diff here
>>> from django.conf import settings
>>> settings.EMAIL_HOST_PASSWORD
'aohRj******'
As you can see Django settings is returning the Value we set in Doppler for
EMAIL_HOST_PASSWORD
After integrating our codebase with Doppler we improved the security of our platform by hosting our variables in safe way but when we change a variable in Doppler we still need to SSH to our server and manually run a command like
sudo /edx/bin/supervisorctl restart all
to restart all the services to get up to date values for our secrets from Doppler. To remove this manual step we can use Doppler webhooks, so as soon as there is a change in one of our variables, we make a call to CircleCI to restart all the services in our server.In CircleCI, I created a project called
doppler-openedx-deployer
. This project has a config.yml
file like following :version: 2
jobs:
build:
working_directory: ~/doppler-openedx-deployer
docker:
- image: circleci/python:3.6.4
steps:
- checkout
- run:
name: Deploy to Open edX
command: ~/doppler-openedx-deployer/scripts/deploy.sh
no_output_timeout: 30m
By running
deploy.sh
we download the latest version of our Ansible Playbooks and it deploys a command to Open edX server to update configuration and restart all the services.
In Doppler we can go to the webhooks section in projects and our endpoint there.

The endpoint here is a lambda function that is using CircleCI API to trigger a build.
28