Jun, 21 #CloudGuruChallenge

TL;DR

I completed another #CloudGuruChallenge. This month we were challenged to improve a python app's performance by implementing ElastiCache for Redis.

I thought this challenge was really interesting because application performance is such a big topic in the industry. I've seen Python and Redis used multiple times already.

Unfortunately due to the nature of the services required for this challenge I was unable to take a serverless approach. That also means that I would have already tore down the resources by the time you're reading this. Don't worry though I took some pictures for us to enjoy!

Database Response

Cache Response

If you'd like to checkout the code or connect on LinkedIn then see below.

GitHub
https://github.com/wheelers-websites/CloudGuruChallenge_21.06

LinkedIn
https://www.linkedin.com/in/justin-wheeler-the-cloud-architect/

In addition to the requirements I decided to go above and beyond. If that interests you then keep reading!

Architecture

I decided to manage the architecture for the application and the network separately since the costs vary drastically between both among other reasons.

Network Architecture

VPC
4 Public Subnets
4 Private Subnets
Public Route Table
Private Route Table
Internet Gateway
NAT Gateway
Public Security Group (HTTP and SSH)
Private Security Group (PostgreSQL and Redis)

Application Architecture

EC2 Instance
EC2 KeyPair
ElasticCache Redis Instance
ElasticCache Subnet Group
RDS PostgreSQL Instance
RDS Subnet Group

It's true that this configuration is probably overkill for this simple application. However, I wanted to push myself to design for a production scenario.

Here are some next steps that would improve this architecture that I deemed out of scope.

HTTPS Ingress

update public security group to permit HTTPS instead of HTTP
setup NGINX to use HTTPS

Failover NAT Gateway

setup failover NAT Gateway in another AZ
update private route table to use one or both NAT Gateways

Multiple Application Servers w/ Load Balancer

setup the app to run as part of an ASG instead
setup an ALB/NLB in front of the ASG to load balance traffic

Terraform

Much like in previous projects I decided to use Terraform here for Infrastructure as Code (IaC). It seems that every time I use Terraform I find something new that I like about it.

Below are a list of new found features that I recently learned about and used on this project.

locals block

This feature allows users to define local variables inside their templates.

locals {
  availability_zones = [
    "us-east-1a",
    "us-east-1b",
    "us-east-1c",
    "us-east-1d"
  ]
  cidr_anywhere_block = "0.0.0.0/0"
  ...
}

local.cidr_anywhere_block

file function

This feature allows you to easily pass a file into the resource declaration. That allowed me to bootstrap my EC2 instance while managing that script outside of the template itself. No crazy formatting required here!

resource "aws_instance" "server" {
  ...
  user_data = file("user-data.sh")
}

random_shuffle provider

This feature allows you to shuffle a list of values and return x number of values back. I thought this was far more clever than simply providing a single value and more flexible too!

variable "subnet" {
  description = "subnet ids"
  type = object({
    private = list(string)
    public  = list(string)
  })
}

subnet = {
  ...
  public  = [
    "subnet-01b36f97fa8490e43",
    "subnet-0a4272a81af160127",
    "subnet-03939599dd92f0f07",
    "subnet-0b6f2c6a35b31e4d5"
  ]
}

resource "random_shuffle" "public_subnet" {
  input        = var.subnet.public
  result_count = 1
}

subnet_id = random_shuffle.public_subnet.result[0]

count meta-argument

This feature allows you to provide looping behavior to your templates so you don't need to repeat yourself.

resource "aws_subnet" "public" {
  count = length(local.cidr_public_blocks)
  cidr_block        = local.cidr_public_blocks[count.index]
  ...
}

Flask, Gunicorn, and Styles

I changed the default application that was provided by David Thomas to use some new found knowledge on related topics.

https://github.com/ACloudGuru/elastic-cache-challenge

Jinja Expressions

This allowed me to put logic into my HTML templates similar to ngIf for Angular. I mainly did this because when testing I didn't want to see empty sections drawn on the screen. Now they will be hidden if the values are not passed into the render_template function.

{% if db_host %}
<div class="section">
    <span class="section-title">
        Database Host
    </span>
    <span>{{ db_host }}</span>
</div>
{% endif %}

Furthermore, this allowed me to add an error display on the screen that is hidden if there is no error. Since I know there is a risk with displaying some error logs to the user I modified the code to only pass the actual error if the DEBUG variable is set appropriately on the server.

debug = os.environ.get('DEBUG')
debug = debug.lower() in ['1', 't', 'true', 'y', 'yes'] if debug is not None else False

...
try:
    config = ini('postgres')
    db_host = config['host']
    db_version = select_version()

except Exception as e:
    print('index() error:', e)
    error = e if debug else "we're currently experiencing technical difficulties"

return render_template(
    html,
    error=error,
    db_host=db_host,
    db_version=db_version
)

Jinja Inheritance

This allowed me to create a base template and reuse that in other templates, if I had other templates, so I could keep a consistent style app-wide with low maintenance.

<head>
  <meta charset="UTF-8" />
  <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}" />
  <title>CloudGuruChallenge_21.06</title>
</head>
<body>
  {% block content %}
  {% endblock %}
</body>

{% extends 'base.html' %}
{% block content %}
...
<div class="section">
    <span class="section-title">
        Elapsed Time
    </span>
    <span>{{ g.request_time() }}</span>
</div>
{% endblock %}

Gunicorn Server

I had quickly learned that the built-in flask server is not intended to work in production. Although, it may have been fine for this simple project I wanted to go the extra mile to setup gunicorn instead.

Once installed I just ran the app as a daemon in my bootstrap script. e.g. gunicorn -D app:app

Styling

I couldn't stand the bare HTML page and decided to use some CSS to spruce it up a tad. In addition to styling I wanted to leverage flask's static assets and url_for as well to solidify that knowledge.

<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}" />