Creating AWS resources using Boto3 for deploying Django project

This is a bonus blog post to Django with AWS Lambda series. If you are a Python developer and you need to create AWS resources but you don't want to learn Terraform or use AWS Management Console this blog post is for you.

Boto3 Introduction

We will use Boto3 - the AWS SDK for Python. Boto3 makes it easy to integrate your Python application, library, or script with AWS services including Amazon S3, Amazon EC2, Amazon DynamoDB, and more.

Boto3 has two levels of APIs:

  • Client (or "low-level") APIs provide one-to-one mappings to the underlying HTTP API operations.
  • Resource APIs hide explicit network calls but instead provide resource objects and collections to access attributes and perform actions.

Let's start codding

First, we need to install boto3 python library:

pip install boto3

I used version 1.18.40 of boto3 which was the latest version on the day of writing this post. It is always better to use the latest version of boto3 as the AWS team is actively working on this library.

Second (optional), we can install python-dotenv python library to read environment variables from .env file:

pip install python-dotenv

This step is optional, but I do recommend using environment variables for any sensitive or configurable parameters like passwords, AWS credentials, and such.

Third (if you decided to use environment variables), we should create .env file with the following variables:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION_NAME=

DEFAULT_VPC_ID=
DB_INSTANCE_IDENTIFIER=
RDS_DB_NAME=
RDS_USERNAME=
RDS_PASSWORD=
S3_BUCKET_NAME=

You can find your DEFAULT_VPC_ID using AWS Management Console

Fourth, we need to create Python file where we will write our code to create all the necessary AWS resources for deploying a Django project on AWS Lambda.

touch django_aws_resources.py

Using Boto3 - the AWS SDK for Python

First, we need to import all libraries we are going to use, load the environment variables, and set up some helpful variables.

import json # to dump Python object with S3 bucket policy to JSON string
import os  # to get the necessary environment variables  
import time  # to wait until AWS RDS instance will be created
from datetime import datetime  # to generate a unique timestamp for unique CallerReference

import boto3
from dotenv import load_dotenv

load_dotenv()

S3_BUCKET_NAME = os.environ["S3_BUCKET_NAME"]
REGION_NAME = os.getenv("AWS_REGION_NAME") or "us-east-1"

Second, we should create a SecurityGroup and add inbound and outbound rules:

ec2_resource = boto3.resource("ec2", region_name=REGION_NAME)

security_group = ec2_resource.create_security_group(
    Description="sg-for-lambdas",
    GroupName="django-rds-security-group",
    VpcId=os.environ["DEFAULT_VPC_ID"],
    TagSpecifications=[
        {
            "ResourceType": "security-group",
            "Tags": [
                {"Key": "Name", "Value": "django-demo-rds-security-group"},
            ],
        },
    ],
    DryRun=False,
)

_ = security_group.authorize_egress(
    DryRun=False,
    IpPermissions=[
        {
            "FromPort": 0,
            "IpProtocol": "-1",
            "IpRanges": [],
            "Ipv6Ranges": [
                {"CidrIpv6": "::/0", "Description": "allow all (demo only)"},
            ],
            "PrefixListIds": [],
            "ToPort": 0,
        },
    ],
    TagSpecifications=[
        {
            "ResourceType": "security-group-rule",
            "Tags": [
                {"Key": "Name", "Value": "egress rule"},
            ],
        },
    ],
)

_ = security_group.authorize_ingress(
    DryRun=False,
    IpPermissions=[
        {
            "FromPort": 0,
            "IpProtocol": "-1",
            "IpRanges": [
                {"CidrIp": "0.0.0.0/0", "Description": "allow all (demo only)"},
            ],
            "Ipv6Ranges": [
                {"CidrIpv6": "::/0", "Description": "allow all (demo only)"},
            ],
            "PrefixListIds": [],
            "ToPort": 0,
        },
    ],
    TagSpecifications=[
        {
            "ResourceType": "security-group-rule",
            "Tags": [
                {"Key": "Name", "Value": "ingress rule"},
            ],
        },
    ],
)

Third, we need to create an RDS instance with Postgres engine:

rds_client = boto3.client("rds", region_name=REGION_NAME)
_ = rds_client.create_db_instance(
    DBName=os.environ["RDS_DB_NAME"],
    DBInstanceIdentifier=os.environ["DB_INSTANCE_IDENTIFIER"],
    AllocatedStorage=20,
    DBInstanceClass="db.t2.micro",
    Engine="postgres",
    EngineVersion="12.5",
    MasterUsername=os.environ["RDS_USERNAME"],
    MasterUserPassword=os.environ["RDS_PASSWORD"],
    VpcSecurityGroupIds=[security_group.id],
    Tags=[{"Key": "name", "Value": "django_demo_rds"}],
)

Fourth, we should create an S3 bucket for static files:

s3_client = boto3.client("s3", region_name=REGION_NAME)
_ = s3_client.create_bucket(
    ACL="private",
    Bucket=S3_BUCKET_NAME,
)

Fifth, we need to create a CloudFront Origin Access Identity and update the S3 bucket policy to allow a CloudFront distribution serving static files from the bucket:

cloudfront_client = boto3.client("cloudfront", region_name=REGION_NAME)

response = cloudfront_client.create_cloud_front_origin_access_identity(
    CloudFrontOriginAccessIdentityConfig={
        "CallerReference": str(datetime.utcnow().timestamp()),
        "Comment": f'access-identity-{S3_BUCKET_NAME}.s3.amazonaws.com"',
    }
)
origin_access_identity_id = response["CloudFrontOriginAccessIdentity"]["Id"]

s3_resource = boto3.resource("s3")
bucket_policy = s3_resource.BucketPolicy(S3_BUCKET_NAME)
_ = bucket_policy.put(
    Policy=json.dumps(
        {
            "Version": "2008-10-17",
            "Statement": [
                {
                    "Sid": "1",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": f"arn:aws:iam::cloudfront:user/CloudFront "
                        f"Origin Access Identity {origin_access_identity_id}",
                    },
                    "Action": "s3:GetObject",
                    "Resource": f"arn:aws:s3:::{S3_BUCKET_NAME}/*",
                }
            ],
        }
    ),
)

Sixth, we need to create a CloudFront Distribution to serve static files from the S3 bucket

cloudfront_distribution_response = cloudfront_client.create_distribution(
    DistributionConfig={
        "CallerReference": str(datetime.utcnow().timestamp()),
        "Origins": {
            "Quantity": 1,
            "Items": [
                {
                    "Id": S3_BUCKET_NAME,
                    "DomainName": f"{S3_BUCKET_NAME}.s3.amazonaws.com",
                    "S3OriginConfig": {
                        "OriginAccessIdentity": f"origin-access-identity/cloudfront/{origin_access_identity_id}"
                    },
                },
            ],
        },
        "Restrictions": {"GeoRestriction": {"RestrictionType": "none", "Quantity": 0}},
        "ViewerCertificate": {
            "CloudFrontDefaultCertificate": True,
        },
        "DefaultCacheBehavior": {
            "TargetOriginId": S3_BUCKET_NAME,
            "Compress": True,
            "ViewerProtocolPolicy": "allow-all",
            "AllowedMethods": {
                "Quantity": 3,
                "Items": ["GET", "HEAD", "OPTIONS"],
                "CachedMethods": {
                    "Quantity": 2,
                    "Items": ["GET", "HEAD"],
                },
            },
            "ForwardedValues": {
                "QueryString": False,
                "Cookies": {
                    "Forward": "none",
                },
            },
            "MinTTL": 0,
            "DefaultTTL": 3600,
            "MaxTTL": 86400,
        },
        "Enabled": True,
        "IsIPV6Enabled": True,
        "DefaultRootObject": "index.html",
        "Comment": "Django React static distribution",
    }
)

Then, we can wait until the RDS instance will be created to get the Database Host Name for Django project configurations:

while True:
    rds_db_instances = rds_client.describe_db_instances(
        DBInstanceIdentifier="string",
        Filters=[
            {"Name": "db-instance-id", "Values": [os.environ["DB_INSTANCE_IDENTIFIER"]]},
        ],
        MaxRecords=20,
    )

    if rds_db_instances["DBInstances"][0].get("Endpoint"):
        db_host_name = rds_db_instances["DBInstances"][0]["Endpoint"]["Address"]
        break
    time.sleep(100)

Finally, we can print Database Host Name and CloudFront Distribution Domain Name to use them in Django project configurations:

print(f"Database Host Name: {db_host_name}")
print(f"CloudFront Domain Name: {cloudfront_distribution_response['Distribution']['DomainName']}")

Note, that this is an example of configuring AWS resources. Your production configuration can be different.

Final Words

Today we saw one more way of how to prepare AWS infrastructure for a Django project. You can find a full version of the code showed here in this GitHub repository.

Don't forget to follow me on Twitter @vadim_khodak or on LinkedIn so you do not miss the next posts.

12

Kategorien:

Python

Tags:

Boto3PythonAwsInfrastructure