27
Automating the Deployment of CloudWatch Canaries with CloudFormation
At Grid Smarter Cities we use CloudFormation to automate the deployment of a wide range of stacks, including:
We do this because it makes the process of automatically deploying resources efficient and less time consuming. CloudFormation treats infrastructure as code, meaning both resources and their dependencies can be created, validated and deployed from a single template file.
If the stack needs to be updated, all the user has to do is make the necessary changes to the template, reupload it, and let CloudFormation verify and make all the necessary changes to your resources. If a stack needs duplicating, upload the template again and adjust the parameters and redeploy. If the stack is no longer needed, everything can be deleted with a single button click.
Before the stack is deployed, we will need to have a prepared script for the canary and two S3 Buckets. The first bucket is used by the Canary to store its artifacts saved, while it is running - this could be screenshots, logs or generated reports. The second bucket will be used to store the source code used by the canary - we are using a Node script for this canary. The code for this can be found here.
These files need to be in a particular folder structure - this wasn't clear on the AWS CloudFormation Documentation. However, I eventually found that your canary's code file structure should be /nodejs/node_modules/ .
The following two commands will create this folder structure, and prepare the zip folder:
cp canary-function.js ./nodejs/node_modules/
zip -r sourcecode.zip ./nodejs/*The new sourcecode.zip file just needs to be uploaded directly into the S3 bucket you are using to store the Canary source code.
In this template, we have added parameters for:
All of these values will be inputted by the user and passed as parameters when the user has uploaded the template.
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: 'Service'
Parameters:
- ServiceName
- CanaryName
- AlertEmail
- Label:
default: 'Buckets'
Parameters:
- CanarySourceCodeBucketName
- CanarySourceCodeKey
- CanaryArtifactBucketName
ParameterLabels:
ServiceName:
default: Service Name
CanaryName:
default: Canary Name
AlertEmail:
default: Email
CanarySourceCodeBucketName:
default: Source code bucket name
CanarySourceCodeKey:
default: Key for the object
CanaryArtifactBucketName:
default: The name of the S3 bucket where the artefacts will be storeParameters:
ServiceName:
Description: Enter a lower case, high level service name without environment details. Used to autofill service names. For example, your-service-name
Type: String
CanaryName:
Description: Enter a lower case name for the canary, as they should be known. For example dave not Dave
Type: String
AlertEmail:
Description: Email address to send staging build alerts to, or example you@example.com
Type: String
CanarySourceCodeBucketName:
Description: S3 Bucket Name where the source code lives
Type: String
CanarySourceCodeKey:
Description: Key of the object which contains the source code
Type: String
CanaryArtifactBucketName:
Description: S3 Bucket Name where the canary saves the screenshots and other
Type: StringThe resources we need are:
The first resource is of type AWS::Synthetics::Canary. The property specifying the canary's name and relevant S3 buckets have been read from the parameters defined earlier in the template. For now, the runtime version, handler and schedule have been hard coded, but these could easily be extracted as parameters to make the template more flexible.
CloudwatchCanary:
Type: AWS::Synthetics::Canary
Properties:
ArtifactS3Location: !Sub s3://${CanaryArtifactBucketName}
Code:
Handler: canary-function.handler
S3Bucket: !Sub ${CanarySourceCodeBucketName}
S3Key: !Sub ${CanarySourceCodeKey}
Name: dave-the-canary
RuntimeVersion: syn-nodejs-puppeteer-3.1
Schedule:
Expression: rate(5 minutes)
StartCanaryAfterCreation: true
ExecutionRoleArn: !GetAtt CloudwatchCanaryRole.ArnThe second and third resources are of type AWS::IAM::Policy and AWS::IAM::Policy. We chose to give the CloudWatch Canary a policy which had been modified from CloudWatchSyntheticsFullAccess. This gave our canary permission to:
We had to add a few extra actions and modify the resource selectors to match our resource names. There were also some unnecessary actions which - for best practices and security - were removed.
CloudwatchCanaryPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub ${ServiceName}-dave-the-canary-policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- synthetics:*
Resource: "*"
- Effect: Allow
Action:
- s3:CreateBucket
- s3:PutEncryptionConfiguration
Resource:
- arn:aws:s3:::*
- Effect: Allow
Action:
- iam:ListRoles
- s3:ListAllMyBuckets
- s3:GetBucketLocation
- xray:GetTraceSummaries
- xray:BatchGetTraces
- apigateway:GET
Resource: "*"
- Effect: Allow
Action:
- s3:GetObject
- s3:ListBucket
- s3:PutObject
Resource: arn:aws:s3:::*
- Effect: Allow
Action:
- s3:GetObjectVersion
Resource: arn:aws:s3:::*
- Effect: Allow
Action:
- iam:PassRole
Resource:
- !Sub arn:aws:iam::*:role/service-role/${CloudwatchCanaryRole}
Condition:
StringEquals:
iam:PassedToService:
- lambda.amazonaws.com
- synthetics.amazonaws.com
- Effect: Allow
Action:
- iam:GetRole
Resource:
- !Sub arn:aws:iam::*:role/service-role/*
- Effect: Allow
Action:
- cloudwatch:GetMetricData
- cloudwatch:GetMetricStatistics
Resource: "*"
- Effect: Allow
Action:
- cloudwatch:PutMetricAlarm
- cloudwatch:PutMetricData
- cloudwatch:DeleteAlarms
Resource:
- '*'
- Effect: Allow
Action:
- cloudwatch:DescribeAlarms
Resource:
- arn:aws:cloudwatch:*:*:alarm:*
- Effect: Allow
Action:
- lambda:CreateFunction
- lambda:AddPermission
- lambda:PublishVersion
- lambda:UpdateFunctionConfiguration
- lambda:GetFunctionConfiguration
Resource:
- arn:aws:lambda:*:*:function:cwsyn-*
- Effect: Allow
Action:
- lambda:GetLayerVersion
- lambda:PublishLayerVersion
Resource:
- arn:aws:lambda:*:*:layer:cwsyn-*
- arn:aws:lambda:*:*:layer:Synthetics:*
- Effect: Allow
Action:
- ec2:DescribeVpcs
- ec2:DescribeSubnets
- ec2:DescribeSecurityGroups
Resource:
- "*"
- Effect: Allow
Action:
- sns:ListTopics
Resource:
- "*"
- Effect: Allow
Action:
- sns:CreateTopic
- sns:Subscribe
- sns:ListSubscriptionsByTopic
Resource:
- arn:*:sns:*:*:Synthetics-*
Roles:
- !Ref CloudwatchCanaryRoleCloudwatchCanaryRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${ServiceName}-${CanaryName}-the-canary-role
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRoleOn its own, the deployed canary will monitor resources as canary's source code files. Any issues found by the canary will be visible on the relevant dashboard in CloudWatch. Unfortunately, this will not trigger any email alerts or notifications. Adding this functionality to our stack is straightforward, we need to add a CloudWatchAlarm and an SNS Topic.
By adding the following sections, alert emails will be sent to the specified email address when submitting the template.
The CloudWatchCanaryAlarm is triggered when the average success count falls below 100% for one evaluation period; this period was hard coded to five minutes. If this failure is detected, the CloudWatchCanaryAlarmTopic is triggered.
CloudwatchCanaryAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
ActionsEnabled: true
AlarmDescription: !Sub ${ServiceName}-${CanaryName}-the-canary-alert
ComparisonOperator: LessThanThreshold
EvaluationPeriods: 1
DatapointsToAlarm: 1
MetricName: SuccessPercent
Namespace: CloudWatchSynthetics
Period: 60
Statistic: Average
Threshold: 100
TreatMissingData: notBreaching
AlarmActions:
- !Ref CloudwatchCanaryAlarmTopic
Dimensions:
- Name: CanaryName
Value: !Sub ${CanaryName}-the-canaryThe CloudWatchCanaryAlarmTopic sends a message to the AlertEmail address, specified by the CloudFormation parameters inputted by the user when deploying the stack.
CloudwatchCanaryAlarmTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: !Sub ${CanaryName}-cw-canary-alarms
Subscription:
- Endpoint: !Ref AlertEmail
Protocol: emailAll that's left to do is deploy Dave the Canary. To do this, upload the template, fill in the form defining the stack parameters as shown below, click next(we'll not configure any of the default stack options), click next again, confirm that you "acknowledge that AWS CloudFormation might create IAM resources with custom names" then click the Create Stack button. Your resources should then be deployed and running.
This CloudFormation template was written to help developers at Grid Smarter Cities deploy tools which monitor our products. Deploying a canary from the console is straightforward but using CloudFormation allows us to automate the deployment process, allowing us to set up monitoring tools and alerts for all of our products.
Writing this template may have required a bit of research and some trial and error but, in doing so, our developers and QAs will never have to manually check that our products are live. If the products aren't, the CloudWatch canary will automatically inform us quickly and more reliably than if we were manually checking ourselves.
Full copies of both templates (with and without the alerts) and example canary function can be found in the Repo
27