AWS CloudFormation UPDATE_ROLLBACK_FAILED fix in production

Recently, while trying to deploy a serverless application, the pipeline failed thus putting the cloud formation stack in "UPDATE_ROLLBACK_FAILED" state. This is due to an error in your pipeline. In my own case, I was trying to use a base layer - "arn:aws:lambda:us-east-1:770693421928:layer:klayers-python38-pandas:35" which at the point of writing this article does not exist.

If you are familiar with AWS SAM Cli or cloud formation generally, you will know that you wont be able to deploy a new update until this status changes to "UPDATE_COMPLETE".

Now, you have two options facing this situation (at least that I am aware of):

1. Attempt to complete the update rollback process.
2. Delete the stack and create another.

The second option is almost perfect and simple except for the fact that you might not want to do this in production has is basically shutting down your application.

The first option however is much more advisable. Now, how do you complete the update rollback.

Using AWS CLI

You can make use of the aws cloudformation continue-update-rollback command to complete your update rollback.

aws cloudformation continue-update-rollback \
--stack-name STACK_NAME \
--resources-to-skip LIST_OF_RESOURCES

Where STACK_NAME is the name of your stack while LIST_OF_RESOURCES is the list of the logical IDs of resources you will like to skip. Please note that for LIST_OF_RESOURCES you have to specify resources that are in the UPDATE_FAILED state only.

Sample

aws cloudformation continue-update-rollback \
--stack-name eazido-app-stack \
--resources-to-skip CustomerApi PaymentApi

Using AWS Console

1. Visit your cloud formation area of the AWS console - https://console.aws.amazon.com/cloudformation/
2. Select the stack that requires rollback.
3. Under the "Stack Action" select "Continue update rollback" Note If this update rollback still fails or you want to skip some resources, then select "Advanced troubleshooting" on the "Continue update rollback" dialog and tick the resources you will like to skip.

Once this is done, your stack should now carry the "** UPDATE_ROLLBACK_COMPLETE**" status. If you try to deploy your updates again, it should work just fine.

However, remember how the update failed because of an error, you will need to identify and fix this before you deploy. In my own case, I was using a layer whose version does not exist. I had to update the layer by updating the stack template (we will talk about this in a bit). You can tell why your stack failed by checking the Status reason column under the Events tab of the failed stack.

Updating CloudFormation Template

• Select the stack you want to update its template and select "View in designer"" under the "Template" tab.

Once the "View in designer"'s page loads, you should see a page similar to this.

You can get the template for a particular stack using AWS CLI's (get-template)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/get-template.html] command.

aws cloudformation get-template --stack-name STACK_NAME

• You can either edit your template as json or Yaml file. Once you are done making changes, click on the file icon on the top left corner of the page and click "save". You have the option of saving on your laptop or in an s3 bucket. Once that is done, you can exit the page.

Creating and executing a Change Set

1. Select the stack you want to update it's template.

2. Under "Stack Actions" select "Create change set for current stack". You should see a page like this
   

3. Click on "Replace current template", select a template source (local or s3) based on where you saved the edited template.

4. Review the change set and Execute. This is will starts updating the AWS cloud formation stack and you can see the progress on the event tab.

You can update a particular stack using AWS CLI's (update-stack)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html] command.

aws cloudformation update-stack --stack-name STACK_NAME --template-url https://s3.amazonaws.com/sample/updated_template.template

Note: If you are getting AWS cli errors, you might want to take a look at the - AWS cli troubleshooting guide.