Managing external libraries in AWS lambda functions

The popularity of microservices is increasing everyday. The main reason behind this is that we do not need to take care of the server related configurations. Just write down your code and see the magic. Definitely aws lambda is one of the best microservices to perform varieties of tasks.

When we write a lambda function, it is quite obvious that we may need third party modules and those will not be present in that environment. So we need to add them to lambda. Now the question comes how we can manage them properly? The best answer would be using lambda layers. According to aws document:

A Lambda layer is a .zip file archive that can contain additional code or data. A layer can contain libraries, a custom runtime, data, or configuration files.

So the idea is that we can create a lambda layer and keep other third party libraries in a zipped file. We can create multiple layers as per our requirements and use the same layer to multiple lambda functions. Let us create lambda layers in both node & python runtime & use it in respective lambda functions.

Lambda layer using node

In order to add a node lambda layer we will have to keep the installed dependencies and files in a folder named nodejs. Here the folder name has to be nodejs so that aws lambda can identify the dependencies. Let us now build a layer which contains the node-fetch module that creates http requests with node js.

1. Create a folder named nodejs.
2. Go inside that folder.
3. Open the path in terminal.
4. Create a package.json file using npm init -y.
5. Then install required dependencies. Here only node-fetch will be installed using npm i node-fetch.
6. Inside the nodejs folder there will be one folder named node_modules and two files named package.json & package-lock.json. Compress the nodejs folder to a zipped file.

Now we are ready to create our lambda layer. Just go to the aws console Create layer section like the following screenshot. Name your layer, choose a runtime like Node.js 14.x & upload the zipped file created. Thus our lambda layer will be created.

Now we will use this layer in our lambda function. Assuming that you know how to create a lambda function ( if you don't, no worries coz its quite simple), just add the layer you created to your lambda function. In our lambda function we will make a GET request to an api & show the response in a json format. Our lambda function code is:

const fetch = require("node-fetch");

exports.handler = async (event) => {
    try {
        const response = await fetch(event.url);
        const json = response.json();
        return json;
    } catch(e) {
        return e;
    }
};

We will create a test event containing:

{
  "url": "https://reqres.in/api/users"
}

Now if we hit the Test button we should see a response like following:

{
  "page": 1,
  "per_page": 6,
  "total": 12,
  "total_pages": 2,
  "data": [
    {
      "id": 1,
      "email": "[email protected]",
      "first_name": "George",
      "last_name": "Bluth",
      "avatar": "https://reqres.in/img/faces/1-image.jpg"
    },
    {
      "id": 2,
      "email": "[email protected]",
      "first_name": "Janet",
      "last_name": "Weaver",
      "avatar": "https://reqres.in/img/faces/2-image.jpg"
    },
    {
      "id": 3,
      "email": "[email protected]",
      "first_name": "Emma",
      "last_name": "Wong",
      "avatar": "https://reqres.in/img/faces/3-image.jpg"
    },
    {
      "id": 4,
      "email": "[email protected]",
      "first_name": "Eve",
      "last_name": "Holt",
      "avatar": "https://reqres.in/img/faces/4-image.jpg"
    },
    {
      "id": 5,
      "email": "[email protected]",
      "first_name": "Charles",
      "last_name": "Morris",
      "avatar": "https://reqres.in/img/faces/5-image.jpg"
    },
    {
      "id": 6,
      "email": "[email protected]",
      "first_name": "Tracey",
      "last_name": "Ramos",
      "avatar": "https://reqres.in/img/faces/6-image.jpg"
    }
  ],
  "support": {
    "url": "https://reqres.in/#support-heading",
    "text": "To keep ReqRes free, contributions towards server costs are appreciated!"
  }
}

Lambda layer using python

Now we will create a lambda layer to perform the similar functionality in python. We will be installing necessary modules for requests to make http request and fetch data from remote api. Now we have to do as following:

1. Create directories like this build/python/lib/python3.8/site-packages. We must not make any spelling mistake otherwise we will encounter error while running the lambda. Another important thing is the version of python. You must use the same version of python everywhere. Here I am using 3.8 so while selecting runtime I must select Python 3.8 to avoid errors.
2. Now go inside the site-packages folder and install requests using pip like this where -t . means install everything inside this folder.
3. Now come back to the build folder & compress the python folder to a zipped file.
4. Now create a lambda layer like previous way & upload the zipped file. Select Python 3.8 as runtime.

mkdir build/python/lib/python3.8/site-packages
cd build/python/lib/python3.8/site-packages
pip install requests -t .

Now its time to write down the lambda function. After creating the lambda, code will be as following:

import json
import requests

def lambda_handler(event, context):
    response = requests.get(event['url'])
    data = response.json()
    return data

Our test event will be same as before & after running the lambda we will get similar json response as output.