Create an AWS Lambda function using a Container image (Dockerfile) stored in AWS ECR

Overview

While creating an AWS Lambda function deployment package (zip file) for one of my ETL projects, we encountered the frustrating "service quota" issue—where the unzipped package size exceeded the allowable limit, resulting in excessive compute resource requirements.

In this post, we'll explore a more flexible approach by creating and deploying a Docker image to Amazon ECR for our AWS Lambda function:

1. Create a local directory to build our image. Every step will be handled in this directory.

2. Create a Python script that will be used as the Lambda function in this directory.

3. Also, we need to create a requirements.txt file for all the libraries used in the Python script.

4. In the end, we will create a Dockerfile which will be used to build the image.

Pre-requisites

• AWS Command Line Interface (AWS CLI) version 2

• Docker (minimum version 20.10.10 for Python 3.12 and later base images)

• Python

Build a container image using an AWS base image for Python

1. Create a local directory for the project and navigate to it.

mkdir example
cd example

2. Create a new file named lambda_function.py with the following example:

import polars as pl
from helper import write_csv_s3, read_excel_s3

def lambda_handler(event, context):
    es_data_sku_df = read_excel_s3(
        key= AWS_ACCESS_KEY_ID,
        secret= AWS_SECRET_ACCESS_KEY,
        source= f"s3://{AWS_S3_BUCKET}/{AWS_KEY_RAW_EXCEL_ES_DATA_SKU}",
        sheet_name= RAW_WS_EXCEL_ES_DATA_SKU,
    )

if __name__ == "__main__":
    lambda_handler(None, None)

3. Create a new file named requirements.txt with the following example:

polars
s3fs
fastexcel

4. Create a Dockerfile with the following configuration:

# Use a python image from AWS
FROM public.ecr.aws/lambda/python:3.12

# Copy requirements.txt into the right directory for the lambda function
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Install the specified packages (and cache the downloaded packages)
RUN --mount=type=cache,target=/root/.cache/pip  pip install -r requirements.txt

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}
COPY helper.py ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "lambda_function.lambda_handler" ]

Set the FROM property to the URI of the base image.

Use the COPY command to transfer the Lambda function code and runtime dependencies from the local directory to the Lambda function's code path, denoted by the Lambda-defined environment variable {LAMBDA_TASK_ROOT}

Set the CMD argument to the Lambda function handler.

5. Build the Docker image with the docker build command. The following example names the image docker-image and gives it the test tag.

docker build --platform linux/amd64 -t docker-image:test .

(Optional) Test the image locally

1. Start the Docker image using the docker run command. In this example, docker-image is the image name and test is the tag.

docker run --platform linux/amd64 -p 9000:8080 -v ~/.aws:/root/.aws docker-image:test

2. In a new terminal, send an event to the local endpoint at port 9000. In PowerShell, use the following Invoke-WebRequest command:

Invoke-WebRequest -Uri "http://localhost:9000/2015-03-31/functions/function/invocations" -Method Post -Body '{}' -ContentType "application/json"

3. Get the container ID.

docker ps

4. Use the docker kill command to stop the container. Replace 1234567890 with the container ID from the previous step.

docker kill 1234567890

Deploying the image: Upload the Docker image to AWS ECR and create a Lambda function that uses this image as its container

1. Go to AWS ECR -> Repositories -> Create Repository

   
   

   

2. After assigning a specific name to the repository, such as "lambda_function_docker_repository", click on Create repository

3. On the main page of our repository, click View push commands.

3.1. Run the get-login-password command to authenticate the Docker CLI with your AWS ECR registry.

aws ecr get-login-password --region [region] | docker login --username AWS --password-stdin [aws_account_id].dkr.ecr.[region].amazonaws.com

Replace [aws_account_id] with your AWS account ID

Set the [region] value to the AWS Region where you created the ECR repository, such as ap-southeast-1

3.2. Run the docker tag command to tag your local image as the latest version in your AWS ECR repository.

docker tag docker-image:test [aws_account_id].dkr.ecr.[region].amazonaws.com/[repository_name]:latest

Replace [aws_account_id] with your AWS account ID

Set the [region] value to the AWS Region where you created the ECR repository, such as ap-southeast-1

Replace [repository_name] with the name of your AWS ECR repository, such as lambda_function_docker_repository

Make sure to include the :latest tag at the end of the repository URI

3.3. Use the docker push command to upload your local image to the AWS ECR repository for deployment.

docker push [aws_account_id].dkr.ecr.[region].amazonaws.com/[repository_name]:latest

Replace [aws_account_id] with your AWS account ID

Set the [region] value to the AWS Region where you created the ECR repository, such as ap-southeast-1

Replace [repository_name] with the name of your AWS ECR repository, such as lambda_function_docker_repository

Make sure to include the :latest tag at the end of the repository URI

4. Create the Lambda function from the Docker image by following these steps:

   Lambda -> Functions -> Create Function -> Container image

   
   

   Function name: We can give whatever name we want to our function.

   Container image URI: Click Browse for the Docker image we created and select it.

   Architecture: Select x86_64

   Change default execution role: We can choose the suitable IAM role in case we need it.

   Click on Create function

Updating the AWS Lambda function code

To update the function code, rebuild the image, upload the new version to the Amazon ECR repository, and then use the update-function-code command to deploy the updated image to the Lambda function.

aws lambda update-function-code --function-name [my-function] --image-uri [aws_account_id].dkr.ecr.[region].amazonaws.com/[repository_name]:latest

Set the [my-function] to the AWS Lambda function name

Replace [aws_account_id] with your AWS account ID

Set the [region] value to the AWS Region where you created the ECR repository, such as ap-southeast-1

Replace [repository_name] with the name of your AWS ECR repository, such as lambda_function_docker_repository

Make sure to include the :latest tag at the end of the repository URI

Reference

Using an AWS base image for Python

Pushing a Docker image to an Amazon ECR private repository

About

Benjamin ("Benj") Tabares Jr. is an experienced data practitioner with a strong track record of successfully delivering short- and long-term projects in data engineering, business intelligence, and machine learning. Passionate about solving complex customer challenges, Benj leverages data and technology to create impactful solutions. He collaborates closely with clients and stakeholders to deliver scalable data solutions that unlock business value and drive meaningful insights from data.