Semantic Video Search Vector Database > Feature Extraction Using SageMaker and ECS > Extract Keyframes Using ECS

Extract Keyframes Using ECS

1 Create Docker Image for Keyframes Extraction

Create a new directory called ecr-extract-keyframes and navigate to it.

.
|-- Dockerfile
|-- get-object-s3.py
|-- requirements.txt
|-- run-task.py
|-- split-video.py

Create file Dockerfile in the directory.

FROM python:3.9-slim

RUN apt-get update && \
    apt-get install -y \
    libgl1-mesa-glx \
    libglib2.0-0 \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . /app

WORKDIR /app

CMD ["python", "extract-keyframes.py"]

Create file extract-keyframes.py in the directory.

import boto3
import cv2
import os
import numpy as np
import tempfile
import csv
from io import StringIO

class S3KeyframeUpdater:
    def __init__(self, bucket_name, video_key, folder_prefix, output_csv_key):
        self.bucket_name = bucket_name
        self.video_key = video_key
        self.folder_prefix = folder_prefix
        self.output_csv_key = output_csv_key
        self.s3 = boto3.client('s3')

    def list_s3_objects(self, prefix):
        response = self.s3.list_objects_v2(Bucket=self.bucket_name, Prefix=prefix)
        if 'Contents' in response:
            return [obj['Key'] for obj in response['Contents']]
        return []

    def get_s3_object(self, key):
        return self.s3.get_object(Bucket=self.bucket_name, Key=key)['Body'].read()

    def update_keyframe_metadata(self):
        print(f'Download video from {self.video_key}')
        video_data = self.get_s3_object(self.video_key)

        with tempfile.NamedTemporaryFile(delete=False, suffix='.mp4') as temp_video_file:
            temp_video_file.write(video_data)
            temp_video_path = temp_video_file.name

        video = cv2.VideoCapture(temp_video_path)
        keyframe_keys = self.list_s3_objects(self.folder_prefix)
        number_of_keyframes = len(keyframe_keys)
        count = 0
        frame_number = 0

        csv_buffer = StringIO()
        csv_writer = csv.writer(csv_buffer)
        csv_writer.writerow(['VideoName', 'KeyframeName', 'FrameNumber'])

        while True:
            ret, frame = video.read()
            if not ret or count >= number_of_keyframes:
                break

            gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            keyframe_data = self.get_s3_object(keyframe_keys[count])
            keyframe_img = cv2.imdecode(np.frombuffer(keyframe_data, np.uint8), cv2.IMREAD_GRAYSCALE)
            
            result = cv2.matchTemplate(gray_frame, keyframe_img, cv2.TM_CCOEFF_NORMED)
            _, max_val, _, _ = cv2.minMaxLoc(result)

            if max_val > 0.9:
                name = os.path.basename(keyframe_keys[count])
                self.s3.copy_object(
                    Bucket=self.bucket_name,
                    CopySource={'Bucket': self.bucket_name, 'Key': keyframe_keys[count]},
                    Key=keyframe_keys[count],
                    MetadataDirective='REPLACE',
                    Metadata={
                        'VideoKey': self.video_key,
                        'FrameNumber': str(frame_number)
                    }
                )
                print(f'Updated metadata for {name} at frame {frame_number}')
                csv_writer.writerow([os.path.basename(self.video_key), name, frame_number])
                count += 1
            
            frame_number += 1

        video.release()
        os.remove(temp_video_path)

        csv_buffer.seek(0)
        self.s3.put_object(
            Bucket=self.bucket_name,
            Key=self.output_csv_key,
            Body=csv_buffer.getvalue(),
            ContentType='text/csv'
        )

        print(f'Result has saved to {self.output_csv_key} on S3')

if __name__ == "__main__":
    bucket_name = os.environ['BUCKET_NAME']
    folder_prefix = os.environ['FOLDER_PREFIX']
    video_key = os.environ['VIDEO_KEY']
    output_csv_key = f'metadata/{video_key.split("/")[1].split(".")[0]}.csv'

    if not bucket_name or not folder_prefix or not video_key:
        raise ValueError('Missing environment variables')

    print("Start updating keyframe metadata: ", bucket_name, video_key, folder_prefix, output_csv_key)
    updater = S3KeyframeUpdater(bucket_name, video_key, folder_prefix, output_csv_key)
    updater.update_keyframe_metadata()

Create a requirements.txt in the directory.

boto3
opencv-python-headless
numpy

Build the docker image.

docker build -t ai-challenge-2024:keyframe-matcher .

Tag the docker image.

docker tag ai-challenge-2024:keyframe-matcher <aws account_id>.dkr.ecr.ap-southeast-1.amazonaws.com/ai-challenge-2024:keyframe-matcher

Push the docker image to ECR.

Go to the ECR console. Click on the repository ai-challenge-2024. Click on the View push commands button.
Run the commands to login to ECR.

aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS --password-stdin <aws account_id>.dkr.ecr.ap-southeast-1.amazonaws.com

Push the docker image to ECR.

docker push <aws account_id>.dkr.ecr.ap-southeast-1.amazonaws.com/ai-challenge-2024:keyframe-matcher

Docs Version Dropdown

Create Task Definition

Click on Task Definitions in the left navigation pane.
Click on Create new Task Definition button.
- Select Create new task with JSON
Paste this json in the Task Definition editor.

{
    "containerDefinitions": [
        {
            "name": "keyframe-matcher-container-min",
            "image": "<aws account id>.dkr.ecr.ap-southeast-1.amazonaws.com/ai-challenge-2024:keyframe-matcher",
            "cpu": 512,
            "memory": 2048,
            "portMappings": [
                {
                    "name": "80",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                },
                {
                    "name": "433",
                    "containerPort": 443,
                    "hostPort": 443,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "FOLDER_PREFIX",
                    "value": ""
                },
                {
                    "name": "VIDEO_KEY",
                    "value": ""
                },
                {
                    "name": "BUCKET_NAME",
                    "value": ""
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/keyframe-matcher-definition-min",
                    "mode": "non-blocking",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "ap-southeast-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "family": "keyframe-matcher-definition-min",
    "taskRoleArn": "arn:aws:iam::<aws account id>:role/TaskRole",
    "executionRoleArn": "arn:aws:iam::<aws account id>:role/TaskRole",
    "networkMode": "awsvpc",
    "revision": 1,
    "status": "ACTIVE",
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "2048",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
}

The core configuration specifying how the container runs within the task:

Container Definitions

name: keyframe-matcher-container-min – The name of the container.
image: <aws account id>.dkr.ecr.ap-southeast-1.amazonaws.com/ai-challenge-2024:keyframe-matcher – The Docker image stored in Amazon ECR that the container will use.
cpu: 512 – Allocates 0.5 vCPUs to the container.
memory: 2048 – Allocates 2 GB of memory to the container.
portMappings:
- containerPort: 80 – Exposes port 80 for HTTP traffic.
- containerPort: 433 – Exposes port 433 for traffic (note: may need correction to 443 for HTTPS).
essential: true – Indicates that the container is critical for the task; if this container stops, the entire task will stop.

Environment Variables Environment variables passed to the container to configure behavior:

FOLDER_PREFIX: "" – The prefix for the S3 folder containing keyframes.
VIDEO_KEY: "" – The S3 key for the video file to process.
BUCKET_NAME: "" – The name of the S3 bucket containing the video and keyframes.

Logging Configuration Sets up logging to AWS CloudWatch Logs for monitoring and debugging:

logDriver: awslogs – Uses the AWS Logs driver to send logs to CloudWatch.
awslogs-group: /ecs/keyframe-matcher-definition-min – Specifies the CloudWatch Logs group where logs are stored.
awslogs-create-group: true – Automatically creates the log group if it does not exist.
max-buffer-size: 25m – Specifies the maximum buffer size for logs.
awslogs-region: ap-southeast-1 – Sets the AWS region for logging.
awslogs-stream-prefix: ecs – Adds a prefix to log streams to help identify them easily.

Task-Level Settings

family: "keyframe-matcher-definition-min" – The family name of the task, used to identify it in ECS.
taskRoleArn: arn:aws:iam::<aws account id>:role/TaskRole – Specifies the IAM role that grants the task permissions to interact with AWS services.
executionRoleArn: arn:aws:iam::<aws account id>:role/TaskRole – Specifies the IAM role used by the ECS agent to pull images and publish logs.

Network Configuration

networkMode: awsvpc – Uses the AWS VPC network mode, allowing the task to have its own network interfaces, security groups, and elastic network interfaces.
requiresCompatibilities: ["FARGATE"] – Specifies that the task requires Fargate launch type.

Resource Configuration

cpu: "512" – Allocates 0.5 vCPUs to the task.
memory: "2048" – Allocates 2 GB of memory to the task.
ephemeralStorage
- sizeInGiB: 21 – Provides 21 GB of ephemeral storage for temporary data generated by the task.

Platform Configuration

runtimePlatform:
- cpuArchitecture: X86_64 – Specifies the CPU architecture.
- operatingSystemFamily: LINUX – Specifies the operating system family used.

After pasting the JSON, click on Create button. We have successfully created a task definition for extracting keyframes from videos.

Docs Version Dropdown

3 Run Task to Extract Keyframes

3.1 Prepare Script to Run Task

Create run-task.py in the ecr-extract-keyframes directory. This script will run the task definition to extract keyframes from a video file.

import os
import boto3

s3_client = boto3.client("s3")
ecs_client = boto3.client("ecs")


def lambda_handler(event, context):

    cluster_name = os.environ["ECS_CLUSTER_NAME"]
    task_definition = os.environ["TASK_DEFINITION"]
    subnet_ids = os.environ["SUBNET_IDS"]
    security_group_id = os.environ["SECURITY_GROUP_ID"]
    container_name = os.environ["CONTAINER_NAME"]

    network_config = {
        "awsvpcConfiguration": {
            "subnets": subnet_ids,
            "securityGroups": [security_group_id],
            "assignPublicIp": "ENABLED",
        }
    }

    FOLDERS = []
    response = s3.list_objects_v2(Bucket=BUCKET_NAME, Prefix=PREFIX, Delimiter='/')

    # Get the subfolders in the bucket
    if 'CommonPrefixes' in response:
        subfolders = [prefix.get('Prefix') for prefix in response['CommonPrefixes']]
        for subfolder in subfolders:
            FOLDERS.append(subfolder)
    else:
        print("No subfolders found.")

    for folder in FOLDERS:
        run_task_input = {
            "cluster": cluster_name,
            "launchType": "FARGATE",
            "taskDefinition": task_definition,
            "networkConfiguration": network_config,
            "overrides": {
                "containerOverrides": [
                    {
                        "name": container_name,  
                        "environment": [
                            {
                                "name": "BUCKET_NAME",
                                "value": "ai-challenge-2024",
                            },
                            {
                                "name": "FOLDER_PREFIX",
                                "value": folder,
                            },
                            {
                                "name": "VIDEO_KEY",
                                "value": f"video/{folder.split("/")[1]}.mp4",
                            },
                        ],
                    }
                ]
            },
        }

        response = ecs_client.run_task(**run_task_input)

    
    return {
        "statusCode": 200,
        "body": f'Task created successfully with Task ARN: {response["tasks"][0]["taskArn"]}',
    }


if __name__ == "__main__":
    event = {}
    context = None
    lambda_handler(event, context)

Environment Variables:

ECS_CLUSTER_NAME: The name of the ECS cluster where the task will run.
TASK_DEFINITION: The ARN of the task definition created in the previous step.
SUBNET_IDS: The ID of the subnets where the task will run.
SECURITY_GROUP_ID: The ID of the security group for the task.
CONTAINER_NAME: The name of the container in the task definition.

Example Values:

ECS_CLUSTER_NAME: ai-challenge-2024-cluster
TASK_DEFINITION: arn:aws:ecs:ap-southeast-1:<aws account id>:task-definition/keyframe-matcher-definition-min:1
SUBNET_IDS: ["subnet-0a1b2c3d4e5f6g7h","subnet-1a2b3c4d5e6f7g8"]
SECURITY_GROUP_ID: sg-0123456789abcdef0
CONTAINER_NAME: keyframe-matcher-container-min

3.2 Run Task By Local CLI

Run the script to create a task to extract keyframes from the video file.

python run-task.py

The script will create a task in the ECS cluster to extract keyframes from the video file. You can monitor the task status in the ECS console.

Docs Version Dropdown

3.3 Verify Keyframes Extraction

After the task completes, you can verify that the keyframes have been extracted and updated with metadata in the S3 bucket.

Navigate to the S3 console.
Open the bucket ai-challenge-2024.
Check the keyframes folder for the extracted keyframes.
Verify that the keyframes have been updated with metadata in the S3 console.
- Video Key: The S3 key of the video file.
- Frame Number: The frame number in the video where the keyframe was found.

Docs Version Dropdown

Check results in the metadata folder for the CSV file containing the keyframe metadata.

Docs Version Dropdown