Run ECS Task
1 Prepare Script to Run Task
import os
import boto3
s3_client = boto3.client("s3")
ecs_client = boto3.client("ecs")
def lambda_handler():
cluster_name = os.environ["ECS_CLUSTER_NAME"]
task_definition = os.environ["TASK_DEFINITION"]
subnet_id = os.environ["SUBNET_ID"]
security_group_id = os.environ["SECURITY_GROUP_ID"]
# # Cấu hình network cho ECS task
network_config = {
"awsvpcConfiguration": {
"subnets": [subnet_id],
"securityGroups": [security_group_id],
"assignPublicIp": "ENABLED",
}
}
keyframes_urls = [
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Keyframes_L01.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Keyframes_L03.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Keyframes_L04.zip",
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Keyframes_L05.zip",
"https://atm249496-s3user.vcos.cloudstorage.com.vn/aic24-b6/Keyframes_L06.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Keyframes_L07.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Keyframes_L08.zip",
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Keyframes_L09.zip",
"https://atm249496-s3user.vcos.cloudstorage.com.vn/aic24-b6/Keyframes_L10.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Keyframes_L11.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Keyframes_L12.zip",
]
video_urls = [
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Videos_L01.zip",
"https://atm249496-s3user.vcos.cloudstorage.com.vn/aic24-b6/Videos_L02.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Videos_L03.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Videos_L04.zip",
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Videos_L05.zip",
"https://atm249496-s3user.vcos.cloudstorage.com.vn/aic24-b6/Videos_L06.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Videos_L07.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Videos_L08.zip",
"https://atm249495-s3user.vcos.cloudstorage.com.vn/aic24-b5/Videos_L09.zip",
"https://atm249496-s3user.vcos.cloudstorage.com.vn/aic24-b6/Videos_L10.zip",
"https://atm249497-s3user.vcos.cloudstorage.com.vn/aic24-b7/Videos_L11.zip",
"https://atm249498-s3user.vcos.cloudstorage.com.vn/aic24-b8/Videos_L12.zip",
]
for zip_file_url in keyframes_urls:
run_task_input = {
"cluster": cluster_name,
"launchType": "FARGATE",
"taskDefinition": task_definition,
"networkConfiguration": network_config,
"overrides": {
"containerOverrides": [
{
"name": "keyframes-container",
"environment": [
{
"name": "URL_TO_DOWNLOAD",
"value": zip_file_url,
},
{
"name": "S3_BUCKET_NAME",
"value": "ai-challenge-2024",
},
{
"name": "S3_FOLDER_NAME",
"value": "keyframes",
},
],
}
]
},
}
response = ecs_client.run_task(**run_task_input)
for zip_file_url in video_urls:
run_task_input = {
"cluster": cluster_name,
"launchType": "FARGATE",
"taskDefinition": task_definition,
"networkConfiguration": network_config,
"overrides": {
"containerOverrides": [
{
"name": "keyframes-container",
"environment": [
{
"name": "URL_TO_DOWNLOAD",
"value": zip_file_url,
},
{
"name": "S3_BUCKET_NAME",
"value": "ai-challenge-2024",
},
{
"name": "S3_FOLDER_NAME",
"value": "video",
},
],
}
]
},
}
response = ecs_client.run_task(**run_task_input)
return {
"statusCode": 200,
"body": f'Task created successfully with Task ARN: {response["tasks"][0]["taskArn"]}',
}
if __name__ == "__main__":
lambda_handler()
Environment Variables
Below are the key environment variables required for setting up and running the ECS task:
ECS_CLUSTER_NAME: Specifies the name of the ECS cluster where the task will be deployed.TASK_DEFINITION: Defines the name of the ECS task definition used for the task.SUBNET_ID: The subnet ID where the ECS task will be launched, determining the network within the VPC.SECURITY_GROUP_ID: The security group ID that defines firewall rules and access permissions for the ECS task.CONTAINER_NAME: Identifies the name of the container in the ECS task definition.
Example Configuration
ECS_CLUSTER_NAME: ai-challenge-2024
TASK_DEFINITION: keyframes-task
SUBNET_ID: subnet-0f9d3b7b1b7b7b7b7
SECURITY_GROUP_ID: sg-0f9d3b7b1b7b7b7b7
CONTAINER_NAME: keyframes-container
Each of these variables plays a crucial role in the task setup, ensuring proper deployment and access control within your AWS environment.
2 Run Task
Run the script to ingest data to S3:
python run_task.py
The script will create tasks to download and ingest data to S3.
3 Monitor Task
- Navigate to CloudWatch by clicking on the Services menu, under the Management & Governance section.
- Click on Log groups in the left navigation pane.
- Click on the log group with the name
/ecs/keyframes-task. - Click on the log stream with the latest log stream name.
You can monitor the logs of the task to check the status of the task.
4 Check Data in S3
- Navigate to S3 by clicking on the Services menu, under the Storage section.
- Click on the bucket name ai-challenge-2024.
- Check the folders
keyframesandvideoto see the ingested data.
- Check the total size of the
keyframesandvideofolders:- Navigate to the
keyframesfolder. - Click on the Actions dropdown.
- Click on Calculate total size.
- Repeat the above steps for the
videofolder. - keyframes: 18.7 GB
- video: 55.6 GB
- Navigate to the