Deploy with Terraform

Overview

Semantic Video Search Vector Database

We will deploy a search-based system using an existing OpenSearch domain on AWS, managed via Terraform. The deployment consists of five main modules:

  1. Setup: This module sets up the foundational infrastructure:

    • Create an ECR (Elastic Container Registry) repository.
    • Build Docker images and push them to the ECR repository.
    • Create a SageMaker endpoint for machine learning tasks.
    • Optionally, create a new OpenSearch domain if you do not wish to use the provided data and domain.

  2. Processing Data: This module handles the extraction of keyframes from video data:

    • Create ECS (Elastic Container Service) tasks to process videos and extract keyframes.
    • The keyframes are then stored in an S3 bucket for further processing.

  3. Embedding: This module generates vector embeddings from the extracted keyframes:

    • Use SageMaker processing jobs to perform embedding generation for the extracted keyframes.
    • The generated embeddings are stored in S3 or directly indexed into OpenSearch.

  4. VectorDB: This module focuses on managing the OpenSearch domain:

    • Perform indexing of the generated embeddings in the OpenSearch domain.
    • Enable querying of vector data to support semantic search capabilities.

  5. CMS (Content Management System): This module deploys a backend service for managing content and search functionalities:

    • Set up API Gateway for RESTful APIs.
    • Use Cognito for authentication and user management.
    • S3 for storing static assets and data.
    • CloudFront for content delivery and caching.

The system is deployed based on the existing OpenSearch domain and indexed data using these five modules. If you want to use your own video data, you can create a new OpenSearch domain and customize the deployment.

Content

  1. Setup
  2. Processing Data
  3. Embedding
  4. VectorDB
  5. CMS