Video-to-Text Summary & Analysis with AWS BDA and Claude

• This application provides video analysis capabilities using AWS BDA and Claude AI.
• It features a unified pipeline that converts WEBM files to MP4, processes videos through AWS BDA, and performs AI-powered analysis of tasks and activities.
• The system includes intelligent processing checks, failed processing cleanup, and comprehensive MongoDB tracking.

Architecture

1. Pipeline Execution

sequenceDiagram
    participant U as User
    participant APP as app.py
    participant ENV as Environment
    participant CONV as WEBM Converter
    participant BDA as BDA Processor
    
    U->>APP: python app.py [command]
    APP->>ENV: Validate environment variables
    ENV-->>APP: Validation result
    
    alt command == 'full'
        APP->>APP: run_full_pipeline()
        APP->>CONV: run_conversion_phase()
        CONV-->>APP: Conversion results
        APP->>APP: Wait for S3 consistency
        APP->>BDA: run_processing_phase()
        BDA-->>APP: Processing results
        APP->>APP: Generate final summary
    else command == 'convert'
        APP->>CONV: run_conversion_phase()
    else command == 'process'
        APP->>BDA: run_processing_phase()
    else command == 'scan'
        APP->>APP: scan_and_display()
    end
    
    APP-->>U: Pipeline results

Loading

2. WEBM to MP4 Conversion

flowchart TD
    Start([Start Conversion]) --> CheckRole{MEDIA_ROLE_ARN<br/>Configured?}
    CheckRole -->|No| Skip[Skip Conversion]
    CheckRole -->|Yes| ScanWEBM[Scan S3 for WEBM Files]
    
    ScanWEBM --> ParseFiles[Parse Filename Info]
    ParseFiles --> GroupByDate[Group by Date]
    GroupByDate --> GroupByTime[Group by Time Proximity]
    
    GroupByTime --> CheckExisting[Check Existing MP4 Files]
    CheckExisting --> FilterGroups{Already<br/>Converted?}
    FilterGroups -->|Yes| SkipGroup[Skip Group]
    FilterGroups -->|No| CreateJob[Create MediaConvert Job]
    
    CreateJob --> SubmitJob[Submit to MediaConvert]
    SubmitJob --> PollStatus[Poll Job Status]
    PollStatus --> JobDone{Job<br/>Complete?}
    JobDone -->|No| Wait[Wait 5 seconds]
    Wait --> PollStatus
    JobDone -->|Yes| Success[Conversion Success]
    
    SkipGroup --> NextGroup{More<br/>Groups?}
    Success --> NextGroup
    NextGroup -->|Yes| FilterGroups
    NextGroup -->|No| Complete[All Conversions Complete]
    
    Skip --> Complete

Loading

3. Video Processing with Checks

flowchart TD
    Start([Start BDA Processing]) --> ScanMP4[Scan S3 for MP4 Files]
    ScanMP4 --> CheckDB{Check MongoDB<br/>Status}
    
    CheckDB -->|completed| SkipVideo[Skip - Already Processed]
    CheckDB -->|processing/queued| SkipVideo2[Skip - Currently Processing]
    CheckDB -->|failed/error| Cleanup[Cleanup Failed Processing]
    CheckDB -->|not found| Process[Process Video]
    
    Cleanup --> CleanupBDA[Delete Failed BDA Project]
    CleanupBDA --> CleanupS3[Remove S3 Artifacts]
    CleanupS3 --> CleanupMongo[Remove Output Records]
    CleanupMongo --> ResetStatus[Reset Status to Queued]
    ResetStatus --> Process
    
    Process --> CreateMeta[Create Metadata Entry]
    CreateMeta --> CreateProject[Create BDA Project]
    CreateProject --> UpdateStatus1[Update Status: Processing]
    UpdateStatus1 --> InvokeJob[Invoke BDA Job]
    InvokeJob --> UpdateStatus2[Update with Invocation ARN]
    UpdateStatus2 --> PollBDA[Poll BDA Job Status]
    
    PollBDA --> JobStatus{Job<br/>Status?}
    JobStatus -->|Processing| Wait[Wait 5 seconds]
    Wait --> PollBDA
    JobStatus -->|Success| ProcessResults[Process BDA Results]
    JobStatus -->|Failed| HandleFailure[Handle Failure]
    
    ProcessResults --> ExtractData[Extract Transcript, Chapters, etc.]
    ExtractData --> ClaudeAnalysis[Perform Claude Analysis]
    ClaudeAnalysis --> BuildOutput[Build Complete Output Structure]
    BuildOutput --> StoreMongo[Store in MongoDB]
    StoreMongo --> SaveS3[Save JSON to S3]
    SaveS3 --> UpdateCompleted[Update Status: Completed]
    UpdateCompleted --> CleanupProject[Cleanup BDA Project]
    
    HandleFailure --> CleanupFailed[Cleanup Failed Resources]
    CleanupFailed --> UpdateFailed[Update Status: Failed]
    
    SkipVideo --> NextVideo{More<br/>Videos?}
    SkipVideo2 --> NextVideo
    CleanupProject --> NextVideo
    UpdateFailed --> NextVideo
    NextVideo -->|Yes| CheckDB
    NextVideo -->|No| Complete[Processing Complete]

Loading

4. BDA Project Management

sequenceDiagram
    participant BDA as BDA Processor
    participant AWS as AWS BDA Service
    participant S3 as S3 Storage
    participant MONGO as MongoDB
    
    BDA->>AWS: Create BDA Project
    AWS-->>BDA: Return Project ARN
    BDA->>MONGO: Update bda_info with project details
    
    BDA->>AWS: Invoke BDA Job (project + video)
    AWS-->>BDA: Return Invocation ARN
    BDA->>MONGO: Update bda_info with invocation details
    
    loop Poll Job Status (No Max Attempts)
        BDA->>AWS: Get job status
        AWS-->>BDA: Current status
        alt Status is Success/Error
            BDA->>BDA: Break polling loop
        else Status is Processing
            BDA->>BDA: Wait 5 seconds
        end
    end
    
    alt Job Successful
        BDA->>S3: Read BDA output configuration
        S3-->>BDA: Configuration JSON
        BDA->>S3: Read main results JSON
        S3-->>BDA: Analysis results
        BDA->>BDA: Process results + Claude analysis
        BDA->>MONGO: Store complete output
        BDA->>S3: Save analysis JSON
        BDA->>AWS: Delete BDA project (cleanup)
    else Job Failed
        BDA->>AWS: Delete BDA project (cleanup)
        BDA->>S3: Remove partial outputs
        BDA->>MONGO: Update status to failed
    end

Loading

5. Claude Analysis

sequenceDiagram
    participant BDA as BDA Processor
    participant DE as Data Extractor
    participant CLAUDE as Claude Analysis
    participant BR as Bedrock Runtime
    participant MONGO as MongoDB
    
    BDA->>DE: Extract transcript data
    DE-->>BDA: Raw transcript text
    BDA->>DE: Extract chapters data
    DE-->>BDA: Chapters + chapter transcripts
    BDA->>DE: Extract bounding boxes
    DE-->>BDA: Visual elements data
    BDA->>DE: Extract shots data
    DE-->>BDA: Video shots timeline
    
    BDA->>BDA: Prepare visual context
    Note over BDA: Combine bounding boxes, detected text,<br/>visual timeline, shots
    
    BDA->>CLAUDE: analyze_transcript_with_claude()
    Note over CLAUDE: transcript_text, chapter_transcripts,<br/>video_summary, visual_context
    
    CLAUDE->>BR: Send analysis prompt to Claude Sonnet 3.5
    BR-->>CLAUDE: AI analysis response
    
    CLAUDE->>CLAUDE: Parse and structure response
    CLAUDE-->>BDA: Structured task insights
    
    BDA->>BDA: Build complete output structure
    Note over BDA: Combine BDA results + Claude analysis<br/>+ task categorization
    
    BDA->>MONGO: Store complete analysis
    BDA->>MONGO: Update metadata status

Loading

6. MongoDB Data Flow

sequenceDiagram
    participant APP as Pipeline
    participant META as video_metadata
    participant OUTPUT as video_output
    participant BDA as BDA Processor
    
    APP->>META: Check processing status
    META-->>APP: Return existing record or null
    
    alt Record exists
        alt Status is completed
            APP->>APP: Skip processing
        else Status is failed
            APP->>BDA: cleanup_failed_processing()
            BDA->>META: Reset status to queued
            BDA->>OUTPUT: Delete failed output
        else Status is processing
            APP->>APP: Skip (currently processing)
        end
    else No record exists
        APP->>META: Create new metadata entry
        META-->>APP: Return file_id
    end
    
    APP->>BDA: process_single_video()
    BDA->>META: Update status: processing
    BDA->>META: Update bda_info (project_arn)
    BDA->>META: Update bda_info (invocation_arn)
    
    loop BDA Processing
        BDA->>BDA: Poll job status
    end
    
    alt Processing successful
        BDA->>OUTPUT: Store complete analysis
        OUTPUT-->>BDA: Confirm storage
        BDA->>META: Update status: completed
        BDA->>META: Add output_doc_id reference
        BDA->>META: Add output_json_s3_uri
    else Processing failed
        BDA->>META: Update status: failed
        BDA->>META: Add error details
    end

Loading

Installation and Usage

Set up the AWS policies and verify Trust relationships for the MediaConvert and Bedrock Data Automation roles. Ensure you have the necessary permissions to access S3, Bedrock, and MediaConvert services and are attached to the appropriate IAM roles.

# FFmpeg is required for video processing and format conversion.
brew install ffmpeg # On macOS

choco install ffmpeg # On Windows (requires Chocolatey)

git clone https://github.com/sujeethshingade/video-to-text.git
cd video-to-text

# Backend
cd server

python -m venv venv
venv\Scripts\activate  # On Windows
source venv/bin/activate  # On macOS/Linux

pip install -r requirements.txt

cp .env.example .env # Create .env file

# Running locally (uploading video from frontend)
python main.py

# Running in pipeline in AWS (using videos stored in S3)
python app.py          # Full pipeline
python app.py convert  # WEBM conversion only  
python app.py process  # BDA processing only
python app.py scan     # Scan files

# Frontend
cd client

cp .env.example .env.local # Create .env.local file

npm install
npm run dev

View S3 Bucket Contents

# List all folders (prefixes)
python s3_viewer.py folders

# List files in a specific folder
python s3_viewer.py list --prefix "output/"

# Download a specific file
python s3_viewer.py download "output/123abc/result.json" --output "./downloads"

# Download a directory preserving its structure
python s3_viewer.py download-directory "output/123abc/" --output "./downloads"

# Download a directory to the current location (will create a directory named after the last part of the prefix)
python s3_viewer.py download-directory "output/123abc/"

# Download a directory that contains video processing results 
python s3_viewer.py download-directory "videos/output/file_id/" --output "./results"

# Show more files (default is 500)
python s3_viewer.py list --max 1000

# Delete a specific file
python s3_viewer.py delete-file "output/123abc/result.json"

# Delete a directory and all its contents
python s3_viewer.py delete-directory "output/123abc/"

Cost and Limits

• AWS Bedrock Pricing
• AWS Bedrock Data Automation Limits
• AWS Elemental MediaConvert Pricing

References

• AWS Bedrock Data Automation Video Output Documentation
• Amazon Bedrock Data Automation Video Workshop
• Boto3 Bedrock Data Automation Project API
• AWS Blog: Video Summarization with AI/ML Services
• AWS Bedrock Claude Model Parameters
• AWS Bedrock Batch Inference Data Documentation
• AWS Elemental MediaConvert
• AWS Elemental MediaConvert Endpoints
• AWS Directory Service Edit Trust Documentation