Skip to content

Commit 2db6053

Browse files
AtticdmAtticdm
committed
Initial snapshot
0 parents  commit 2db6053

File tree

129 files changed

+9772
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

129 files changed

+9772
-0
lines changed

‎.env.example‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
APP_ENV=dev
2+
JWT_SECRET=dev_secret_key_xyz
3+
POSTGRES_DSN=postgresql://glocal_user:strongpass123@postgres:5432/glocal_db
4+
REDIS_URL=redis://redis:6379/0
5+
RABBITMQ_URL=amqp://glocal:glocalpass@rabbitmq:5672/
6+
S3_ENDPOINT=http://minio:9000
7+
S3_REGION=eu-central-1
8+
S3_BUCKET=glocal-media
9+
S3_ACCESS_KEY=minioadmin
10+
S3_SECRET_KEY=minioadmin
11+
S3_PUBLIC_URL=http://localhost:9000
12+
API_BASE_URL=http://api:8080
13+
PUBLIC_API_URL=http://localhost:8080

‎.github/workflows/ci.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
python:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v3
13+
- uses: actions/setup-python@v4
14+
with:
15+
python-version: "3.11"
16+
- name: Install dependencies
17+
run: |
18+
pip install -r apps/api/requirements.txt
19+
pip install -r apps/api/requirements-dev.txt
20+
for req in services/*/requirements.txt; do pip install -r "$req"; done
21+
- name: Ruff
22+
run: ruff check apps/api services packages
23+
- name: Black
24+
run: black --check apps/api services packages
25+
- name: Mypy
26+
run: mypy apps/api/app
27+
28+
frontend:
29+
runs-on: ubuntu-latest
30+
defaults:
31+
run:
32+
working-directory: apps/frontend
33+
steps:
34+
- uses: actions/checkout@v3
35+
- uses: actions/setup-node@v4
36+
with:
37+
node-version: "20"
38+
- name: Enable corepack
39+
run: corepack enable
40+
- name: Install dependencies
41+
run: pnpm install --frozen-lockfile
42+
- name: Lint
43+
run: pnpm lint

‎.gitignore‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Node
2+
node_modules
3+
.pnpm-store
4+
.next
5+
out
6+
7+
# Python
8+
__pycache__
9+
*.pyc
10+
.venv
11+
venv
12+
.envrc
13+
14+
# macOS
15+
.DS_Store
16+
17+
# Docker
18+
*.log
19+
20+
# IDE
21+
.idea
22+
.vscode
23+
24+
# Build artifacts
25+
coverage
26+
.mypy_cache
27+
ruff_cache
28+
29+
# Python lock
30+
poetry.lock
31+
pip-wheel-metadata
32+
33+
# Misc
34+
.env

‎ARCHITECTURE.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Glocal Ads AI Architecture
2+
3+
## Overview
4+
5+
The repository is a mono-repo containing a Next.js 14 frontend, a FastAPI backend, and a suite of Python worker services connected through RabbitMQ and Redis. Media assets are stored in MinIO (S3 compatible). Postgres stores relational data and Redis offers pub/sub for real-time status updates.
6+
7+
```
8+
frontend (Next.js)
9+
↕ REST/SSE
10+
FastAPI BFF
11+
↕ SQLAlchemy/Redis/RabbitMQ/S3
12+
RabbitMQ (jobs exchange) <—> Orchestrator <—> Worker micro-services
13+
Redis (pub/sub) <—> Frontend SSE progress
14+
Postgres <—> FastAPI + workers
15+
MinIO (S3 bucket) ⇄ Media ingestion & artifacts
16+
```
17+
18+
## Applications
19+
20+
### Frontend (`apps/frontend`)
21+
* Next.js 14 App Router with TypeScript.
22+
* Authentication via FastAPI (`/auth/sign-in`). Token stored in local storage.
23+
* Pages:
24+
* `/` — project list + creation.
25+
* `/projects/[id]` — asset upload to MinIO, localization job form.
26+
* `/jobs/[id]` — SSE-driven pipeline progress per language/stage.
27+
* `/results/[jobId]` — variant previews (HLS) with download & YouTube actions.
28+
* Uses `react-hook-form`, shadcn-inspired UI primitives, `hls.js` for preview.
29+
30+
### API (`apps/api`)
31+
* FastAPI + SQLAlchemy 2.0 + async Postgres (psycopg3).
32+
* Provides auth, project & asset management, job orchestration endpoints, SSE progress stream, variant downloads, and YouTube stub.
33+
* Integrations: Redis pub/sub, RabbitMQ (via `aio-pika`), MinIO (boto3), JWT auth, Alembic-like SQL migrations (raw SQL in `migrations/sql`).
34+
* On startup ensures S3 bucket policy via MinIO client and seeds admin/demo data.
35+
36+
## Services (`services/*`)
37+
* Shared utilities packaged in `packages/service-kit` (config, DB helpers, S3, RabbitMQ, Redis progress helper, path helpers).
38+
* Each worker listens to a dedicated routing key (`stage.<name>`) and publishes completion/error events back (`stage.<name>.completed|failed`).
39+
* Orchestrator sequences stages per language, updates DB, emits Redis progress, and optionally triggers YouTube uploads.
40+
* Workers emulate the media pipeline:
41+
* `asr-agent`: generates dummy segments & transcript.
42+
* `translate-agent`: applies pseudo translation with suffix `[lang]`.
43+
* `tts-agent`: synthesises sine-wave speech from segments.
44+
* `mix-agent`: runs FFmpeg to mux original video + TTS, produces MP4 + HLS.
45+
* `subs-agent`: builds SRT/VTT from translated segments.
46+
* `textinframe-agent`: overlays localized text via FFmpeg drawtext + new HLS.
47+
* `qc-agent`: probes final output and writes JSON QC report.
48+
* `yt-uploader`: logs pseudo YouTube URL and notifies Redis.
49+
50+
## Messaging Flow
51+
52+
1. API `/jobs` inserts job + variants and publishes `job.created` message.
53+
2. Orchestrator consumes `job.created`, sets variants to `processing`, queues first stage (`stage.asr`).
54+
3. Each worker retrieves job/variant context from Postgres via service kit, reads/writes artifacts in MinIO, updates DB fields, and publishes progress using Redis + `stage.<stage>.completed` message.
55+
4. Orchestrator hears completion events, queues next stage (skipping optional ones based on job options), and finally marks variant/job done.
56+
5. Frontend subscribes via `/jobs/{id}/stream` SSE channel and renders per-stage updates.
57+
58+
## Storage Layout
59+
60+
MinIO bucket `glocal-media` stores assets:
61+
* `raw/{projectId}/{assetId}/source.mp4`
62+
* `jobs/{jobId}/{lang}/asr/segments.json`
63+
* `jobs/{jobId}/{lang}/tts/track.wav`
64+
* `jobs/{jobId}/{lang}/mix/out.mp4`
65+
* `jobs/{jobId}/{lang}/mix/hls/...`
66+
* `jobs/{jobId}/{lang}/subs/subtitles.(srt|vtt)`
67+
* `jobs/{jobId}/{lang}/textinframe/out.mp4`
68+
* `jobs/{jobId}/{lang}/qc/report.json`
69+
70+
Postgres tables follow schema defined in `migrations/sql/001_init.sql` (users, projects, assets, jobs, variants, voice profiles, glossaries).
71+
72+
## Infrastructure
73+
74+
* Dockerfiles live in `infrastructure/docker` for API, frontend, and generic Python services.
75+
* `docker-compose.yml` orchestrates Postgres, Redis, RabbitMQ, MinIO (+ mc bootstrap), API, frontend, orchestrator, all agents, and migration job.
76+
* Basic Kubernetes manifests provided under `infrastructure/k8s/` as a starting point.
77+
78+
## Scripts & Automation
79+
80+
* `scripts/dev/generate-test-video.sh` — produces an 8s synthetic demo video.
81+
* `scripts/dev/smoke.sh` — end-to-end happy-path using FastAPI endpoints.
82+
* GitHub Actions workflow (`ci.yml`) runs Ruff/Black/Mypy and Next.js lint.
83+
84+
## Notes
85+
86+
* SSE endpoints and variant preview/download accept either `Authorization` header or `?token=` query string for EventSource/video tags.
87+
* Bucket policy is configured for anonymous read to support direct HLS loads.
88+
* The architecture is modular to later swap emulators with real ML models (WhisperX, NLLB/LLM, XTTS, etc.) without changing orchestration plumbing.

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 GloCal
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Glocal Ads AI
2+
3+
A mono-repo for the Glocal Ads AI localization platform. It ships a Next.js 14 frontend, FastAPI backend, and a fleet of Python workers that emulate the localization pipeline (ASR → Translate → TTS → Mix → Subs → Text-in-frame → QC → Pack). Services communicate via RabbitMQ, store metadata in Postgres, share progress through Redis SSE, and persist media artefacts in MinIO.
4+
5+
## Prerequisites
6+
7+
* Docker & Docker Compose
8+
* Node.js 20 (for local Next.js development) with `pnpm`
9+
* Python 3.11+
10+
* FFmpeg for local smoke scripts
11+
12+
## Initial Setup
13+
14+
```bash
15+
# install frontend dependencies and generate lockfile
16+
cd apps/frontend
17+
pnpm install
18+
cd ../..
19+
20+
# install Python dependencies (API + services)
21+
python3 -m pip install -r apps/api/requirements.txt -r apps/api/requirements-dev.txt
22+
for req in services/*/requirements.txt; do python3 -m pip install -r "$req"; done
23+
```
24+
25+
Populate a demo source clip:
26+
27+
```bash
28+
scripts/dev/generate-test-video.sh
29+
```
30+
31+
## Running with Docker Compose
32+
33+
```bash
34+
docker compose build
35+
docker compose up -d
36+
```
37+
38+
Services exposed:
39+
40+
* UI — http://localhost:3000
41+
* API docs — http://localhost:8080/docs
42+
* MinIO console — http://localhost:9001 (minioadmin/minioadmin)
43+
* RabbitMQ console — http://localhost:15672 (glocal/glocalpass)
44+
45+
Check status:
46+
47+
```bash
48+
docker compose ps
49+
```
50+
51+
## Smoke Test
52+
53+
After the stack is healthy:
54+
55+
```bash
56+
bash scripts/dev/smoke.sh
57+
```
58+
59+
The script logs in, uploads `assets/source.mp4`, launches a job for `es` and `pt-BR`, polls until completion, and prints download URLs.
60+
61+
## Local Development
62+
63+
### Frontend (Next.js)
64+
65+
```bash
66+
cd apps/frontend
67+
pnpm dev
68+
```
69+
70+
### API (FastAPI)
71+
72+
```bash
73+
cd apps/api
74+
uvicorn app.main:app --reload --host 0.0.0.0 --port 8080
75+
```
76+
77+
Workers can be run individually, e.g.:
78+
79+
```bash
80+
cd services/asr-agent
81+
python main.py
82+
```
83+
84+
## Scripts
85+
86+
* `scripts/dev/generate-test-video.sh` — generate demo video (8s, 1920×1080).
87+
* `scripts/dev/smoke.sh` — end-to-end happy-path automation.
88+
89+
## Testing & Linting
90+
91+
```bash
92+
# Ruff + Black + Mypy
93+
ruff check apps/api services packages
94+
black apps/api services packages
95+
mypy apps/api/app
96+
97+
# Frontend lint
98+
cd apps/frontend
99+
pnpm lint
100+
```
101+
102+
CI is configured in `.github/workflows/ci.yml` to execute the same checks.
103+
104+
## Environment Variables
105+
106+
The root `.env` powers Docker Compose and runtime services. Key values:
107+
108+
```
109+
APP_ENV=dev
110+
JWT_SECRET=dev_secret_key_xyz
111+
POSTGRES_DSN=postgresql://glocal_user:strongpass123@postgres:5432/glocal_db
112+
REDIS_URL=redis://redis:6379/0
113+
RABBITMQ_URL=amqp://glocal:glocalpass@rabbitmq:5672/
114+
S3_ENDPOINT=http://minio:9000
115+
S3_PUBLIC_URL=http://localhost:9000
116+
S3_BUCKET=glocal-media
117+
S3_ACCESS_KEY=minioadmin
118+
S3_SECRET_KEY=minioadmin
119+
API_BASE_URL=http://api:8080
120+
PUBLIC_API_URL=http://localhost:8080
121+
```
122+
123+
Frontend build-time variables live in `apps/frontend/.env.local` and mirror the public endpoints.
124+
125+
## Project Structure
126+
127+
```
128+
apps/
129+
api/ FastAPI BFF
130+
frontend/ Next.js App Router UI
131+
services/ RabbitMQ worker fleet (orchestrator + agents)
132+
packages/
133+
shared-schemas/ Pydantic models shared across services
134+
service-kit/ Shared worker utilities (DB, S3, RabbitMQ, Redis)
135+
infrastructure/
136+
docker/ Dockerfiles
137+
k8s/ Example Kubernetes manifests
138+
migrations/sql/ SQL migrations & seed data
139+
scripts/dev/ Developer utilities
140+
```
141+
142+
See [ARCHITECTURE.md](ARCHITECTURE.md) for a deeper dive.

‎apps/api/app/__init__.py‎

Whitespace-only changes.

‎apps/api/app/__main__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from __future__ import annotations
2+
3+
import uvicorn
4+
5+
if __name__ == "__main__":
6+
uvicorn.run("app.main:app", host="0.0.0.0", port=8080, reload=False)

‎apps/api/app/api/__init__.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
from . import (
2+
routes_assets,
3+
routes_auth,
4+
routes_jobs,
5+
routes_projects,
6+
routes_variants,
7+
routes_voice,
8+
routes_youtube,
9+
)
10+
11+
__all__ = [
12+
"routes_assets",
13+
"routes_auth",
14+
"routes_jobs",
15+
"routes_projects",
16+
"routes_variants",
17+
"routes_voice",
18+
"routes_youtube",
19+
]

0 commit comments

Comments
 (0)