Skip to content

Commit 88be8a4

Browse files
alokemajumderalokemajumder
committed
Fix: Core APIs
1 parent 43145bd commit 88be8a4

39 files changed

+4222
-639
lines changed

‎.env.example

Lines changed: 163 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,182 @@
1-
# Copy this file to .env and update with your values
1+
# Rendiff FFmpeg API - Production Environment Configuration
2+
# Copy this file to .env and configure for your environment
23

3-
# Database Configuration
4-
POSTGRES_PASSWORD=your_secure_password_here
5-
POSTGRES_USER=ffmpeg_user
6-
POSTGRES_DB=ffmpeg_api
7-
DATABASE_URL=postgresql://ffmpeg_user:your_secure_password_here@postgres:5432/ffmpeg_api
4+
# =============================================================================
5+
# CORE APPLICATION SETTINGS
6+
# =============================================================================
87

9-
# Redis/Queue Configuration
10-
REDIS_URL=redis://redis:6379/0
8+
# Application Environment
9+
DEBUG=false
10+
TESTING=false
11+
VERSION=1.0.0
1112

12-
# API Configuration
13+
# API Server Configuration
1314
API_HOST=0.0.0.0
1415
API_PORT=8000
1516
API_WORKERS=4
16-
LOG_LEVEL=info
17-
DEBUG=false
17+
API_RELOAD=false
18+
API_LOG_LEVEL=info
19+
20+
# =============================================================================
21+
# DATABASE CONFIGURATION
22+
# =============================================================================
23+
24+
# Production PostgreSQL (Recommended)
25+
DATABASE_URL=postgresql://ffmpeg_user:your_secure_password@postgres:5432/ffmpeg_api
26+
27+
# Alternative: SQLite (Development Only)
28+
# DATABASE_URL=sqlite+aiosqlite:///data/rendiff.db
29+
30+
# Database Pool Settings
31+
DATABASE_POOL_SIZE=20
32+
DATABASE_MAX_OVERFLOW=40
33+
34+
# =============================================================================
35+
# QUEUE & CACHE CONFIGURATION
36+
# =============================================================================
37+
38+
# Redis Configuration
39+
REDIS_URL=redis://redis:6379/0
40+
REDIS_MAX_CONNECTIONS=100
1841

1942
# Worker Configuration
2043
WORKER_CONCURRENCY=4
21-
CPU_WORKERS=2
22-
GPU_WORKERS=0
44+
WORKER_PREFETCH_MULTIPLIER=1
45+
WORKER_MAX_TASKS_PER_CHILD=100
46+
WORKER_TASK_TIME_LIMIT=21600
2347

24-
# Storage Configuration
25-
STORAGE_PATH=./storage
48+
# =============================================================================
49+
# STORAGE CONFIGURATION
50+
# =============================================================================
51+
52+
# Storage Paths
2653
STORAGE_CONFIG=/app/config/storage.yml
54+
STORAGE_PATH=./storage
55+
TEMP_PATH=/tmp/rendiff
56+
57+
# Data Persistence Paths (for Docker volumes)
58+
POSTGRES_DATA_PATH=./data/postgres
59+
REDIS_DATA_PATH=./data/redis
60+
PROMETHEUS_DATA_PATH=./data/prometheus
61+
GRAFANA_DATA_PATH=./data/grafana
62+
63+
# =============================================================================
64+
# FFMPEG & PROCESSING CONFIGURATION
65+
# =============================================================================
2766

28-
# Security Configuration
29-
ADMIN_API_KEYS=your_admin_key_1,your_admin_key_2
30-
RENDIFF_API_KEYS=your_client_key_1,your_client_key_2,your_client_key_3
67+
# FFmpeg Settings
68+
FFMPEG_THREADS=0
69+
FFMPEG_PRESET=medium
70+
FFMPEG_CRF=23
71+
FFMPEG_HARDWARE_ACCELERATION=auto
72+
73+
# =============================================================================
74+
# SECURITY & AUTHENTICATION
75+
# =============================================================================
76+
77+
# API Security
78+
API_KEY_HEADER=X-API-Key
3179
ENABLE_API_KEYS=true
32-
CORS_ORIGINS=https://localhost,http://localhost
80+
ENABLE_IP_WHITELIST=false
81+
IP_WHITELIST=10.0.0.0/8,192.168.0.0/16
82+
83+
# Admin API Keys (comma-separated)
84+
ADMIN_API_KEYS=your_admin_key_here
85+
86+
# Rate Limiting
87+
ENABLE_RATE_LIMITING=true
88+
RATE_LIMIT_CALLS=2000
89+
RATE_LIMIT_PERIOD=3600
3390

34-
# SSL/HTTPS Configuration (for production)
35-
DOMAIN_NAME=localhost
36-
CERTBOT_EMAIL=[email protected]
37-
CERT_RESOLVER=letsencrypt
38-
LETSENCRYPT_STAGING=false
91+
# CORS Origins (comma-separated)
92+
API_CORS_ORIGINS=http://localhost,https://localhost,https://yourdomain.com
93+
API_TRUSTED_HOSTS=localhost,yourdomain.com
3994

40-
# Monitoring Configuration
41-
GRAFANA_PASSWORD=your_grafana_password_here
42-
PROMETHEUS_AUTH=admin:your_prometheus_password_here
43-
TRAEFIK_AUTH=admin:your_traefik_password_here
95+
# Database Passwords
96+
POSTGRES_PASSWORD=your_secure_postgres_password
4497

45-
# Resource Limits
98+
# =============================================================================
99+
# MONITORING & OBSERVABILITY
100+
# =============================================================================
101+
102+
# Metrics & Monitoring
103+
ENABLE_METRICS=true
104+
METRICS_PORT=9000
105+
ENABLE_TRACING=false
106+
TRACING_ENDPOINT=
107+
108+
# Grafana Configuration
109+
GRAFANA_PASSWORD=your_secure_grafana_password
110+
111+
# =============================================================================
112+
# RESOURCE LIMITS
113+
# =============================================================================
114+
115+
# Upload & Processing Limits
46116
MAX_UPLOAD_SIZE=10737418240
47-
MAX_CONCURRENT_JOBS_PER_KEY=10
48117
MAX_JOB_DURATION=21600
118+
MAX_CONCURRENT_JOBS_PER_KEY=10
119+
JOB_RETENTION_DAYS=7
120+
121+
# =============================================================================
122+
# WEBHOOKS & NOTIFICATIONS
123+
# =============================================================================
124+
125+
# Webhook Configuration
126+
WEBHOOK_TIMEOUT=30
127+
WEBHOOK_MAX_RETRIES=3
128+
WEBHOOK_RETRY_DELAY=60
129+
130+
# =============================================================================
131+
# OPTIONAL SERVICES
132+
# =============================================================================
133+
134+
# Virus Scanning (Optional)
135+
ENABLE_VIRUS_SCAN=false
136+
CLAMAV_HOST=
137+
CLAMAV_PORT=3310
138+
139+
# =============================================================================
140+
# DEPLOYMENT SPECIFIC
141+
# =============================================================================
142+
143+
# Docker Compose Profiles
144+
# Uncomment the profile you want to use:
145+
# COMPOSE_PROFILES=standard # Standard CPU-only deployment
146+
# COMPOSE_PROFILES=gpu # GPU-accelerated deployment
147+
# COMPOSE_PROFILES=monitoring # Include Prometheus/Grafana
148+
# COMPOSE_PROFILES=gpu,monitoring # GPU + Monitoring
149+
150+
# Network Configuration
151+
# COMPOSE_PROJECT_NAME=ffmpeg-api
152+
153+
# =============================================================================
154+
# CLOUD STORAGE (Optional)
155+
# =============================================================================
156+
157+
# AWS S3
158+
# AWS_ACCESS_KEY_ID=your_access_key
159+
# AWS_SECRET_ACCESS_KEY=your_secret_key
160+
# AWS_DEFAULT_REGION=us-west-2
161+
# S3_BUCKET_NAME=your-video-bucket
162+
163+
# Azure Blob Storage
164+
# AZURE_STORAGE_ACCOUNT=your_account
165+
# AZURE_STORAGE_KEY=your_key
166+
# AZURE_CONTAINER_NAME=videos
167+
168+
# Google Cloud Storage
169+
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
170+
# GCS_BUCKET_NAME=your-video-bucket
171+
172+
# =============================================================================
173+
# SSL/TLS CONFIGURATION
174+
# =============================================================================
175+
176+
# SSL Certificate Paths (for production)
177+
# SSL_CERT_PATH=/etc/ssl/certs/your-cert.crt
178+
# SSL_KEY_PATH=/etc/ssl/private/your-key.key
49179

50-
# External URLs
51-
EXTERNAL_URL=https://localhost
180+
# Let's Encrypt (for automatic SSL)
181+
# [email protected]
182+
# LETSENCRYPT_HOST=yourdomain.com

‎.github/workflows/yaml-lint.yml

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
name: YAML Lint
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
paths:
7+
- '**/*.yml'
8+
- '**/*.yaml'
9+
- '.github/workflows/*.yml'
10+
pull_request:
11+
branches: [ main, develop ]
12+
paths:
13+
- '**/*.yml'
14+
- '**/*.yaml'
15+
- '.github/workflows/*.yml'
16+
17+
jobs:
18+
yaml-lint:
19+
name: YAML Lint
20+
runs-on: ubuntu-latest
21+
22+
steps:
23+
- name: Checkout code
24+
uses: actions/checkout@v4
25+
26+
- name: Set up Python
27+
uses: actions/setup-python@v4
28+
with:
29+
python-version: '3.12'
30+
31+
- name: Install yamllint
32+
run: |
33+
python -m pip install --upgrade pip
34+
pip install yamllint
35+
36+
- name: Create yamllint config
37+
run: |
38+
cat > .yamllint.yml << EOF
39+
extends: default
40+
41+
rules:
42+
# Allow longer lines for docker compose
43+
line-length:
44+
max: 120
45+
level: warning
46+
47+
# Allow multiple spaces after operators
48+
colons:
49+
max-spaces-before: 0
50+
max-spaces-after: -1
51+
52+
# Allow indentation of 2 or 4 spaces
53+
indentation:
54+
spaces: consistent
55+
indent-sequences: true
56+
check-multi-line-strings: false
57+
58+
# Don't require document start markers
59+
document-start: disable
60+
61+
# Allow empty values
62+
empty-values:
63+
forbid-in-block-mappings: false
64+
forbid-in-flow-mappings: false
65+
66+
# Allow trailing spaces in comments
67+
trailing-spaces:
68+
level: warning
69+
70+
# Allow truthy values like 'yes', 'on', etc.
71+
truthy:
72+
allowed-values: ['true', 'false', 'yes', 'no', 'on', 'off']
73+
check-keys: false
74+
EOF
75+
76+
- name: Lint YAML files
77+
run: |
78+
echo "Linting YAML files..."
79+
find . -type f \( -name "*.yml" -o -name "*.yaml" \) -not -path "./.git/*" | while read file; do
80+
echo "Checking: $file"
81+
yamllint "$file"
82+
done
83+
84+
- name: Validate Docker Compose files
85+
run: |
86+
echo "Validating Docker Compose syntax..."
87+
88+
# Check main compose file
89+
if [ -f "compose.yml" ]; then
90+
echo "Validating compose.yml..."
91+
docker compose -f compose.yml config > /dev/null
92+
fi
93+
94+
# Check production compose file
95+
if [ -f "compose.prod.yml" ]; then
96+
echo "Validating compose.prod.yml..."
97+
docker compose -f compose.prod.yml config > /dev/null
98+
fi
99+
100+
# Check stable compose file
101+
if [ -f "compose.stable.yml" ]; then
102+
echo "Validating compose.stable.yml..."
103+
docker compose -f compose.stable.yml config > /dev/null
104+
fi
105+
106+
# Check override file
107+
if [ -f "compose.override.yml" ]; then
108+
echo "Validating compose.override.yml..."
109+
docker compose -f compose.yml -f compose.override.yml config > /dev/null
110+
fi
111+
112+
echo "All Docker Compose files are valid!"
113+
114+
- name: Check for common issues
115+
run: |
116+
echo "Checking for common Docker Compose issues..."
117+
118+
# Check for deprecated version field
119+
if grep -r "version:" . --include="*.yml" --include="*.yaml" --exclude-dir=.git; then
120+
echo "::warning::Found 'version:' field in compose files. This is deprecated in modern Docker Compose."
121+
fi
122+
123+
# Check for hardcoded localhost
124+
if grep -r "localhost" . --include="compose*.yml" --exclude-dir=.git; then
125+
echo "::warning::Found hardcoded 'localhost' in compose files. Consider using service names."
126+
fi
127+
128+
# Check for missing health checks on databases
129+
if grep -A 10 "image.*postgres" . --include="compose*.yml" --exclude-dir=.git | grep -v "healthcheck:" > /dev/null; then
130+
echo "::warning::PostgreSQL services should have health checks defined."
131+
fi
132+
133+
echo "Common issues check completed!"

‎.structure.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Optimized Repository Structure
2+
3+
## Core Structure
4+
```
5+
ffmpeg-api/
6+
├── 📁 src/ # Source code (renamed from api/)
7+
│ ├── 📁 api/ # API layer
8+
│ ├── 📁 core/ # Core business logic
9+
│ ├── 📁 models/ # Data models
10+
│ ├── 📁 services/ # Business services
11+
│ └── 📁 utils/ # Utilities
12+
├── 📁 workers/ # Worker processes
13+
├── 📁 tests/ # Test suite
14+
├── 📁 deployment/ # Deployment configs
15+
│ ├── 📁 docker/ # Docker configurations
16+
│ ├── 📁 k8s/ # Kubernetes manifests
17+
│ └── 📁 compose/ # Docker Compose files
18+
├── 📁 config/ # Configuration files
19+
├── 📁 docs/ # Documentation
20+
├── 📁 scripts/ # Utility scripts
21+
└── 📁 monitoring/ # Monitoring and observability
22+
23+
## Changes Made:
24+
1. Consolidated API code under src/
25+
2. Moved deployment files to deployment/
26+
3. Cleaned up root directory
27+
4. Better separation of concerns
28+
5. Removed redundant files
29+
```

0 commit comments

Comments
 (0)