TTV Middleware

Provider-agnostic Text-to-Video middleware with async polling, retry logic, and comprehensive error handling. Currently supports OpenAI Sora (Sora 2, Sora 2 Pro) and Google Veo (Veo 2, Veo 3, Veo 3.1). Features image-to-video, video extension, configurable progress callbacks, and optional auto-download.

Table of Contents

• Features
• Quick Start
• Prerequisites
• Configuration
• Providers & Models
• Image-to-Video
• Video Extension
• API Reference
• Advanced Features
• Testing
• Contributing
• License
• Links

---

Features

• Multi-Provider Architecture: Unified API for all TTV providers
  • OpenAI Sora: Sora 2 & Sora 2 Pro with text-to-video and image-to-video
  • Google Veo: Veo 2, Veo 3, Veo 3.1 (+ fast variants) with video extension support
• Async Polling: Both APIs are asynchronous - the middleware handles polling internally with configurable intervals and exponential backoff
• Image-to-Video: Animate a still image into a video (both providers)
• Video Extension: Continue/extend an existing video (Google Veo)
• Progress Callbacks: Optional onProgress callback for real-time generation status updates
• Auto-Download: Optionally download generated videos to Buffer (downloadToBuffer)
• Retry Logic: Exponential backoff with jitter for transient errors (429, 408, 5xx, timeouts)
• TypeScript First: Full type safety with comprehensive interfaces
• Logging Control: Configurable log levels via environment or API
• Debug Logging: Markdown file logging for debugging prompts and responses
• Error Handling: Typed error classes including PollingTimeoutError and ContentModeratedError
• Dry Mode: Validate requests without API calls (no costs during development)

Quick Start

Installation

Install from npm:

npm install @loonylabs/ttv-middleware

# For OpenAI Sora provider:
npm install openai

# For Google Veo provider:
npm install google-auth-library

Or install directly from GitHub:

npm install github:loonylabs-dev/ttv-middleware

Basic Usage

import { TTVService, OpenAISoraProvider, TTVProvider } from '@loonylabs/ttv-middleware';

// Create service and register provider
const service = new TTVService();
service.registerProvider(new OpenAISoraProvider({
  apiKey: process.env.OPENAI_API_KEY,
}));

// Generate a video
const result = await service.generate({
  prompt: 'A cat sitting on a windowsill watching rain fall outside',
  model: 'sora-2',
  duration: 8,         // 8 seconds
  aspectRatio: '16:9',
});

console.log('Video URL:', result.videos[0].url);
console.log('Duration:', result.metadata.duration, 'ms (total)');
console.log('Generation time:', result.metadata.generationTime, 'ms');

Using Google Veo

import { TTVService, GoogleVeoProvider, TTVProvider } from '@loonylabs/ttv-middleware';

const service = new TTVService();
service.registerProvider(new GoogleVeoProvider({
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  location: 'us-central1',
}));

const result = await service.generate({
  prompt: 'A cinematic aerial shot of a mountain range at sunrise',
  model: 'veo-3.0-generate-001',
  duration: 8,
  aspectRatio: '16:9',
  resolution: '1080p',
  generateAudio: true,  // Veo 3+ generates audio natively
});

console.log('Video URL:', result.videos[0].url);

With Progress Callback

const result = await service.generate({
  prompt: 'A futuristic city with flying cars',
  model: 'sora-2',
  duration: 12,
  onProgress: (progress) => {
    console.log(`Status: ${progress.status}`, progress.message || '');
    // Status: queued
    // Status: in_progress
    // Status: completed
  },
});

Download to Buffer

import * as fs from 'fs';

const result = await service.generate({
  prompt: 'Ocean waves crashing on a rocky shore',
  model: 'sora-2',
  downloadToBuffer: true,  // Download video into memory
});

// Save to disk
fs.writeFileSync('output.mp4', result.videos[0].buffer!);

Switching Providers

// Use OpenAI Sora
const soraResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'sora-2',
}, TTVProvider.OPENAI_SORA);

// Use Google Veo
const veoResult = await service.generate({
  prompt: 'A mountain landscape timelapse',
  model: 'veo-3.0-generate-001',
}, TTVProvider.GOOGLE_VEO);

Prerequisites

Required Dependencies

• Node.js 18+
• TypeScript 5.3+

For OpenAI Sora provider:

npm install openai

For Google Veo provider:

npm install google-auth-library

Configuration

Environment Setup

Create a .env file in your project root:

# Default provider
TTV_DEFAULT_PROVIDER=openai-sora

# Logging level (debug, info, warn, error, silent)
TTV_LOG_LEVEL=info

# OpenAI Sora
OPENAI_API_KEY=sk-...

# Google Veo (Vertex AI)
GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=./service-account.json
GOOGLE_CLOUD_REGION=us-central1

Providers & Models

OpenAI Sora

Model	ID	Duration	Resolutions	Audio	Image-to-Video
Sora 2	sora-2	4, 8, 12s	720p, 1080p	Yes	Yes
Sora 2 Pro	sora-2-pro	10, 15, 25s	720p, 1080p	Yes	Yes

Pricing: ~$0.10-$0.50/sec depending on model and resolution.

Google Veo

Model	ID	Duration	Resolutions	Audio	Video Extension
Veo 2	veo-2.0-generate-001	5-8s	720p	No	Yes
Veo 3	veo-3.0-generate-001	4-8s	720p, 1080p	Yes	Yes
Veo 3 Fast	veo-3.0-fast-generate-001	4-8s	720p, 1080p	Yes	Yes
Veo 3.1	veo-3.1-generate-001	4-8s	720p, 1080p, 4K	Yes	Yes
Veo 3.1 Fast	veo-3.1-fast-generate-001	4-8s	720p, 1080p, 4K	Yes	Yes

Pricing: ~$0.15-$0.75/sec depending on model.

Image-to-Video

Animate a still image into a video. Both providers support this:

import * as fs from 'fs';

// Load a reference image
const imageBase64 = fs.readFileSync('character.png').toString('base64');

// OpenAI Sora: image becomes first frame
const result = await service.generate({
  prompt: 'She turns around and smiles, then slowly walks out of frame',
  model: 'sora-2',
  referenceImage: {
    base64: imageBase64,
    mimeType: 'image/png',
  },
  duration: 8,
});

// Google Veo: first frame + optional last frame for interpolation
const veoResult = await service.generate({
  prompt: 'Camera slowly zooms out revealing the landscape',
  model: 'veo-3.0-generate-001',
  referenceImage: {
    base64: firstFrameBase64,
    mimeType: 'image/png',
  },
  lastFrameImage: {         // Veo-specific: interpolate between two keyframes
    base64: lastFrameBase64,
    mimeType: 'image/png',
  },
}, TTVProvider.GOOGLE_VEO);

Video Extension

Extend an existing video (Google Veo only):

const extended = await service.extend({
  prompt: 'The camera continues to pan revealing a hidden waterfall',
  videoBuffer: existingVideoBuffer,
  videoMimeType: 'video/mp4',
  duration: 7,  // Extend by 7 seconds
  downloadToBuffer: true,
}, TTVProvider.GOOGLE_VEO);

fs.writeFileSync('extended.mp4', extended.videos[0].buffer!);

API Reference

TTVService

class TTVService {
  registerProvider(provider: BaseTTVProvider): void;
  generate(request: TTVRequest, provider?: TTVProvider): Promise<TTVResponse>;
  extend(request: TTVExtendRequest, provider?: TTVProvider): Promise<TTVResponse>;
  getProvider(name: TTVProvider): BaseTTVProvider | undefined;
  getAvailableProviders(): TTVProvider[];
  listAllModels(): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
  findProvidersWithCapability(capability: keyof TTVCapabilities): Array<{ provider: TTVProvider; models: ModelInfo[] }>;
}

TTVRequest

interface TTVRequest {
  prompt: string;
  model?: string;              // 'sora-2', 'veo-3.0-generate-001', etc.
  duration?: number;           // Desired duration in seconds
  aspectRatio?: string;        // '16:9', '9:16'
  resolution?: '720p' | '1080p' | '4k';
  n?: number;                  // Number of videos (default: 1)

  // Image-to-video
  referenceImage?: TTVReferenceImage;
  lastFrameImage?: TTVReferenceImage;  // Google Veo only

  // Audio & content
  generateAudio?: boolean;
  negativePrompt?: string;

  // Output control
  downloadToBuffer?: boolean;  // Download video to Buffer (default: false)
  onProgress?: TTVProgressCallback;

  // Retry & debug
  retry?: boolean | RetryOptions;
  dry?: boolean;

  providerOptions?: Record<string, unknown>;
}

TTVResponse

interface TTVResponse {
  videos: TTVVideo[];
  metadata: {
    provider: string;
    model: string;
    region?: string;
    duration: number;          // Total request time (ms)
    generationTime?: number;   // Polling time only (ms)
  };
  usage: {
    videosGenerated: number;
    totalDurationSeconds: number;
    modelId: string;
  };
  billing?: {
    cost: number;
    currency: string;
    source: 'provider' | 'estimated';
  };
}

TTVVideo

interface TTVVideo {
  url?: string;           // Video URL (Sora) or undefined (Veo returns buffer directly)
  buffer?: Buffer;        // Video data (if downloadToBuffer)
  contentType: string;    // 'video/mp4'
  duration?: number;      // Duration in seconds
}

Advanced Features

Polling Configuration

Video generation is asynchronous. The middleware polls for completion automatically. You can configure the polling behavior:

import { GoogleVeoProvider } from '@loonylabs/ttv-middleware';

const provider = new GoogleVeoProvider({
  projectId: 'my-project',
  polling: {
    intervalMs: 10000,       // Start polling every 10s (default)
    maxIntervalMs: 30000,    // Cap at 30s between polls
    backoffMultiplier: 1.5,  // Increase interval by 1.5x each time
    timeoutMs: 600000,       // Give up after 10 minutes (default)
  },
});

Option	Default	Description
intervalMs	10000	Initial polling interval (ms)
maxIntervalMs	30000	Maximum polling interval (ms)
backoffMultiplier	1.5	Multiplier per poll attempt
timeoutMs	600000	Maximum wait time (10 minutes)

Retry Configuration

Automatic retry with exponential backoff and jitter for transient errors (429, 408, 5xx, network timeouts):

// Default: 3 retries, exponential backoff (1s -> 2s -> 4s), jitter enabled
const result = await service.generate({
  prompt: 'A sunset over mountains',
  // retry: true (default)
});

// Custom retry configuration
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: {
    maxRetries: 5,
    delayMs: 1000,
    backoffMultiplier: 2.0,
    maxDelayMs: 30000,
    jitter: true,
  },
});

// Disable retry
const result = await service.generate({
  prompt: 'A sunset over mountains',
  retry: false,
});

Retryable errors: 429, 408, 500, 502, 503, 504, timeouts, ECONNRESET, ECONNREFUSED, socket hang up Not retried: 400, 401, 403, and other client errors

Logging Configuration

Control logging via environment variable or API:

import { setLogLevel } from '@loonylabs/ttv-middleware';

// Set log level programmatically
setLogLevel('warn');  // Only show warnings and errors

// Or via environment variable
// TTV_LOG_LEVEL=error

Available levels: debug, info, warn, error, silent

Debug Logging (Markdown Files)

Log all TTV requests and responses to markdown files for debugging:

import { TTVDebugger } from '@loonylabs/ttv-middleware';

// Enable via environment variable
// DEBUG_TTV_REQUESTS=true

// Or programmatically
TTVDebugger.setEnabled(true);
TTVDebugger.setLogsDir('./logs/ttv/requests');

TTVDebugger.configure({
  enabled: true,
  logsDir: './logs/ttv/requests',
  consoleLog: true,
});

Error Handling

Typed error classes for precise error handling:

import {
  TTVError,
  InvalidConfigError,
  QuotaExceededError,
  ProviderUnavailableError,
  GenerationFailedError,
  NetworkError,
  CapabilityNotSupportedError,
  PollingTimeoutError,
  ContentModeratedError,
} from '@loonylabs/ttv-middleware';

try {
  const result = await service.generate({ prompt: 'test', duration: 8 });
} catch (error) {
  if (error instanceof PollingTimeoutError) {
    console.log('Video generation timed out - try again or increase timeout');
  } else if (error instanceof ContentModeratedError) {
    console.log('Content was blocked by safety filters');
  } else if (error instanceof QuotaExceededError) {
    console.log('Rate limit hit, try again later');
  } else if (error instanceof CapabilityNotSupportedError) {
    console.log('Model does not support this feature');
  } else if (error instanceof TTVError) {
    console.log(`TTV Error [${error.code}]: ${error.message}`);
  }
}

Dry Mode

Test your integration without making API calls or incurring costs:

const result = await service.generate({
  prompt: 'A test video',
  duration: 8,
  dry: true,  // No API call, returns placeholder response
});

console.log(result.videos.length);       // 1
console.log(result.metadata.duration);   // 0 (no actual generation)

Provider-Specific Options

Use providerOptions as an escape hatch for provider-specific features:

// Google Veo: seed for deterministic output
const result = await service.generate({
  prompt: 'A sunset timelapse',
  model: 'veo-3.0-generate-001',
  providerOptions: {
    seed: 42,
    personGeneration: 'allow_adult',
    enhancePrompt: true,
    storageUri: 'gs://my-bucket/output/',  // Direct output to GCS
  },
}, TTVProvider.GOOGLE_VEO);

// OpenAI Sora: remix (reinterpret an existing video)
// Use the Sora API directly for remix via providerOptions

Testing

# Run all tests
npm test

# Unit tests only
npm run test:unit

# Unit tests with watch mode
npm run test:unit:watch

# Unit tests with coverage report
npm run test:unit:coverage

# Integration tests (requires TTV_INTEGRATION_TESTS=true)
npm run test:integration

# CI/CD mode
npm run test:ci

Manual Smoke Tests

The scripts/ directory contains manual smoke tests for real API calls:

# Text-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo.ts

# Image-to-video with Google Veo (veo-3.0-fast)
npx ts-node scripts/manual-test-veo-i2v.ts

Both scripts use veo-3.0-fast-generate-001 (cheapest model), generate short 4s/720p videos, and save output to the output/ directory. Requires Google Veo credentials in .env.

Integration Tests

Integration tests make real API calls and cost money. They are skipped by default.

# Enable and run integration tests
TTV_INTEGRATION_TESTS=true npm run test:integration

Prerequisites:

• OPENAI_API_KEY for Sora tests
• GOOGLE_CLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS for Veo tests

Contributing

We welcome contributions! Please ensure:

1. Tests: Add tests for new features

2. Linting: Run npm run lint before committing

3. Conventions: Follow the existing project structure

4. Fork the repository

5. Create your feature branch (git checkout -b feature/amazing-feature)

6. Commit your changes (git commit -m 'Add some amazing feature')

7. Push to the branch (git push origin feature/amazing-feature)

8. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Links

• Documentation
• Issues
• NPM Package

---

Made with care by the LoonyLabs Team