diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..6fd01e33a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,338 @@ +# Kleinkram - AI Agent Documentation + +> **Purpose**: This document helps AI agents understand the Kleinkram codebase structure, architecture, and key concepts to work effectively with the system. + +## System Overview + +Kleinkram is a **structured bag and mcap file storage solution** for ROS1 and ROS2, developed by the Robotic Systems Lab (RSL) at ETH Zurich. It provides a complete platform for uploading, storing, processing, and managing robotics dataset files. + +## Architecture Diagram + +``` +┌─────────────┐ +│ Users │ +└──────┬──────┘ + │ + ├──────────────────────────────────────────┐ + │ │ +┌──────▼──────┐ ┌───────▼────────┐ +│ Frontend │ │ CLI │ +│ (Vue 3) │ │ (Python) │ +└──────┬──────┘ └───────┬────────┘ + │ │ + └──────────────────┬──────────────────────┘ + │ + ┌──────▼──────┐ + │ Backend │ + │ (NestJS) │ + └──────┬──────┘ + │ + ┏━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━┓ + ┃ ┃ +┌──────▼──────┐ ┌────────▼────────┐ +│ PostgreSQL │ │ Queue Consumer │ +│ Database │ │ (NestJS) │ +└─────────────┘ └────────┬────────┘ + │ + ┏━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━┓ + ┃ ┃ + ┌──────▼──────┐ ┌───────▼────────┐ + │ MinIO │ │ Docker │ + │ (S3 Store) │ │ (Actions) │ + └──────┬──────┘ └───────┬────────┘ + │ │ + └──────────┬────────────────────────┘ + │ + ┌──────▼──────┐ + │ Google Drive│ + │ (Artifacts) │ + └─────────────┘ +``` + +## Core Components + +### 1. Backend (`/backend/`) +- **Technology**: NestJS, TypeORM, PostgreSQL +- **Purpose**: Main REST API server +- **Key Responsibilities**: + - File metadata management + - Project/Mission organization + - Authentication & authorization (JWT, API keys, OAuth) + - Action scheduling + - Queue management + - Tag and topic management + +📖 [Detailed Backend Documentation](./backend/AGENTS.md) + +### 2. Queue Consumer (`/queueConsumer/`) +- **Technology**: NestJS, Bull, Redis, Docker +- **Purpose**: Background job processor +- **Key Responsibilities**: + - File processing (bag→mcap conversion, topic extraction) + - Action execution (Docker container orchestration) + - File cleanup + - Access group expiry management + +📖 [Detailed Queue Consumer Documentation](./queueConsumer/AGENTS.md) + +### 3. Frontend (`/frontend/`) +- **Technology**: Vue 3, Quasar, TypeScript +- **Purpose**: Web user interface +- **Key Responsibilities**: + - Project/Mission/File browsing + - File upload interface + - Action submission and monitoring + - Access control management + - Data visualization + +📖 [Detailed Frontend Documentation](./frontend/AGENTS.md) + +### 4. CLI (`/cli/`) +- **Technology**: Python +- **Purpose**: Command-line interface +- **Key Responsibilities**: + - File upload/download + - Project/Mission listing + - Batch operations + - CI/CD integration + +📖 [Detailed CLI Documentation](./cli/AGENTS.md) + +### 5. Common (`/common/`) +- **Technology**: TypeScript +- **Purpose**: Shared code library +- **Key Responsibilities**: + - Entity definitions (TypeORM) + - API DTOs and validation + - Shared utilities and helpers + - Environment configuration + +📖 [Detailed Common Documentation](./common/AGENTS.md) + +## Key Concepts + +### Data Model Hierarchy + +``` +Project (e.g., "ANYmal Experiments") + └── Mission (e.g., "2024-01-15_outdoor_navigation") + └── File (e.g., "run_001.bag", "run_001.mcap") + └── Topic (e.g., "/camera/image_raw", "/imu/data") +``` + +### Core Entities + +1. **Project**: Top-level organizational unit, typically representing a research project or robot +2. **Mission**: A collection of related files, typically from a single recording session or experiment +3. **File**: A bag or mcap file containing ROS data +4. **Topic**: ROS topics extracted from mcap files, with metadata (type, message count, etc.) +5. **Action**: A Docker container-based processing job that operates on mission data +6. **Tag**: Custom key-value metadata attached to files +7. **User**: System users with access permissions +8. **Access Group**: Groups of users with specific permissions on projects/missions +9. **Worker**: A machine capable of executing actions +10. **Queue**: Job queue entries for file processing or action execution + +### File Types + +- **BAG** (`.bag`): ROS1 bag files +- **MCAP** (`.mcap`): MCAP files (ROS2 compatible, more efficient) + +**Note**: The system automatically converts uploaded bag files to mcap format for efficient processing and storage. + +## Data Flows + +### 1. File Upload Flow + +``` +User → Upload (CLI/Frontend) + → Backend creates QueueEntity + → File stored in MinIO + → Queue Consumer processes file + → Convert bag → mcap (if needed) + → Extract topics and metadata + → Store metadata in PostgreSQL + → File ready for use +``` + +Key files: +- Backend: `backend/src/endpoints/file/file.controller.ts:temporaryAccess` +- Queue Consumer: `queueConsumer/src/files/file-queue-processor.provider.ts` +- Conversion: `queueConsumer/src/files/helper/converter.ts` + +### 2. Action Execution Flow + +``` +User → Submit Action (Frontend) + → Backend creates Action entity + → Action queued in Redis (Bull) + → Worker picks up action + → Create Docker container with resources + → Container accesses files via API + → Container produces artifacts + → Artifacts uploaded to Google Drive + → Action marked complete +``` + +Key files: +- Backend: `backend/src/endpoints/action/action.controller.ts` +- Queue Consumer: `queueConsumer/src/actions/action-queue-processor.provider.ts` +- Action Manager: `queueConsumer/src/actions/services/action-manager.service.ts` +- Docker Daemon: `queueConsumer/src/actions/services/docker-daemon.service.ts` + +### 3. File Download Flow + +``` +User → Request Download (CLI/Frontend) + → Backend validates permissions + → Generate pre-signed MinIO URL + → User downloads directly from MinIO +``` + +Key files: +- Backend: `backend/src/endpoints/file/file.controller.ts:download` +- File Service: `backend/src/services/file.service.ts:generateDownload` + +## Technology Stack + +### Backend/Queue Consumer +- **Framework**: NestJS +- **Database**: PostgreSQL with TypeORM +- **Queue**: Redis + Bull +- **Storage**: MinIO (S3-compatible) +- **Container Runtime**: Docker/Dockerode +- **Observability**: OpenTelemetry, Prometheus, Loki + +### Frontend +- **Framework**: Vue 3 +- **UI Library**: Quasar +- **HTTP Client**: Axios +- **State Management**: Tanstack Query (Vue Query) +- **Charts**: ECharts + +### CLI +- **Language**: Python 3.8+ +- **HTTP Client**: requests (likely) +- **Compression**: zstd support for rosbags + +### Infrastructure +- **Deployment**: Docker Compose +- **Authentication**: JWT, OAuth (GitHub, Google) +- **File Storage**: MinIO, Google Drive +- **Message Queue**: Redis +- **Monitoring**: Prometheus, Grafana, Loki + +## Development Setup + +```bash +# Clone repository +git clone git@github.com:leggedrobotics/kleinkram.git +cd kleinkram + +# Start all services with Docker Compose +docker compose up --build + +# Services will be available at: +# - Frontend: http://localhost:8003 +# - Backend API: http://localhost:3000 +# - MinIO Console: http://localhost:9001 +# - Documentation: http://localhost:4000 +``` + +For development of individual components, see their respective AGENTS.md files. + +## Project Structure + +``` +kleinkram/ +├── backend/ # NestJS backend API +├── queueConsumer/ # Background job processor +├── frontend/ # Vue 3 web interface +├── cli/ # Python CLI tool +├── common/ # Shared TypeScript code +├── docs/ # Documentation (VitePress) +├── examples/ # Example files and scripts +├── observability/ # Monitoring configuration +├── docker-compose.yml # Main docker compose file +└── AGENTS.md # This file +``` + +## Common Development Tasks + +### Working with Files +- **Upload processing**: See `queueConsumer/src/files/` +- **File metadata**: See `backend/src/endpoints/file/` +- **Topic extraction**: See `queueConsumer/src/files/helper/converter.ts` + +### Working with Actions +- **Action scheduling**: See `backend/src/endpoints/action/` +- **Action execution**: See `queueConsumer/src/actions/` +- **Container management**: See `queueConsumer/src/actions/services/docker-daemon.service.ts` + +### Working with Authentication +- **JWT/OAuth**: See `backend/src/endpoints/auth/` +- **Middleware**: See `backend/src/routing/middlewares/` +- **Guards**: See `backend/src/endpoints/auth/roles.decorator.ts` + +### Working with Database +- **Entities**: See `common/entities/` +- **Migrations**: TypeORM synchronize is enabled in DEV mode +- **Seeds**: See `common/seeds/` + +## Important Patterns + +### 1. Queue Pattern +All asynchronous operations use Bull queues: +- File processing +- Action execution +- File cleanup + +### 2. Entity Relations +TypeORM entities use decorators for relations. Always load relations explicitly when needed: +```typescript +const file = await this.fileRepository.findOne({ + where: { uuid }, + relations: ['mission', 'mission.project', 'creator'] +}); +``` + +### 3. Authentication Flow +1. Middleware resolves user from JWT or API key +2. Guards check permissions using decorators +3. User/Apikey attached to request via `@AddUser()` decorator + +### 4. File Storage Strategy +- Original files in MinIO +- Metadata in PostgreSQL +- bag files automatically converted to mcap +- Topics extracted from mcap and stored in DB + +## Testing + +- **Backend**: Jest tests in `backend/tests/` +- **CLI**: Pytest tests in `cli/tests/` +- **E2E**: Integration tests require backend running locally + +## External Documentation + +- **Main docs**: https://docs.datasets.leggedrobotics.com/ +- **User guide**: See `docs/usage/` +- **Development guide**: See `docs/development/` + +## Contribution Guidelines + +1. Follow existing code style (ESLint/Prettier configured) +2. Add tests for new features +3. Update relevant AGENTS.md files for architectural changes +4. Use conventional commits for version management +5. Run `yarn tsc:check` and `yarn eslint` before committing + +## Version Management + +- Monorepo with synchronized versions across packages +- Use `yarn bump` for version increments +- Versions aligned in root, backend, frontend, queueConsumer, cli, common + +## Getting Help + +For questions about specific components, refer to their individual AGENTS.md files in each directory. diff --git a/backend/AGENTS.md b/backend/AGENTS.md new file mode 100644 index 000000000..c4c7778f4 --- /dev/null +++ b/backend/AGENTS.md @@ -0,0 +1,575 @@ +# Backend - AI Agent Documentation + +> **Purpose**: This document provides a detailed overview of the Kleinkram backend architecture for AI agents. + +## Overview + +The backend is a **NestJS REST API** that serves as the main entry point for all client interactions. It handles: +- File metadata management +- Project and Mission organization +- User authentication and authorization +- Action scheduling and management +- Tag and topic queries +- Access control + +**Technology Stack**: NestJS, TypeORM, PostgreSQL, Passport.js, Bull (Redis) + +## Project Structure + +``` +backend/ +├── src/ +│ ├── endpoints/ # API endpoint modules +│ │ ├── action/ # Action management endpoints +│ │ ├── auth/ # Authentication endpoints +│ │ ├── category/ # Category management +│ │ ├── file/ # File operations +│ │ ├── mission/ # Mission management +│ │ ├── project/ # Project management +│ │ ├── queue/ # Queue status endpoints +│ │ ├── tag/ # Tag management +│ │ ├── topic/ # Topic queries +│ │ ├── user/ # User management +│ │ └── worker/ # Worker management +│ ├── routing/ # Middleware and guards +│ │ └── middlewares/ # Request processing middleware +│ ├── services/ # Business logic services +│ ├── types/ # TypeScript type definitions +│ ├── validation/ # Request validation decorators +│ ├── app.module.ts # Root application module +│ ├── main.ts # Application entry point +│ ├── serialization.ts # Response serialization +│ ├── logger.ts # Winston logger configuration +│ └── tracing.ts # OpenTelemetry tracing +├── tests/ # Jest test files +├── tsconfig.json # TypeScript configuration +└── package.json # Dependencies + +Key File: backend/src/app.module.ts:29-101 +``` + +## Architecture + +### Module Structure + +The backend follows NestJS module architecture with clear separation of concerns: + +``` +AppModule + ├── ConfigModule (Global) + ├── TypeOrmModule (Database) + ├── PrometheusModule (Metrics) + ├── ScheduleModule (Cron Jobs) + ├── PassportModule (Auth) + └── Feature Modules: + ├── FileModule + ├── ProjectModule + ├── MissionModule + ├── ActionModule + ├── AuthModule + ├── UserModule + ├── QueueModule + ├── TagModule + ├── TopicModule + ├── WorkerModule + └── CategoryModule +``` + +### Middleware Pipeline + +Every request goes through this middleware pipeline (in order): + +1. **APIKeyResolverMiddleware** (`routing/middlewares/api-key-resolver-middleware.service.ts`) + - Resolves user from JWT token or API key + - Attaches user/apikey to request object + - Handles both cookie-based and header-based auth + +2. **AuditLoggerMiddleware** (`routing/middlewares/audit-logger-middleware.service.ts`) + - Logs all API requests for auditing + - Captures user, endpoint, method, and response time + +3. **VersionCheckerMiddlewareService** (`routing/middlewares/version-checker-middleware.service.ts`) + - Validates client version compatibility + - Returns warning if client is outdated + +Key File: backend/src/app.module.ts:91-99 + +## API Endpoints + +### File Endpoints (`/file` or `/files`) + +**Controller**: `backend/src/endpoints/file/file.controller.ts` + +Key endpoints: +- `GET /file` - Query files with complex filtering +- `GET /file/filtered` - Legacy filtered file query +- `GET /file/download?uuid=` - Generate download URL +- `GET /file/one?uuid=` - Get single file details +- `GET /file/ofMission?uuid=` - Get files in a mission +- `PUT /file/:uuid` - Update file metadata +- `POST /file/moveFiles` - Move files between missions +- `DELETE /file/:uuid` - Delete a file +- `POST /file/temporaryAccess` - Get pre-signed upload URLs +- `POST /file/cancelUpload` - Cancel an in-progress upload +- `POST /file/deleteMultiple` - Batch delete files +- `GET /file/exists?uuid=` - Check if file exists +- `GET /file/storage` - Get storage statistics +- `GET /file/isUploading` - Check if user is uploading + +**Service**: `backend/src/services/file.service.ts` + +### Project Endpoints (`/project` or `/projects`) + +**Controller**: `backend/src/endpoints/project/project.controller.ts` + +Key endpoints: +- `GET /project` - List all accessible projects +- `GET /project/one?uuid=` - Get project details +- `POST /project` - Create new project +- `PUT /project/:uuid` - Update project +- `DELETE /project/:uuid` - Delete project + +**Service**: `backend/src/services/project.service.ts` + +### Mission Endpoints (`/mission` or `/missions`) + +**Controller**: `backend/src/endpoints/mission/mission.controller.ts` + +Key endpoints: +- `GET /mission` - List missions (with filtering) +- `GET /mission/one?uuid=` - Get mission details +- `POST /mission` - Create new mission +- `PUT /mission/:uuid` - Update mission +- `DELETE /mission/:uuid` - Delete mission +- `GET /mission/ofProject?uuid=` - Get missions in a project + +**Service**: `backend/src/services/mission.service.ts` + +### Action Endpoints (`/action`) + +**Controller**: `backend/src/endpoints/action/action.controller.ts` + +Key endpoints: +- `GET /action` - List actions +- `GET /action/one?uuid=` - Get action details +- `POST /action` - Submit new action +- `POST /action/cancel` - Cancel a running action +- `GET /action/ofMission?uuid=` - Get actions in a mission +- `GET /action/templates` - List available action templates + +**Service**: `backend/src/services/action.service.ts` + +### Auth Endpoints (`/auth`) + +**Controller**: `backend/src/endpoints/auth/auth.controller.ts` + +Key endpoints: +- `POST /auth/login` - Login with credentials +- `POST /auth/logout` - Logout +- `GET /auth/google` - OAuth login with Google +- `GET /auth/github` - OAuth login with GitHub +- `GET /auth/me` - Get current user info +- `POST /auth/apikey` - Create new API key +- `DELETE /auth/apikey/:uuid` - Delete API key + +### Tag Endpoints (`/tag`) + +**Controller**: `backend/src/endpoints/tag/tag.controller.ts` + +Key endpoints: +- `GET /tag` - List tags with filtering +- `GET /tag/types` - List tag types +- `POST /tag` - Create tag +- `PUT /tag/:uuid` - Update tag +- `DELETE /tag/:uuid` - Delete tag + +**Service**: `backend/src/services/tag.service.ts` + +### Topic Endpoints (`/topic`) + +**Controller**: `backend/src/endpoints/topic/topic.controller.ts` + +Key endpoints: +- `GET /topic` - Query topics +- `GET /topic/ofFile?uuid=` - Get topics in a file + +**Service**: `backend/src/services/topic.service.ts` + +### User Endpoints (`/user`) + +**Controller**: `backend/src/endpoints/user/user.controller.ts` + +Key endpoints: +- `GET /user` - List users (admin only) +- `GET /user/me` - Get current user +- `PUT /user/:uuid` - Update user + +**Service**: `backend/src/services/user.service.ts` + +### Worker Endpoints (`/worker`) + +**Controller**: `backend/src/endpoints/worker/worker.controller.ts` + +Key endpoints: +- `GET /worker` - List workers +- `GET /worker/one?uuid=` - Get worker details + +**Service**: `backend/src/services/worker.service.ts` + +### Queue Endpoints (`/queue`) + +**Controller**: `backend/src/endpoints/queue/queue.controller.ts` + +Key endpoints: +- `GET /queue` - List queue entries +- `GET /queue/ofMission?uuid=` - Get queue entries for a mission + +**Service**: `backend/src/services/queue.service.ts` + +## Authentication & Authorization + +### Authentication Methods + +The backend supports three authentication methods: + +1. **JWT Tokens** (Cookie or Header) + - Set via `accessToken` cookie or `Authorization: Bearer ` header + - Used by frontend + - Expires after configured time + +2. **API Keys** (Header) + - Set via `x-api-key` header + - Used by CLI and programmatic access + - Can be scoped to specific missions + - Types: USER (full access), CONTAINER (limited, for actions) + +3. **OAuth** (GitHub, Google) + - Handled by Passport.js strategies + - Creates/links user accounts automatically + - Returns JWT token after successful OAuth + +### Authorization Guards + +Guards are implemented as decorators in `backend/src/endpoints/auth/roles.decorator.ts`: + +- `@LoggedIn()` - User must be authenticated +- `@UserOnly()` - Must be a real user (not API key) +- `@AdminOnly()` - Must be an admin user +- `@CanReadFile()` - Can read a specific file +- `@CanWriteFile()` - Can write to a specific file +- `@CanDeleteFile()` - Can delete a specific file +- `@CanReadMission()` - Can read a mission +- `@CanWriteMission()` - Can write to a mission +- `@CanDeleteMission()` - Can delete a mission +- `@CanReadProject()` - Can read a project +- `@CanWriteProject()` - Can write to a project +- `@CanCreateInMissionByBody()` - Can create resources in a mission (from body) +- `@CanMoveFiles()` - Can move files between missions + +### Access Control + +Access control is managed through: + +1. **Project Access** (`common/entities/auth/project-access.entity`) + - Links users/groups to projects + - Defines read/write/admin permissions + +2. **Mission Access** (`common/entities/auth/mission-access.entity`) + - Links users/groups to missions + - Inherits from project access + +3. **Access Groups** (`common/entities/auth/accessgroup.entity`) + - Groups of users + - Can have expiry dates + - Managed in `backend/src/endpoints/auth/` + +**Services**: +- `backend/src/services/access.service.ts` - Access control logic +- `backend/src/services/file-guard.service.ts` - File-level permissions +- `backend/src/services/project-guard.service.ts` - Project-level permissions + +## Database Schema + +The backend uses TypeORM with PostgreSQL. Key entities are defined in `common/entities/`: + +### Core Entities + +1. **User** (`common/entities/user/user.entity.ts`) + - id, uuid, email, username, isAdmin + - Relations: submittedActions, apikeys, projectAccess, missionAccess + +2. **Project** (`common/entities/project/project.entity.ts`) + - id, uuid, name, description, autoConvert + - Relations: missions, access + +3. **Mission** (`common/entities/mission/mission.entity.ts`) + - id, uuid, name, description, date + - Relations: project, files, actions, queues, access + +4. **File** (`common/entities/file/file.entity.ts`) + - id, uuid, filename, size, hash, type, state, origin + - Relations: mission, creator, topics, tags, relatedFile + +5. **Topic** (`common/entities/topic/topic.entity.ts`) + - id, uuid, name, type, messageCount, frequency + - Relations: file + +6. **Action** (`common/entities/action/action.entity.ts`) + - id, uuid, state, exitCode, logs, artifacts + - Relations: mission, template, createdBy, worker + +7. **ActionTemplate** (`common/entities/action/action-template.entity.ts`) + - id, uuid, name, version, imageName, command + - Runtime requirements: cpuCores, cpuMemory, gpuMemory, maxRuntime + +8. **QueueEntity** (`common/entities/queue/queue.entity.ts`) + - id, uuid, state, location, identifier, displayName + - Relations: mission, creator + +9. **Tag** (`common/entities/tag/tag.entity.ts`) + - id, uuid, value + - Relations: file, type + +10. **Worker** (`common/entities/worker/worker.entity.ts`) + - id, uuid, hostname, identifier, reachable, storage + - Relations: actions + +### Database Operations + +- **Synchronize**: Enabled in DEV mode (auto-creates/updates schema) +- **Migrations**: Not currently used (relies on synchronize) +- **Transactions**: Used for atomic operations +- **Query Builder**: Used for complex queries with joins + +## Services Layer + +Services contain the business logic and are injected into controllers. + +### Key Services + +1. **FileService** (`backend/src/services/file.service.ts`) + - File CRUD operations + - Pre-signed URL generation (MinIO) + - File filtering and search + - Upload cancellation + - Storage statistics + +2. **ProjectService** (`backend/src/services/project.service.ts`) + - Project CRUD operations + - Access control integration + +3. **MissionService** (`backend/src/services/mission.service.ts`) + - Mission CRUD operations + - Filtering and pagination + +4. **ActionService** (`backend/src/services/action.service.ts`) + - Action submission + - Worker assignment + - Action cancellation + - Template management + +5. **AuthService** (`backend/src/services/auth.service.ts`) + - JWT token generation/validation + - API key management + - OAuth integration + +6. **QueueService** (`backend/src/services/queue.service.ts`) + - Queue entry creation + - Job submission to Redis (Bull) + - Queue status queries + +7. **AccessService** (`backend/src/services/access.service.ts`) + - Permission checking + - Access group management + +8. **FileGuardService** (`backend/src/services/file-guard.service.ts`) + - File-level permission checking + - Used by guards + +9. **ProjectGuardService** (`backend/src/services/project-guard.service.ts`) + - Project-level permission checking + - Used by guards + +## Validation & Serialization + +### Request Validation + +Custom decorators in `backend/src/validation/`: + +**Query Parameters**: +- `@QueryUUID()` - UUID validation +- `@QueryString()` - String parameter +- `@QueryBoolean()` - Boolean parameter +- `@QueryOptionalString()` - Optional string +- `@QueryOptionalDate()` - Optional date +- `@QuerySkip()` - Pagination offset +- `@QueryTake()` - Pagination limit +- `@QuerySortBy()` - Sort field +- `@QuerySortDirection()` - Sort direction + +**Body Parameters**: +- `@BodyUUID()` - UUID in body +- `@BodyUUIDArray()` - Array of UUIDs + +**Path Parameters**: +- `@ParameterUuid()` - UUID in path + +### Response Serialization + +Responses are serialized using `class-transformer` in `backend/src/serialization.ts`: +- Removes sensitive fields +- Transforms entities to DTOs +- Handles nested relations + +Key File: backend/src/serialization.ts:1-288 + +### DTOs + +Data Transfer Objects are defined in `common/api/types/`: +- Request validation +- Response typing +- Shared between backend and frontend + +## Queue Integration + +The backend enqueues jobs to Redis using Bull: + +```typescript +// Example: Enqueue file processing job +await this.fileQueue.add('processMinioFile', { + queueUuid: queue.uuid, + md5: fileMd5Hash, +}); +``` + +Queue types: +- `file-queue` - File processing jobs +- `action-queue-{hostname}` - Action execution jobs (per worker) +- `file-cleanup` - File deletion jobs +- `move` - File move operations + +The Queue Consumer picks up these jobs and processes them. + +## MinIO Integration + +MinIO is used for object storage (S3-compatible): + +**Configuration**: +- `MINIO_ENDPOINT` - MinIO server endpoint +- `MINIO_ACCESS_KEY` - Access key +- `MINIO_SECRET_KEY` - Secret key +- `MINIO_BAG_BUCKET_NAME` - Bucket for bag files +- `MINIO_MCAP_BUCKET_NAME` - Bucket for mcap files + +**Operations**: +- Pre-signed URLs for upload/download +- Object tagging (missionUuid, projectUuid, filename) +- Bucket listing +- Object deletion + +Helper functions in `common/minio-helper.ts` + +## Observability + +### Logging + +Winston logger with Loki integration: +- `backend/src/logger.ts` +- Structured logging with labels +- Log levels: debug, info, warn, error +- Logs sent to Loki (if configured) + +### Tracing + +OpenTelemetry integration: +- `backend/src/tracing.ts` +- Automatic instrumentation for HTTP, DB, etc. +- Custom spans with `@tracing()` decorator +- Traces exported to OTLP endpoint + +### Metrics + +Prometheus metrics: +- Endpoint metrics (requests, duration, errors) +- Custom metrics via `@willsoto/nestjs-prometheus` +- Available at `/metrics` endpoint + +## Scheduled Jobs + +NestJS Schedule module is used for cron jobs: + +**DBDumper** (`backend/src/services/dbdumper.service.ts`): +- Scheduled database backups +- Cron expression configurable + +## Environment Configuration + +Configuration loaded from: +1. `.env` file (root) +2. Environment variables +3. `common/environment.ts` - Default values + +Key environment variables: +- `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_USERNAME`, etc. - Database config +- `MINIO_*` - MinIO configuration +- `JWT_SECRET` - JWT signing secret +- `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` - OAuth +- `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` - OAuth +- `ENDPOINT` - Backend URL +- `DEV` - Development mode flag + +## Error Handling + +- Global exception filter (NestJS built-in) +- Custom error classes in `common/` +- HTTP status codes used appropriately +- Detailed error messages in development +- Sanitized errors in production + +## Testing + +Tests are in `backend/tests/`: +- Unit tests for services +- Integration tests for endpoints +- E2E tests (require full stack) + +Run tests: +```bash +yarn test +``` + +## Common Development Tasks + +### Adding a New Endpoint + +1. Create module in `backend/src/endpoints//` +2. Create controller, service, module files +3. Define DTOs in `common/api/types/` +4. Add guards for authorization +5. Register module in `app.module.ts` +6. Add tests + +### Adding Database Field + +1. Update entity in `common/entities/` +2. Synchronize will auto-update in DEV +3. For production, create migration + +### Adding Permission Check + +1. Use existing guards or create new guard +2. Implement logic in AccessService or create specialized service +3. Apply guard decorator to endpoint + +### Debugging + +1. Use `logger.debug()` for detailed logs +2. Check OpenTelemetry traces +3. Use NestJS debug mode: `yarn start:debug` +4. Attach debugger to port 9229 + +## Related Documentation + +- [Root AGENTS.md](../AGENTS.md) - System overview +- [Common AGENTS.md](../common/AGENTS.md) - Shared entities and DTOs +- [Queue Consumer AGENTS.md](../queueConsumer/AGENTS.md) - Job processing diff --git a/cli/AGENTS.md b/cli/AGENTS.md new file mode 100644 index 000000000..28e9bc1cb --- /dev/null +++ b/cli/AGENTS.md @@ -0,0 +1,680 @@ +# CLI - AI Agent Documentation + +> **Purpose**: This document provides a detailed overview of the Kleinkram CLI architecture for AI agents. + +## Overview + +The Kleinkram CLI is a **Python command-line interface** for interacting with the Kleinkram dataset storage system. It provides fast, scriptable access to upload, download, and manage files without using the web interface. + +**Key Responsibilities**: +- File upload/download with progress tracking +- Project/Mission/File management (CRUD operations) +- Batch operations for CI/CD pipelines +- File verification (hash checking) +- Transparent compression support (zstd) + +**Technology Stack**: Python 3.8+, Typer, Rich, Boto3 (S3), httpx, zstandard + +## Project Structure + +``` +cli/ +├── kleinkram/ +│ ├── api/ # API client and routes +│ │ ├── client.py # Authenticated HTTP client +│ │ ├── routes.py # API endpoint wrappers +│ │ ├── query.py # Query builders +│ │ ├── file_transfer.py # Upload/download logic +│ │ ├── pagination.py # Pagination utilities +│ │ └── deser.py # Response deserialization +│ ├── cli/ # CLI commands +│ │ ├── app.py # Main CLI application +│ │ ├── _download.py # Download commands +│ │ ├── _upload.py # Upload commands +│ │ ├── _list.py # List commands +│ │ ├── _verify.py # Verify commands +│ │ ├── _file.py # File commands +│ │ ├── _mission.py # Mission commands +│ │ ├── _project.py # Project commands +│ │ ├── _endpoint.py # Endpoint commands +│ │ └── error_handling.py # Error handling +│ ├── auth.py # OAuth authentication +│ ├── compression.py # Compression/decompression +│ ├── config.py # Configuration management +│ ├── core.py # Core operations +│ ├── errors.py # Custom exceptions +│ ├── models.py # Data models +│ ├── printing.py # Output formatting +│ ├── utils.py # Utility functions +│ ├── main.py # Entry point +│ └── _version.py # Version info +├── tests/ # Pytest tests +├── pyproject.toml # Project configuration +└── README.md + +Key File: cli/kleinkram/cli/app.py:1-238 +Key File: cli/kleinkram/core.py:1-315 +Key File: cli/kleinkram/compression.py:1-234 +``` + +## Architecture + +### CLI Command Structure + +Built with **Typer** for type-safe, composable CLI: + +``` +klein [OPTIONS] COMMAND [ARGS] + ├── Authentication Commands: + │ ├── login # OAuth login + │ ├── logout # Logout + │ ├── endpoint # Manage endpoints + │ └── claim # Claim admin (hidden) + ├── Core Commands: + │ ├── upload # Upload files + │ ├── download # Download files + │ ├── verify # Verify uploaded files + │ └── list # List resources + └── CRUD Commands: + ├── project # Project operations + ├── mission # Mission operations + └── file # File operations +``` + +**Global Options**: +- `--verbose`: Enable verbose output (default: True) +- `--debug`: Enable debug mode +- `--log-level`: Set log level +- `--max-lines`: Max table rows (default: 10,000) +- `--version`: Show version + +Key File: cli/kleinkram/cli/app.py:96-113 + +### Configuration System + +**Location**: `~/.config/kleinkram/config.json` (Linux/macOS) or `%APPDATA%\kleinkram\config.json` (Windows) + +```json +{ + "selected_endpoint": "https://api.datasets.leggedrobotics.com", + "endpoint_credentials": { + "https://api.datasets.leggedrobotics.com": { + "jwt_token": "...", + "refresh_token": "...", + "expires_at": 1234567890 + } + } +} +``` + +**Functions**: +- `get_config()`: Load configuration +- `save_config()`: Save configuration +- `check_config_compatibility()`: Version check + +Key File: cli/kleinkram/config.py + +### API Client + +**Class**: `AuthenticatedClient` + +Wraps `httpx.Client` with: +- Automatic JWT token handling +- Token refresh on expiry +- Endpoint configuration +- Timeout handling + +**Usage**: +```python +client = AuthenticatedClient() +response = client.get('/projects') +response.raise_for_status() +data = response.json() +``` + +Key File: cli/kleinkram/api/client.py + +## Core Operations + +### 1. Authentication + +**Command**: `klein login` + +**Flow**: +``` +1. klein login --oauth-provider google + ↓ +2. Open browser with OAuth URL + ↓ +3. User authorizes via Google/GitHub + ↓ +4. Redirect to /auth/callback + ↓ +5. CLI polls backend for JWT token + ↓ +6. Save token to config +``` + +**Supported Providers**: +- `google` (default) +- `github` +- `fake-oauth` (dev only) + +**Headless Mode**: +```bash +klein login --headless --key +``` + +Key File: cli/kleinkram/auth.py + +### 2. File Upload + +**Command**: `klein upload` + +**Usage**: +```bash +# Upload to existing mission +klein upload /path/to/files/*.bag --mission "2024-01-15" --project "ANYmal" + +# Create mission if it doesn't exist +klein upload *.mcap --mission "new-mission" --project "MyProject" --create + +# With compression support +klein upload data.mcap.zst --mission "test" --project "MyProject" +``` + +**Upload Flow**: +``` +1. Detect if files are compressed (.zst/.zstd) + ↓ +2. Decompress to temp directory (if compressed) + ↓ +3. Verify mission exists (or create if --create) + ↓ +4. Request temporary S3 credentials + POST /files/temporaryAccess + ↓ +5. Upload to MinIO via boto3 (multipart for large files) + - Parallel uploads (2 workers) + - Progress bar per file + - MD5 hash calculation + - Retry on failure (3 attempts) + ↓ +6. Confirm upload + POST /queue/confirmUpload + ↓ +7. Clean up temp files +``` + +**Key Features**: +- **Compression Support**: Automatic decompression of zstd files +- **Multipart Upload**: Boto3 handles large files efficiently +- **Parallel Uploads**: 2 concurrent uploads by default +- **Progress Tracking**: Rich progress bars with tqdm +- **Hash Verification**: MD5 calculated during upload +- **Retry Logic**: Up to 3 retries on failure +- **Error Handling**: Graceful handling of existing files + +Key File: cli/kleinkram/core.py:83-145 +Key File: cli/kleinkram/api/file_transfer.py:191-248 + +#### Compression Support + +**Module**: `kleinkram.compression` + +**Supported**: zstd only (`.zst`, `.zstd` extensions) + +**Why zstd?** +- 5-10x faster decompression than gzip/bzip2 +- Excellent compression ratios +- Industry standard for ROS bags +- Tunable compression levels + +**Decompression Flow**: +```python +# Detect compression +detect_compression(Path("data.mcap.zst")) # → CompressionType.ZSTD + +# Decompress single file +decompress_file(Path("data.mcap.zst"), output_dir=temp_dir) + → Creates temp file: /tmp/kleinkram.../data.mcap + +# Decompress multiple files +mapping, temp_dir = decompress_files([Path("a.zst"), Path("b.mcap")]) + → {Path("a.zst"): Path("/tmp/.../a.mcap"), + Path("b.mcap"): Path("b.mcap")} # Uncompressed files mapped to themselves + +# Upload decompressed files +upload_files(client, mapping.values(), mission_id) + +# Cleanup +cleanup_temp_dir(temp_dir) +``` + +**Implementation**: +- Uses `zstandard` library for decompression +- Creates per-file temp directories +- Automatic cleanup on error or completion +- Preserves original filename without compression extension + +Key File: cli/kleinkram/compression.py:1-234 + +### 3. File Download + +**Command**: `klein download` + +**Usage**: +```bash +# Download all files in a mission +klein download --mission "2024-01-15" --project "ANYmal" --dest /path/to/dest + +# Download specific files +klein download --filename "run_001.mcap" --mission "test" --project "MyProject" + +# Download with nested directory structure +klein download --mission "..." --project "..." --dest . --nested +``` + +**Download Flow**: +``` +1. Query files matching criteria + GET /files + ↓ +2. Generate destination paths + ↓ +3. Request download URLs + GET /files/download?uuid={uuid}&expires=true + ↓ +4. Download via HTTP with resume support + - Range requests for partial downloads + - Progress bars + - Retry on failure (5 attempts with backoff) + ↓ +5. Verify hash (MD5) +``` + +**Key Features**: +- **Resume Support**: Range requests for partial downloads +- **Parallel Downloads**: 2 concurrent downloads by default +- **Progress Tracking**: Per-file progress bars +- **Hash Verification**: MD5 check after download +- **Retry Logic**: Exponential backoff (2^attempt seconds) +- **Overwrite Protection**: Skip existing files unless --overwrite + +Key File: cli/kleinkram/core.py:47-80 +Key File: cli/kleinkram/api/file_transfer.py:342-415 + +### 4. File Verification + +**Command**: `klein verify` + +**Usage**: +```bash +# Verify files were uploaded correctly +klein verify /path/to/local/*.bag --mission "test" --project "MyProject" + +# Check file size only (fast) +klein verify *.mcap --mission "..." --check-file-size + +# Skip hash checking +klein verify *.bag --mission "..." --check-file-hash=False +``` + +**Verification States**: +- `UPLOADED`: File uploaded, hash matches +- `MISSING`: File not on server +- `UPLOADING`: File currently uploading +- `MISMATCHED_HASH`: Hash mismatch +- `MISMATCHED_SIZE`: Size mismatch +- `COMPUTING_HASH`: Server still computing hash +- `UNKNOWN`: Unknown state + +Key File: cli/kleinkram/core.py:147-221 + +### 5. List Operations + +**Command**: `klein list` + +**Subcommands**: +```bash +# List projects +klein list projects + +# List missions in a project +klein list missions --project "ANYmal" + +# List files in a mission +klein list files --mission "..." --project "..." + +# With filtering +klein list files --mission "..." --project "..." --max-lines 100 +``` + +**Output**: Rich formatted tables with columns: +- Projects: name, uuid, description, created +- Missions: name, uuid, project, date, files_count +- Files: filename, uuid, size, state, date, topics_count + +Key File: cli/kleinkram/cli/_list.py + +### 6. CRUD Operations + +#### Project Commands + +```bash +# Create project +klein project create --name "MyProject" --description "..." + +# Update project +klein project update --name "NewName" --description "..." + +# Delete project +klein project delete +``` + +Key File: cli/kleinkram/cli/_project.py + +#### Mission Commands + +```bash +# Create mission +klein mission create --name "mission-name" --project "ProjectName" --date "2024-01-15" + +# Update mission +klein mission update --metadata key=value + +# Delete mission +klein mission delete +``` + +Key File: cli/kleinkram/cli/_mission.py + +#### File Commands + +```bash +# Delete file +klein file delete + +# Get file info +klein file info +``` + +Key File: cli/kleinkram/cli/_file.py + +### 7. Endpoint Management + +**Commands**: +```bash +# Add endpoint +klein endpoint add https://api.example.com + +# Switch endpoint +klein endpoint select https://api.example.com + +# List endpoints +klein endpoint list + +# Remove endpoint +klein endpoint remove https://api.example.com +``` + +Key File: cli/kleinkram/cli/_endpoint.py + +## Query System + +**Module**: `kleinkram.api.query` + +**Query Classes**: +- `ProjectQuery`: Filter projects by name, UUID, IDs +- `MissionQuery`: Filter missions by name, UUID, project, date range +- `FileQuery`: Filter files by name, mission, state, tags, topics, date range + +**Example**: +```python +from kleinkram.api.query import FileQuery, MissionQuery, ProjectQuery + +# Build query +query = FileQuery( + mission_query=MissionQuery( + project_query=ProjectQuery(name="ANYmal"), + name="2024-01-15" + ), + filename="run_001.mcap" +) + +# Execute query +files = get_files(client, file_query=query) +``` + +Key File: cli/kleinkram/api/query.py + +## Error Handling + +**Module**: `kleinkram.errors` + +**Custom Exceptions**: +- `KleinkramError`: Base exception +- `MissionNotFound`: Mission doesn't exist +- `ProjectNotFound`: Project doesn't exist +- `FileNotFound`: File doesn't exist +- `AccessDenied`: Insufficient permissions +- `InvalidCLIVersion`: Version mismatch with API + +**Error Handler**: +```python +@app.error_handler(Exception) +def base_handler(exc: Exception) -> int: + display_error(exc, verbose=shared_state.verbose) + logger.error(format_traceback(exc)) + if shared_state.debug: + raise exc + return 1 +``` + +**Display**: +- Verbose mode: Full traceback +- Normal mode: Error message only +- Debug mode: Re-raises exception + +Key File: cli/kleinkram/cli/error_handling.py + +## Logging + +**Location**: `~/.local/state/kleinkram/` (Linux) or `%LOCALAPPDATA%\kleinkram\` (Windows) + +**Format**: `{timestamp}.log` + +**Levels**: +- DEBUG: Detailed information +- INFO: General information +- WARNING: Warnings (default) +- ERROR: Errors +- CRITICAL: Critical errors + +**Usage**: +```bash +klein --log-level DEBUG upload ... +``` + +Key File: cli/kleinkram/cli/app.py:44-50 + +## Models + +**Module**: `kleinkram.models` + +**Data Classes**: +- `File`: File metadata +- `Project`: Project metadata +- `Mission`: Mission metadata +- `Topic`: ROS topic metadata +- `Action`: Action metadata +- `User`: User metadata + +**Enums**: +- `FileState`: UPLOADING, OK, ERROR, LOST, FOUND, CONVERSION_ERROR +- `FileType`: BAG, MCAP +- `FileOrigin`: API, GOOGLE_DRIVE, UNKNOWN +- `FileVerificationStatus`: UPLOADED, MISSING, MISMATCHED_HASH, ... + +Key File: cli/kleinkram/models.py + +## Output Formatting + +**Module**: `kleinkram.printing` + +**Functions**: +- `files_to_table()`: Format files as Rich table +- `missions_to_table()`: Format missions as Rich table +- `projects_to_table()`: Format projects as Rich table +- `verification_to_table()`: Format verification results + +**Features**: +- Rich tables with borders +- Color-coded status (green/yellow/red) +- Truncation for long fields +- Sortable columns + +Key File: cli/kleinkram.printing.py + +## Utilities + +**Module**: `kleinkram.utils` + +**Key Functions**: +- `b64_md5(path)`: Calculate base64-encoded MD5 hash +- `format_bytes(bytes, speed=False)`: Human-readable byte sizes +- `format_traceback(exc)`: Format exception traceback +- `check_file_paths(paths)`: Validate file paths +- `get_filename_map(paths)`: Map filenames to paths + +Key File: cli/kleinkram/utils.py + +## Common Development Tasks + +### Adding a New Command + +1. Create command file in `cli/kleinkram/cli/_.py` +2. Define Typer app: + ```python + import typer + typer_app = typer.Typer() + ``` +3. Add command functions: + ```python + @typer_app.command() + def my_command(arg: str): + ... + ``` +4. Register in `cli/kleinkram/cli/app.py`: + ```python + from kleinkram.cli._ import typer_app as _typer + app.add_typer(_typer, name="", rich_help_panel=CommandTypes.CORE) + ``` + +### Adding a New API Endpoint Wrapper + +1. Add function to `cli/kleinkram/api/routes.py`: + ```python + def get_my_resource(client: AuthenticatedClient, id: UUID) -> MyResource: + resp = client.get(f'/my-resource/{id}') + resp.raise_for_status() + return MyResource(**resp.json()) + ``` + +### Adding Compression Support for Another Format + +1. Add compression type to `CompressionType` enum +2. Add extension mapping to `COMPRESSION_EXTENSIONS` +3. Implement decompression function (e.g., `decompress_gzip()`) +4. Add case to `decompress_file()` + +### Testing + +**Framework**: pytest + +**Run Tests**: +```bash +cd cli +pytest +``` + +**Test Categories**: +- Unit tests: Individual functions +- Integration tests: API interaction (requires backend) +- E2E tests: Full workflow tests + +## Performance Considerations + +- **Parallel Uploads/Downloads**: 2 workers by default (configurable) +- **Multipart Uploads**: Boto3 handles large files efficiently +- **Streaming**: Downloads stream to disk (low memory usage) +- **Compression**: zstd decompression is very fast +- **Connection Pooling**: httpx client reuses connections +- **Retry Logic**: Exponential backoff prevents server overload + +## Environment Variables + +- `KLEIN_CONFIG_DIR`: Override config directory +- `KLEIN_ENDPOINT`: Override default endpoint +- `KLEIN_LOG_LEVEL`: Override log level + +## Version Compatibility + +**Version Check**: +```python +cli_version = (0, 51, 1) # Major, Minor, Patch +api_version = get_api_version() # From backend + +# Major version must match +assert cli_version[0] == api_version[0] + +# Minor version mismatch: warning +if cli_version[1] != api_version[1]: + print(f"Warning: CLI v{cli_version} may not be compatible with API v{api_version}") +``` + +Key File: cli/kleinkram/cli/app.py:173-187 + +## Security + +- **JWT Tokens**: Stored in config file (permissions: 600) +- **OAuth**: Secure authorization flow via browser +- **S3 Credentials**: Temporary, short-lived credentials +- **API Keys**: Alternative to OAuth for headless environments + +## Installation + +**From PyPI** (when released): +```bash +pip install kleinkram +``` + +**From source**: +```bash +cd cli +pip install -e . +``` + +**Entry Point**: `klein` command + +## Related Documentation + +- [Root AGENTS.md](../AGENTS.md) - System overview +- [Backend AGENTS.md](../backend/AGENTS.md) - API endpoints +- [Common AGENTS.md](../common/AGENTS.md) - Shared DTOs + +## Important Notes + +- **Compression**: Only zstd is supported (`.zst`, `.zstd`) +- **File Types**: Only `.bag` and `.mcap` files allowed +- **Filename Rules**: Alphanumeric, underscores, hyphens, dots, spaces, brackets, umlauts only; max 50 chars +- **Upload Limit**: No hard limit, but large files may timeout +- **Concurrent Operations**: Uploads and downloads run in parallel (2 workers default) + +## Troubleshooting + +1. **Authentication fails**: Check `~/.config/kleinkram/config.json` for valid tokens +2. **Upload hangs**: Check network connection and MinIO accessibility +3. **Hash mismatch**: Re-upload file or check for corruption +4. **Version incompatible**: Update CLI: `pip install --upgrade kleinkram` +5. **Compression error**: Ensure zstandard library installed: `pip install zstandard` diff --git a/common/AGENTS.md b/common/AGENTS.md new file mode 100644 index 000000000..18129e273 --- /dev/null +++ b/common/AGENTS.md @@ -0,0 +1,919 @@ +# Common - AI Agent Documentation + +> **Purpose**: This document provides a detailed overview of the Kleinkram common module for AI agents. + +## Overview + +The common module is a **shared TypeScript library** used by backend, queueConsumer, and frontend. It provides a single source of truth for: +- Database entity definitions (TypeORM) +- API request/response DTOs +- Validation decorators +- Shared enums and constants +- Utility functions +- Environment configuration + +**Key Responsibilities**: +- Type safety across the entire stack +- Consistent data structures +- Validation rules +- Database schema definitions (via TypeORM entities) + +**Technology Stack**: TypeScript, TypeORM, class-validator, class-transformer + +## Project Structure + +``` +common/ +├── api/ # API contracts +│ └── types/ # DTOs for API requests/responses +│ ├── access-control/ # Access control DTOs +│ ├── actions/ # Action DTOs +│ ├── exceptions/ # Error DTOs +│ ├── file/ # File DTOs +│ ├── mission/ # Mission DTOs +│ ├── project/ # Project DTOs +│ ├── queue/ # Queue DTOs +│ ├── tags/ # Tag DTOs +│ ├── *.dto.ts # Misc DTOs +│ └── pagination.ts # Pagination types +├── entities/ # TypeORM database entities +│ ├── action/ # Action entities +│ │ ├── action.entity.ts +│ │ ├── action-template.entity.ts +│ │ └── docker-resource.entity.ts +│ ├── auth/ # Authentication & authorization entities +│ │ ├── accessgroup.entity.ts +│ │ ├── api-key.entity.ts +│ │ ├── group-membership.entity.ts +│ │ ├── mission-access.entity.ts +│ │ └── project-access.entity.ts +│ ├── category/ # Category entities +│ ├── file/ # File entities +│ │ └── file.entity.ts +│ ├── mission/ # Mission entities +│ │ └── mission.entity.ts +│ ├── project/ # Project entities +│ │ └── project.entity.ts +│ ├── queue/ # Queue entities +│ │ └── queue.entity.ts +│ ├── tag/ # Tag entities +│ │ ├── tag.entity.ts +│ │ └── tag-type.entity.ts +│ ├── topic/ # Topic entities +│ │ └── topic.entity.ts +│ ├── user/ # User entities +│ │ └── user.entity.ts +│ ├── worker/ # Worker entities +│ │ └── worker.entity.ts +│ └── base-entity.entity.ts # Base entity class +├── frontend_shared/ # Shared frontend code +│ └── enum.ts # Enums +├── helpers/ # Utility functions +├── viewEntities/ # Database views +│ ├── mission-access-view.entity.ts +│ └── project-access-view.entity.ts +├── consts.ts # Constants +├── environment.ts # Environment configuration +├── minio-helper.ts # MinIO utilities +├── seeds/ # Database seeds +├── package.json +└── tsconfig.json + +Key File: common/entities/ +Key File: common/api/types/ +``` + +## Core Entities + +### Base Entity + +All entities extend `BaseEntity`: + +```typescript +@Entity() +export class BaseEntity { + @PrimaryGeneratedColumn() + id!: number; + + @Column({ type: 'uuid', unique: true }) + @Generated('uuid') + uuid!: string; + + @CreateDateColumn() + createdAt!: Date; + + @UpdateDateColumn() + updatedAt!: Date; + + @DeleteDateColumn({ nullable: true }) + deletedAt?: Date | null; +} +``` + +**Fields**: +- `id`: Auto-incrementing primary key (internal use) +- `uuid`: Universally unique identifier (public facing) +- `createdAt`: Timestamp of creation +- `updatedAt`: Timestamp of last update +- `deletedAt`: Soft delete timestamp (null if not deleted) + +Key File: common/entities/base-entity.entity.ts + +### File Entity + +**Location**: `common/entities/file/file.entity.ts` + +```typescript +@Entity({ name: 'file' }) +export default class FileEntity extends BaseEntity { + @Column() + name!: string; // Original filename + + @Column() + filename!: string; // Display filename + + @Column({ nullable: true }) + hash?: string | null; // MD5 hash (base64) + + @Column({ type: 'bigint' }) + size!: number; // Size in bytes + + @Column({ type: 'enum', enum: FileType }) + type!: FileType; // BAG or MCAP + + @Column({ type: 'enum', enum: FileState }) + state!: FileState; // UPLOADING, OK, ERROR, etc. + + @Column({ type: 'enum', enum: FileOrigin }) + origin!: FileOrigin; // API, GOOGLE_DRIVE, UNKNOWN + + @Column({ nullable: true }) + date?: Date | null; // Recording date + + @ManyToOne(() => Mission) + mission!: Relation; + + @ManyToOne(() => User) + creator!: Relation; + + @OneToMany(() => Topic, (topic) => topic.file) + topics!: Relation; + + @OneToMany(() => Tag, (tag) => tag.file) + tags!: Relation; + + @OneToOne(() => FileEntity, { nullable: true }) + @JoinColumn() + relatedFile?: Relation | null; // bag ↔ mcap link +} +``` + +**Key Relations**: +- `mission`: Many files belong to one mission +- `creator`: Many files created by one user +- `topics`: One file has many topics +- `tags`: One file has many tags +- `relatedFile`: One-to-one (bag ↔ mcap) + +### Mission Entity + +**Location**: `common/entities/mission/mission.entity.ts` + +```typescript +@Entity({ name: 'mission' }) +export default class Mission extends BaseEntity { + @Column() + name!: string; + + @Column({ nullable: true }) + description?: string | null; + + @Column({ nullable: true }) + date?: Date | null; // Recording date + + @ManyToOne(() => Project) + project!: Relation; + + @ManyToOne(() => Category, { nullable: true }) + category?: Relation | null; + + @OneToMany(() => FileEntity, (file) => file.mission) + files!: Relation; + + @OneToMany(() => Action, (action) => action.mission) + actions!: Relation; + + @OneToMany(() => QueueEntity, (queue) => queue.mission) + queues!: Relation; + + @OneToMany(() => MissionAccess, (access) => access.mission) + access!: Relation; + + // Metadata (custom JSON fields) + @Column({ type: 'jsonb', default: {} }) + metadata!: Record; +} +``` + +### Project Entity + +**Location**: `common/entities/project/project.entity.ts` + +```typescript +@Entity({ name: 'project' }) +export default class Project extends BaseEntity { + @Column() + name!: string; + + @Column({ nullable: true }) + description?: string | null; + + @Column({ default: true }) + autoConvert!: boolean; // Auto-convert bag → mcap + + @OneToMany(() => Mission, (mission) => mission.project) + missions!: Relation; + + @OneToMany(() => ProjectAccess, (access) => access.project) + access!: Relation; + + @ManyToMany(() => TagType) + @JoinTable() + requiredTags!: Relation; + + @ManyToMany(() => Category) + @JoinTable() + categories!: Relation; +} +``` + +### Action Entity + +**Location**: `common/entities/action/action.entity.ts` + +```typescript +@Entity({ name: 'action' }) +export default class Action extends BaseEntity { + @Column({ type: 'enum', enum: ActionState }) + state!: ActionState; // PENDING, PROCESSING, DONE, FAILED, CANCELLED + + @Column({ nullable: true }) + stateCause?: string | null; // Error message + + @Column({ type: 'jsonb', default: [] }) + logs!: ContainerLog[]; // Container logs + + @Column({ nullable: true }) + exitCode?: number | null; + + @Column({ nullable: true }) + artifacts?: string | null; // Google Drive URL + + @ManyToOne(() => Mission) + mission!: Relation; + + @ManyToOne(() => ActionTemplate) + template!: Relation; + + @ManyToOne(() => User) + createdBy!: Relation; + + @ManyToOne(() => Worker, { nullable: true }) + worker?: Relation | null; + + @Column({ type: 'simple-json', nullable: true }) + resources?: DockerResourceEntity | null; // CPU, memory, GPU requirements +} +``` + +### User Entity + +**Location**: `common/entities/user/user.entity.ts` + +```typescript +@Entity({ name: 'user' }) +export default class User extends BaseEntity { + @Column({ unique: true }) + email!: string; + + @Column({ nullable: true }) + username?: string | null; + + @Column({ type: 'enum', enum: UserRole, default: UserRole.USER }) + role!: UserRole; // ADMIN or USER + + @OneToMany(() => Action, (action) => action.createdBy) + submittedActions!: Relation; + + @OneToMany(() => ApiKey, (apikey) => apikey.user) + apikeys!: Relation; + + @OneToMany(() => ProjectAccess, (access) => access.user) + projectAccess!: Relation; + + @OneToMany(() => MissionAccess, (access) => access.user) + missionAccess!: Relation; + + @ManyToMany(() => AccessGroup, (group) => group.members) + groups!: Relation; +} +``` + +### Topic Entity + +**Location**: `common/entities/topic/topic.entity.ts` + +```typescript +@Entity({ name: 'topic' }) +export default class Topic extends BaseEntity { + @Column() + name!: string; // e.g., /camera/image_raw + + @Column() + type!: string; // e.g., sensor_msgs/Image + + @Column({ type: 'bigint' }) + nrMessages!: bigint; + + @Column({ type: 'float' }) + frequency!: number; // Hz + + @Column() + messageEncoding!: string; // e.g., cdr + + @ManyToOne(() => FileEntity) + file!: Relation; +} +``` + +### Queue Entity + +**Location**: `common/entities/queue/queue.entity.ts` + +```typescript +@Entity({ name: 'queue' }) +export default class QueueEntity extends BaseEntity { + @Column() + display_name!: string; // User-facing name + + @Column() + identifier!: string; // Internal identifier (file UUID or Drive ID) + + @Column({ type: 'enum', enum: QueueState }) + state!: QueueState; // AWAITING_UPLOAD, AWAITING_PROCESSING, PROCESSING, COMPLETED, ERROR, etc. + + @Column({ type: 'enum', enum: FileLocation }) + location!: FileLocation; // MINIO, GOOGLE_DRIVE, UNKNOWN + + @ManyToOne(() => Mission) + mission!: Relation; + + @ManyToOne(() => User) + creator!: Relation; +} +``` + +### Worker Entity + +**Location**: `common/entities/worker/worker.entity.ts` + +```typescript +@Entity({ name: 'worker' }) +export default class Worker extends BaseEntity { + @Column() + hostname!: string; // Docker hostname + + @Column() + identifier!: string; // Unique machine identifier + + @Column() + cpuCores!: number; + + @Column() + cpuMemory!: number; // GB + + @Column() + cpuModel!: string; + + @Column({ nullable: true }) + gpuModel?: string | null; + + @Column() + gpuMemory!: number; // GB (-1 if no GPU) + + @Column() + storage!: number; // GB + + @Column() + reachable!: boolean; + + @Column() + lastSeen!: Date; + + @OneToMany(() => Action, (action) => action.worker) + actions!: Relation; +} +``` + +### Access Control Entities + +#### Access Group + +**Location**: `common/entities/auth/accessgroup.entity.ts` + +```typescript +@Entity({ name: 'accessgroup' }) +export default class AccessGroup extends BaseEntity { + @Column() + name!: string; + + @Column({ nullable: true }) + description?: string | null; + + @ManyToMany(() => User, (user) => user.groups) + @JoinTable() + members!: Relation; + + @OneToMany(() => GroupMembership, (membership) => membership.group) + memberships!: Relation; + + @OneToMany(() => ProjectAccess, (access) => access.group) + projectAccess!: Relation; + + @OneToMany(() => MissionAccess, (access) => access.group) + missionAccess!: Relation; +} +``` + +#### Project Access + +**Location**: `common/entities/auth/project-access.entity.ts` + +```typescript +@Entity({ name: 'project_access' }) +export default class ProjectAccess extends BaseEntity { + @ManyToOne(() => Project) + project!: Relation; + + @ManyToOne(() => User, { nullable: true }) + user?: Relation | null; + + @ManyToOne(() => AccessGroup, { nullable: true }) + group?: Relation | null; + + @Column({ type: 'enum', enum: AccessGroupRights }) + rights!: AccessGroupRights; // NONE, READ, WRITE, ADMIN +} +``` + +**Note**: Either `user` OR `group` is set, never both. + +#### Mission Access + +Similar to ProjectAccess, but for mission-level permissions. + +**Location**: `common/entities/auth/mission-access.entity.ts` + +## Enums + +**Location**: `common/frontend_shared/enum.ts` + +### FileState +```typescript +enum FileState { + UPLOADING = 'UPLOADING', + OK = 'OK', + ERROR = 'ERROR', + LOST = 'LOST', + FOUND = 'FOUND', + CONVERSION_ERROR = 'CONVERSION_ERROR', +} +``` + +### FileType +```typescript +enum FileType { + BAG = 'BAG', + MCAP = 'MCAP', +} +``` + +### FileOrigin +```typescript +enum FileOrigin { + API = 'API', + GOOGLE_DRIVE = 'GOOGLE_DRIVE', + UNKNOWN = 'UNKNOWN', +} +``` + +### QueueState +```typescript +enum QueueState { + AWAITING_UPLOAD = 'AWAITING_UPLOAD', + AWAITING_PROCESSING = 'AWAITING_PROCESSING', + PROCESSING = 'PROCESSING', + COMPLETED = 'COMPLETED', + ERROR = 'ERROR', + CANCELED = 'CANCELED', + CORRUPTED = 'CORRUPTED', +} +``` + +### ActionState +```typescript +enum ActionState { + PENDING = 'PENDING', + PROCESSING = 'PROCESSING', + DONE = 'DONE', + FAILED = 'FAILED', + CANCELLED = 'CANCELLED', +} +``` + +### AccessGroupRights +```typescript +enum AccessGroupRights { + NONE = 0, + READ = 1, + WRITE = 2, + ADMIN = 3, +} +``` + +### UserRole +```typescript +enum UserRole { + USER = 'USER', + ADMIN = 'ADMIN', +} +``` + +Key File: common/frontend_shared/enum.ts + +## API DTOs + +**Location**: `common/api/types/` + +DTOs define request and response formats for API endpoints. They use `class-validator` decorators for validation. + +### File DTOs + +**Location**: `common/api/types/file/` + +- **FileDto**: Basic file information +- **FileWithTopicDto**: File with topics included +- **UpdateFileDto**: File update request +- **FileQueryDto**: File query parameters + +### Mission DTOs + +**Location**: `common/api/types/mission/` + +- **MissionDto**: Full mission data +- **FlatMissionDto**: Mission without nested data +- **CreateMissionDto**: Mission creation request +- **UpdateMissionDto**: Mission update request + +### Project DTOs + +**Location**: `common/api/types/project/` + +- **ProjectDto**: Full project data +- **CreateProjectDto**: Project creation request +- **UpdateProjectDto**: Project update request + +### Action DTOs + +**Location**: `common/api/types/actions/` + +- **ActionDto**: Full action data +- **SubmitActionDto**: Action submission request +- **ActionWorkerDto**: Worker information for actions + +### Pagination + +**Location**: `common/api/types/pagination.ts` + +```typescript +export interface PaginatedResponse { + data: T[]; + total: number; + page: number; + limit: number; +} + +export interface PaginationOptions { + page?: number; + limit?: number; + sortBy?: string; + sortDirection?: 'ASC' | 'DESC'; +} +``` + +## Environment Configuration + +**Location**: `common/environment.ts` + +Provides type-safe access to environment variables: + +```typescript +export default { + // Database + DB_DATABASE: string; + DB_USER: string; + DB_PASSWORD: string; + DB_PORT: number; + DB_HOST: string; + + // MinIO + MINIO_ACCESS_KEY: string; + MINIO_SECRET_KEY: string; + MINIO_MCAP_BUCKET_NAME: string; + MINIO_BAG_BUCKET_NAME: string; + MINIO_ENDPOINT: string; + + // OAuth + GOOGLE_CLIENT_ID: string; + GOOGLE_CLIENT_SECRET: string; + GITHUB_CLIENT_ID: string; + GITHUB_CLIENT_SECRET: string; + + // JWT + JWT_SECRET: string; + + // App + DEV: boolean; + SERVER_PORT: number; + BASE_URL: string; + ENDPOINT: string; + FRONTEND_URL: string; + + // Google Drive + GOOGLE_ARTIFACT_FOLDER_ID: string; + GOOGLE_ARTIFACT_UPLOADER_KEY_FILE: string; + ARTIFACTS_UPLOADER_IMAGE: string; +} +``` + +**Validation**: Throws error if required variables are missing or invalid type. + +Key File: common/environment.ts:1-194 + +## MinIO Helper + +**Location**: `common/minio-helper.ts` + +Provides utilities for MinIO operations: + +```typescript +// Get MinIO client (internal or external) +export const internalMinio: Client; +export const externalMinio: Client; + +// Get bucket name from file type +export function getBucketFromFileType(type: FileType): string; + +// Other helpers +export function getMinioEndpoint(): string; +``` + +## Constants + +**Location**: `common/consts.ts` + +```typescript +// System user (for automated operations) +export const systemUser = { + uuid: '00000000-0000-0000-0000-000000000000', + email: 'system@kleinkram.local', + username: 'system', +}; + +// Redis configuration +export const redis = { + host: 'redis', + port: 6379, +}; +``` + +## Database Views + +### Mission Access View + +**Location**: `common/viewEntities/mission-access-view.entity.ts` + +Materialized view for efficient access control checks: + +```typescript +@ViewEntity({ + expression: ` + SELECT DISTINCT + ma.missionUUID, + u.uuid AS userUUID, + MAX(ma.rights) AS rights + FROM mission_access_view ma + JOIN user u ON u.uuid = ma.userUUID + GROUP BY ma.missionUUID, u.uuid + `, +}) +export class MissionAccessViewEntity { + @ViewColumn() + missionUUID!: string; + + @ViewColumn() + userUUID!: string; + + @ViewColumn() + rights!: AccessGroupRights; +} +``` + +Combines user-level and group-level access into a single view. + +### Project Access View + +Similar to MissionAccessView, but for projects. + +**Location**: `common/viewEntities/project-access-view.entity.ts` + +## Common Development Tasks + +### Adding a New Entity + +1. Create entity file in `common/entities/{category}/` +2. Extend `BaseEntity` +3. Add decorators: + ```typescript + @Entity({ name: 'my_entity' }) + export default class MyEntity extends BaseEntity { + @Column() + name!: string; + + @ManyToOne(() => OtherEntity) + other!: Relation; + } + ``` +4. Export from `index.ts` +5. Run migrations (if needed) + +### Adding a New DTO + +1. Create DTO file in `common/api/types/{category}/` +2. Add validation decorators: + ```typescript + export class CreateMyEntityDto { + @IsString() + @IsNotEmpty() + name!: string; + + @IsUUID() + otherEntityUuid!: string; + } + ``` +3. Export from `index.ts` + +### Adding a New Enum + +1. Add to `common/frontend_shared/enum.ts`: + ```typescript + export enum MyEnum { + VALUE_1 = 'VALUE_1', + VALUE_2 = 'VALUE_2', + } + ``` +2. Use in entities and DTOs + +### Adding Environment Variable + +1. Add to `common/environment.ts`: + ```typescript + get MY_VAR(): string { + return asString(process.env['MY_VAR']); + } + ``` +2. Add to `.env` file +3. Update docker-compose if needed + +## Validation + +Uses `class-validator` decorators: + +```typescript +@IsString() +@IsNotEmpty() +@MinLength(3) +@MaxLength(50) +name!: string; + +@IsUUID('4') +uuid!: string; + +@IsEmail() +email!: string; + +@IsEnum(FileType) +type!: FileType; + +@IsOptional() +@IsString() +description?: string; +``` + +## Serialization + +Uses `class-transformer` for: +- Excluding sensitive fields (`@Exclude()`) +- Transforming dates (`@Type(() => Date)`) +- Custom serialization (`@Transform()`) + +## Database Migrations + +TypeORM migrations (not currently used, relies on synchronize in DEV): + +```bash +# Generate migration +npm run typeorm migration:generate -- -n MigrationName + +# Run migrations +npm run typeorm migration:run + +# Revert migration +npm run typeorm migration:revert +``` + +## Testing + +Common module is tested indirectly through backend and queueConsumer tests. + +## Related Documentation + +- [Root AGENTS.md](../AGENTS.md) - System overview +- [Backend AGENTS.md](../backend/AGENTS.md) - Backend usage +- [Queue Consumer AGENTS.md](../queueConsumer/AGENTS.md) - Queue consumer usage +- [Frontend AGENTS.md](../frontend/AGENTS.md) - Frontend usage + +## Important Notes + +- **Entity Relations**: Always use `Relation` type for relations +- **UUID**: Use UUID as public identifier, never expose internal `id` +- **Soft Deletes**: All entities support soft delete via `deletedAt` +- **Timestamps**: `createdAt` and `updatedAt` are automatic +- **Validation**: DTOs must have validation decorators +- **Enums**: Use string enums for database storage +- **BigInt**: Use for large numbers (topic message counts, file sizes) + +## TypeORM Patterns + +### Lazy vs Eager Loading + +```typescript +// Lazy loading (default) +const file = await fileRepository.findOne({ where: { uuid } }); +const mission = await file.mission; // Loads on access + +// Eager loading +const file = await fileRepository.findOne({ + where: { uuid }, + relations: ['mission', 'mission.project'], +}); +``` + +### Query Builder + +```typescript +const files = await fileRepository + .createQueryBuilder('file') + .leftJoinAndSelect('file.mission', 'mission') + .leftJoinAndSelect('mission.project', 'project') + .where('project.name = :name', { name: 'ANYmal' }) + .andWhere('file.state = :state', { state: FileState.OK }) + .orderBy('file.createdAt', 'DESC') + .take(50) + .getMany(); +``` + +### Transactions + +```typescript +await dataSource.transaction(async (manager) => { + const file = await manager.save(FileEntity, newFile); + const queue = await manager.save(QueueEntity, newQueue); + return { file, queue }; +}); +``` + +## Performance Considerations + +- **Relations**: Load only necessary relations +- **Pagination**: Always paginate large queries +- **Indexes**: Add indexes for frequently queried columns +- **Views**: Use database views for complex access control queries +- **Caching**: Consider caching frequently accessed data + +## Security + +- **Validation**: All DTOs validated on API boundary +- **Soft Deletes**: Prevent accidental data loss +- **Access Control**: Views optimize permission checks +- **Enums**: Prevent invalid state values diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md new file mode 100644 index 000000000..d2cf24b21 --- /dev/null +++ b/frontend/AGENTS.md @@ -0,0 +1,579 @@ +# Frontend - AI Agent Documentation + +> **Purpose**: This document provides a detailed overview of the Kleinkram frontend architecture for AI agents. + +## Overview + +The frontend is a **Vue 3 web application** built with Quasar framework that provides a user interface for managing robotics datasets. It communicates with the backend REST API for all data operations. + +**Key Responsibilities**: +- Project/Mission/File browsing and management +- File upload (direct to MinIO via S3 SDK) +- Action submission and monitoring +- Access control management +- Data visualization and querying + +**Technology Stack**: Vue 3, Quasar, TypeScript, Tanstack Query (Vue Query), Axios, AWS S3 SDK, ECharts + +## Project Structure + +``` +frontend/ +├── src/ +│ ├── api/ # API client and axios setup +│ ├── boot/ # Quasar boot files +│ ├── components/ # Vue components +│ │ ├── button-wrapper/ # Dialog opener components +│ │ └── *.vue # Reusable components +│ ├── css/ # Global styles +│ ├── dialogs/ # Modal dialogs +│ ├── enums/ # TypeScript enums +│ ├── hooks/ # Vue composables +│ │ ├── crumbs.ts # Breadcrumb hooks +│ │ ├── query-hooks.ts # Tanstack Query hooks +│ │ ├── mutation-hooks.ts # Mutation hooks +│ │ ├── router-hooks.ts # Router utilities +│ │ └── utility-hooks.ts # General utilities +│ ├── layouts/ # Page layouts +│ │ ├── main-layout/ # Main app layout (with nav) +│ │ └── no-top-nav-layout.vue # Auth layout +│ ├── pages/ # Page components +│ ├── router/ # Vue Router configuration +│ │ ├── routes.ts # Route definitions +│ │ └── index.ts # Router setup +│ ├── services/ # Business logic and API calls +│ │ ├── queries/ # GET request services +│ │ ├── mutations/ # POST/PUT/DELETE services +│ │ ├── auth.ts # Authentication +│ │ ├── file-service.ts # File upload logic +│ │ └── *.ts # Other services +│ ├── app.vue # Root component +│ ├── environment.ts # Environment configuration +│ └── quasar.d.ts # Quasar TypeScript declarations +├── public/ # Static assets +├── quasar.config.js # Quasar configuration +├── tsconfig.json # TypeScript configuration +└── package.json + +Key File: frontend/src/router/routes.ts:1-200 +Key File: frontend/src/services/file-service.ts:1-472 +``` + +## Architecture + +### Component Architecture + +``` +App.vue + └── Router View + └── Layout (main-layout or no-top-nav-layout) + └── Page Component + └── Components & Dialogs +``` + +**Key Layouts**: +1. **main-layout**: Standard app layout with top navigation, breadcrumbs, and sidebar +2. **no-top-nav-layout**: Minimal layout for login and error pages + +### State Management + +Kleinkram uses **Tanstack Query (Vue Query)** instead of traditional Vuex/Pinia for state management: + +**Benefits**: +- Automatic caching and invalidation +- Background refetching +- Optimistic updates +- Loading and error states +- Request deduplication + +**Query Keys Pattern**: +```typescript +// Queries are organized by entity type +['files', missionUuid] // Files in a mission +['missions', projectUuid] // Missions in a project +['projects'] // All projects +['action', actionUuid] // Single action +['isUploading'] // Upload status +``` + +Key File: frontend/src/hooks/query-hooks.ts + +### API Client + +**Axios Instance** with interceptors: +- Automatic cookie handling (JWT tokens) +- Error handling +- Response transformation + +Key File: frontend/src/api/axios.ts + +### Routing + +Vue Router with lazy-loaded components: + +**Route Structure**: +``` +/ - Home/Landing +/login - Login page +/projects - Projects list +/project/:projectUuid/missions - Missions in project +/project/:projectUuid/mission/:missionUuid/files - Files in mission +/project/.../file/:fileUuid - File details +/actions - Actions list +/action/:actionUuid - Action details +/upload - Upload page +/access-groups - Access groups +/user-profile - User profile +``` + +**Protected Routes**: All routes except `/login`, `/`, `/error-403`, `/error-404` require authentication. + +Key File: frontend/src/router/routes.ts:1-200 + +## Key Features + +### 1. Authentication + +**Service**: `frontend/src/services/auth.ts` + +```typescript +// Login via OAuth +login(provider: 'google' | 'github') + → Redirects to /auth/{provider} + → Backend handles OAuth flow + → Returns with JWT cookie + +// Get current user +getUser() + → Cached user object + → Fetches from /auth/me + +// Check authentication +isAuthenticated() + +// Logout +logout() + → POST /auth/logout + → Reloads page to clear cache +``` + +**User Cache**: Single in-memory cache to avoid redundant API calls. + +Key File: frontend/src/services/auth.ts:1-57 + +### 2. File Upload + +**Service**: `frontend/src/services/file-service.ts` + +#### Upload Flow + +``` +1. User selects files (drag & drop or file input) + ↓ +2. Validate filenames (.bag/.mcap, valid chars, <50 chars) + ↓ +3. Request temporary S3 credentials + POST /files/temporaryAccess + → Returns access key, secret, session token, bucket, fileUUID + ↓ +4. Upload directly to MinIO via S3 SDK + - Multipart upload (50MB parts) + - MD5 hash calculation (SparkMD5) + - Progress tracking + - Retry logic (up to 60 retries per part) + ↓ +5. Confirm upload with backend + POST /queue/confirmUpload + → Backend enqueues file processing job + ↓ +6. Poll for file status + GET /file/exists + → Every 20 parts during upload +``` + +**Key Features**: +- **Multipart Upload**: 50MB chunks for large files +- **Parallel Uploads**: Up to 5 concurrent files (p-limit) +- **MD5 Verification**: Client-side hash calculation +- **Upload Cancellation**: Abort multipart upload, notify backend +- **Before Unload Warning**: Prevents accidental tab closure + +**Google Drive Import**: +```typescript +driveUpload(missionUuid, driveUrl) + → POST /queue/createDrive + → Backend downloads from Drive + → Files processed asynchronously +``` + +Key File: frontend/src/services/file-service.ts:30-471 + +### 3. File Download + +Files are downloaded via pre-signed MinIO URLs: + +``` +1. GET /files/download?uuid={fileUuid}&expires=true + ↓ +2. Backend generates pre-signed URL (4 hours validity) + ↓ +3. Frontend opens URL in new tab + ↓ +4. Browser downloads directly from MinIO +``` + +### 4. Queries (GET Requests) + +**Location**: `frontend/src/services/queries/` + +Organized by entity: +- **file.ts**: File queries (list, single, topics) +- **mission.ts**: Mission queries +- **project.ts**: Project queries +- **action.ts**: Action queries +- **user.ts**: User queries +- **worker.ts**: Worker queries +- **queue.ts**: Queue status +- **tag.ts**: Tag queries +- **access.ts**: Access control queries + +**Example Query**: +```typescript +// frontend/src/services/queries/file.ts +export const getFiles = (missionUuid: string) => + useQuery({ + queryKey: ['files', missionUuid], + queryFn: () => axios.get(`/files/ofMission?uuid=${missionUuid}`), + staleTime: 30000, // 30 seconds + }); +``` + +### 5. Mutations (POST/PUT/DELETE) + +**Location**: `frontend/src/services/mutations/` + +Organized by entity: +- **file.ts**: File operations (upload, delete, move) +- **mission.ts**: Mission CRUD +- **project.ts**: Project CRUD +- **action.ts**: Action submission +- **queue.ts**: Queue operations +- **access.ts**: Access control + +**Example Mutation**: +```typescript +// frontend/src/services/mutations/file.ts +export const deleteFile = (fileUuid: string) => + axios.delete(`/file/${fileUuid}`); +``` + +**Optimistic Updates**: Some mutations use optimistic updates to improve UX. + +### 6. Data Visualization + +**Library**: ECharts (vue-echarts) + +Used for: +- Storage usage charts +- Action timeline +- File size distribution +- Topic frequency visualization + +### 7. Access Control UI + +**Pages**: +- `/access-groups`: List all access groups +- `/access-group/:uuid`: Access group details + +**Features**: +- Create/edit access groups +- Add/remove users from groups +- Set expiration dates +- Assign group permissions to projects/missions + +## Pages + +### Main Pages + +1. **Dashboard** (`pages/dashboard-page.vue`) + - Storage overview + - Recent activity + - Quick actions + +2. **Projects Explorer** (`pages/projects-explorer-page.vue`) + - List all accessible projects + - Create new projects + - Search and filter + +3. **Missions Explorer** (`pages/missions-explorer-page.vue`) + - List missions in a project + - Create new missions + - Mission metadata + +4. **Files Explorer** (`pages/files-explorer-page.vue`) + - List files in a mission + - Advanced filtering (topics, tags, date range) + - File operations (download, delete, move) + - Pagination + +5. **File Details** (`pages/file-info-page.vue`) + - File metadata + - Topic list + - Tags + - Related files (bag ↔ mcap) + +6. **Actions** (`pages/action-page.vue`) + - List all actions + - Filter by mission, status + - Submit new actions + +7. **Action Details** (`pages/action-details-page.vue`) + - Action status + - Live logs + - Exit code + - Artifacts link + +8. **Upload** (`pages/upload-page.vue`) + - File upload interface + - Google Drive import + - Upload queue status + +9. **User Profile** (`pages/user-profile-page.vue`) + - User information + - API keys management + - Preferences + +### Key Components + +1. **running-actions.vue** + - Shows running actions in sidebar + - Real-time status updates + - Quick access to action logs + +2. **button-wrapper/* ** + - Dialog opener components + - Consistent dialog patterns + - Examples: create-project, delete-file, move-file + +## Hooks (Composables) + +### Query Hooks + +**File**: `frontend/src/hooks/query-hooks.ts` + +Reusable query logic: +```typescript +// Example: Use file query with auto-refresh +const { data: files, isLoading, error } = useFilesQuery(missionUuid); +``` + +### Mutation Hooks + +**File**: `frontend/src/hooks/mutation-hooks.ts` + +Reusable mutation logic with automatic cache invalidation: +```typescript +// Example: Delete file mutation +const { mutate: deleteFile } = useDeleteFileMutation(); + +deleteFile(fileUuid, { + onSuccess: () => { + queryClient.invalidateQueries(['files']); + }, +}); +``` + +### Router Hooks + +**File**: `frontend/src/hooks/router-hooks.ts` + +Router utilities: +- Route parameter extraction +- Navigation helpers +- Breadcrumb generation + +### Utility Hooks + +**File**: `frontend/src/hooks/utility-hooks.ts` + +General utilities: +- Date formatting +- File size formatting +- Permission checks + +## Environment Configuration + +**File**: `frontend/src/environment.ts` + +```typescript +ENV.ENDPOINT // Backend API URL +ENV.BASE_URL // Frontend base URL +ENV.PROJECT_NAME // Project name +ENV.DOCS_URL // Documentation URL +``` + +Loaded from environment variables (set by Quasar build). + +## Common Development Tasks + +### Adding a New Page + +1. Create page component in `src/pages/` +2. Add route in `src/router/routes.ts` +3. Add breadcrumbs if needed +4. Use query hooks for data fetching +5. Add to navigation if needed + +### Adding a New API Call + +1. **Query** (GET): + - Add function in `src/services/queries/{entity}.ts` + - Use Tanstack Query's `useQuery` + - Add query key for caching + +2. **Mutation** (POST/PUT/DELETE): + - Add function in `src/services/mutations/{entity}.ts` + - Use Axios directly or `useMutation` + - Invalidate relevant queries on success + +### Adding a New Dialog + +1. Create dialog component in `src/dialogs/` +2. Create button-wrapper component in `src/components/button-wrapper/` +3. Use Quasar dialog (`useDialogPluginComponent`) +4. Emit events for actions + +### Debugging + +1. **Vue Devtools**: Inspect component state, props, emits +2. **Network Tab**: Check API requests and responses +3. **Tanstack Query Devtools**: Inspect query cache +4. **Console**: Use `console.log()` strategically + +## Important Patterns + +### 1. Query Invalidation + +Always invalidate queries after mutations: +```typescript +await deleteFile(fileUuid); +await queryClient.invalidateQueries(['files', missionUuid]); +``` + +### 2. Loading States + +Use Tanstack Query's built-in loading states: +```typescript +const { data, isLoading, error } = useQuery(...); + +if (isLoading) return ; +if (error) return ; +``` + +### 3. Optimistic Updates + +For better UX, update cache optimistically: +```typescript +const { mutate } = useMutation({ + mutationFn: updateFile, + onMutate: async (newData) => { + // Cancel outgoing queries + await queryClient.cancelQueries(['file', fileUuid]); + + // Snapshot current value + const previous = queryClient.getQueryData(['file', fileUuid]); + + // Optimistically update + queryClient.setQueryData(['file', fileUuid], newData); + + return { previous }; + }, + onError: (err, variables, context) => { + // Rollback on error + queryClient.setQueryData(['file', fileUuid], context.previous); + }, +}); +``` + +### 4. File Upload Progress + +Track upload progress using refs: +```typescript +const uploadingFiles = ref>({}); + +// Update progress during upload +newFileUpload.value.uploaded = (current ?? 0) + chunkSize; +``` + +### 5. Before Unload + +Prevent accidental tab closure during uploads: +```typescript +window.addEventListener('beforeunload', confirmDialog); +// ... upload logic +window.removeEventListener('beforeunload', confirmDialog); +``` + +## Styling + +**Framework**: Quasar (Material Design) + +**Global Styles**: `src/css/` + +**Component Styles**: Scoped `