diff --git a/README.md b/README.md index bb027388..d5b9cfe6 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Quick setup for testing and development - requires Python 3.9+ and Node.js. ## See It in Action -![Kasal UI Screenshot](src/docs/images/kasal-ui-screenshot.png) +![Kasal UI Screenshot](src/images/kasal-ui-screenshot.png) *Visual workflow designer for creating AI agent collaborations* Create your first agent workflow in under 2 minutes: @@ -50,20 +50,15 @@ Create your first agent workflow in under 2 minutes: | Topic | Description | |-------|-------------| -| **[Getting Started](src/docs/GETTING_STARTED.md)** | Complete setup guide for development and deployment | -| **[Deployment Guide](src/docs/DEPLOYMENT_GUIDE.md)** | Build, deployment, and OAuth configuration | -| **[Architecture](src/docs/ARCHITECTURE.md)** | System architecture and design patterns | -| **[CrewAI Engine](src/docs/CREWAI_ENGINE.md)** | AI agent orchestration engine documentation | -| **[Database Migrations](src/docs/DATABASE_MIGRATIONS.md)** | Database schema management with Alembic | -| **[Models & Schemas](src/docs/MODELS.md)** | SQLAlchemy models and Pydantic schemas | -| **[Repository Pattern](src/docs/REPOSITORY_PATTERN.md)** | Data access layer implementation | -| **[LLM Manager](src/docs/LLM_MANAGER.md)** | Multi-provider LLM configuration and management | -| **[Embeddings](src/docs/EMBEDDINGS.md)** | Documentation embeddings for enhanced crew generation | +| **[Why Kasal](src/docs/WHY_KASAL.md)** | What problems it solves and who it's for | +| **[Solution Architecture](src/docs/ARCHITECTURE_GUIDE.md)** | Layers, lifecycles, and platform integration | +| **[Code Structure](src/docs/CODE_STRUCTURE_GUIDE.md)** | Where things live and how to navigate the repo | +| **[Developer Guide](src/docs/DEVELOPER_GUIDE.md)** | Local setup, config, and extension patterns | +| **[API Reference](src/docs/API_REFERENCE.md)** | REST endpoints, payloads, and errors | ### More Documentation -- **[API Documentation](src/docs/)** - Complete API reference -- **[Best Practices](src/docs/BEST_PRACTICES.md)** - Development guidelines -- **[Security Model](src/docs/SECURITY_MODEL.md)** - Authentication and authorization +- **[Docs Hub](src/docs/README.md)** - Documentation index +- **[End‑User Tutorial Catalog](src/docs/END_USER_TUTORIAL_CATALOG.md)** - Screenshot-ready walkthroughs - **[Testing Guide](src/backend/tests/README.md)** - Testing strategy and coverage --- diff --git a/src/docs/API_REFERENCE.md b/src/docs/API_REFERENCE.md index 7fbdc73f..7b9a6b4f 100644 --- a/src/docs/API_REFERENCE.md +++ b/src/docs/API_REFERENCE.md @@ -5,6 +5,7 @@ --- ## Getting Started +Base URLs, authentication, and rate limits you need before calling endpoints. ### Base URL ``` @@ -35,6 +36,7 @@ curl -X GET https://api.example.com/v1/crews \ --- ## Authentication Endpoints +Login, refresh, and logout flows to manage tokens. ### POST /auth/login **Login with credentials** @@ -77,6 +79,7 @@ Response: 204 No Content --- ## Crew Management +Create and manage multi-agent crews and their configurations. ### GET /crews **List all crews** @@ -160,6 +163,7 @@ Response: 204 No Content --- ## Agent Management +Create and list individual agents with roles, models, and tools. ### GET /agents **List all agents** @@ -201,6 +205,7 @@ Response: 201 Created --- ## Execution Management +Start executions, get status, retrieve traces, and stop runs. ### POST /executions **Start crew execution** @@ -264,6 +269,7 @@ Response: 200 OK --- ## Task Management +Create and list tasks assigned to agents. ### GET /tasks **List tasks** @@ -302,6 +308,7 @@ Response: 201 Created --- ## Tool Management +Discover built-in tools and register custom tools. ### GET /tools **List available tools** @@ -344,6 +351,7 @@ Response: 201 Created --- ## Memory Management +Fetch and clear short/long-term memory for a crew. ### GET /memory/{crew_id} **Get crew memory** @@ -378,40 +386,10 @@ Response: 204 No Content --- -## Analytics -### GET /analytics/usage -**Get usage statistics** -```json -Response: 200 OK -{ - "period": "2024-01", - "executions": 1543, - "tokens_used": 2450000, - "success_rate": 0.98, - "avg_duration": 245 -} -``` -### GET /analytics/performance -**Get performance metrics** -```json -Response: 200 OK -{ - "crews": [ - { - "crew_id": "crew_abc123", - "executions": 500, - "success_rate": 0.99, - "avg_duration": 180 - } - ] -} -``` - ---- - -## WebSocket Events +## 🔵 WebSocket Events +Real-time updates for task lifecycle, errors, and progress. ### Connection ```javascript @@ -454,7 +432,8 @@ ws.onopen = () => { --- -## Error Codes +## 🔷 Error Codes +Standardized error responses and meanings. | Code | Message | Description | |------|---------|-------------| @@ -482,7 +461,8 @@ ws.onopen = () => { --- -## Testing +## 🔹 Testing +Sandbox, Postman collection, and OpenAPI spec. ### Sandbox Environment ```bash @@ -493,55 +473,6 @@ curl -X POST https://sandbox-api.kasal.ai/v1/crews \ -d @crew.json ``` -### Postman Collection -[Download Collection](https://api.kasal.ai/postman/kasal-api-v1.json) - -### OpenAPI Spec -[View OpenAPI 3.0 Spec](https://api.kasal.ai/openapi.json) - ---- - -## SDKs & Libraries - -### Python -```python -pip install kasal-sdk - -from kasal import KasalClient - -client = KasalClient(api_key="your_key") -crew = client.crews.create(name="My Crew") -result = client.execute(crew.id, inputs={}) -``` - -### JavaScript/TypeScript -```javascript -npm install @kasal/sdk - -import { Kasal } from '@kasal/sdk'; - -const kasal = new Kasal({ apiKey: 'your_key' }); -const crew = await kasal.crews.create({ name: 'My Crew' }); -const result = await kasal.execute(crew.id, {}); -``` - -### Go -```go -import "github.com/kasal/kasal-go" - -client := kasal.NewClient("your_key") -crew, _ := client.Crews.Create("My Crew") -result, _ := client.Execute(crew.ID, inputs) -``` - ---- - -## API Support - -- **Status Page**: [status.kasal.ai](https://status.kasal.ai) -- **API Console**: [console.kasal.ai](https://console.kasal.ai) -- **Developer Forum**: [forum.kasal.ai](https://forum.kasal.ai) -- **Email**: api-support@kasal.ai --- diff --git a/src/docs/ARCHITECTURE_GUIDE.md b/src/docs/ARCHITECTURE_GUIDE.md index c2a54fdb..0fd234fa 100644 --- a/src/docs/ARCHITECTURE_GUIDE.md +++ b/src/docs/ARCHITECTURE_GUIDE.md @@ -1,4 +1,4 @@ -# Kasal Solution Architecture +# 🔵 Kasal Solution Architecture > **Enterprise AI Orchestration Platform** - Scalable, secure, cloud-native ![Kasal Overvew](././/images/Kasal-overview.png) @@ -6,6 +6,7 @@ --- ## System Overview +What the platform aims to achieve and core design principles. ### Platform Vision **Transform business workflows with autonomous AI agents** - Zero infrastructure complexity @@ -22,6 +23,7 @@ --- ## High-Level Architecture +A big-picture view of the client, application, AI, and data layers. ```mermaid graph TB @@ -71,81 +73,87 @@ graph TB --- ## Architecture Pattern +The layered approach and how requests flow through components. ### High-level - Layered architecture: - Frontend (React SPA) → API (FastAPI) → Services → Repositories → Database - Async-first (async SQLAlchemy, background tasks, queues) -- Config via environment (`src/backend/src/config/settings.py`) -- Pluggable orchestration engine (`src/backend/src/engines/` with CrewAI) +- Config via environment (src/backend/src/config/settings.py) +- Pluggable orchestration engine (src/backend/src/engines/ with CrewAI) ### Request lifecycle (CRUD path) -1) Router in `api/` receives request, validates using `schemas/` -2) Router calls `services/` for business logic -3) Service uses `repositories/` for DB/external I/O -4) Data persisted via `db/session.py` +From HTTP request to response: validation, business logic, and persistence. +1) Router in api/ receives request, validates using schemas/ +2) Router calls services/ for business logic +3) Service uses repositories/ for DB/external I/O +4) Data persisted via db/session.py 5) Response serialized with Pydantic schemas ### Orchestration lifecycle (AI execution) -- Entry via `executions_router.py` → `execution_service.py` -- Service prepares agents/tools/memory and selects engine (`engines/engine_factory.py`) +How executions are prepared, run, and observed using the engine. +- Entry via executions_router.py → execution_service.py +- Service prepares agents/tools/memory and selects engine (engines/engine_factory.py) - CrewAI path: - - Prep: `engines/crewai/crew_preparation.py` and `flow_preparation.py` - - Run: `engines/crewai/execution_runner.py` with callbacks/guardrails - - Observability: `execution_logs_service.py`, `execution_trace_service.py` -- Persist status/history: `execution_repository.py`, `execution_history_repository.py` + - Prep: engines/crewai/crew_preparation.py and flow_preparation.py + - Run: engines/crewai/execution_runner.py with callbacks/guardrails + - Observability: execution_logs_service.py, execution_trace_service.py +- Persist status/history: execution_repository.py, execution_history_repository.py ### Background processing -- Scheduler at startup: `scheduler_service.py` -- Embedding queue (SQLite): `embedding_queue_service.py` (batches writes) -- Startup/shutdown cleanup: `execution_cleanup_service.py` +Schedulers and queues for recurring and long-running tasks. +- Scheduler at startup: scheduler_service.py +- Embedding queue (SQLite): embedding_queue_service.py (batches writes) +- Startup/shutdown cleanup: execution_cleanup_service.py ### Data modeling -- ORM in `models/*` mirrors `schemas/*` +ORM entities, Pydantic schemas, and repository boundaries. +- ORM in models/* mirrors schemas/* - Repositories encapsulate all SQL/external calls (Databricks APIs, Vector Search, MLflow) -- `db/session.py`: +- db/session.py: - Async engine and session factory - SQLite lock retry w/ backoff - - Optional SQL logging via `SQL_DEBUG=true` + - Optional SQL logging via SQL_DEBUG=true ### Auth, identity, and tenancy -- Databricks Apps headers parsed by `utils/user_context.py` +User context, group isolation, and authorization controls. +- Databricks Apps headers parsed by utils/user_context.py - Group-aware multi-tenant context propagated via middleware -- JWT/basic auth routes in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` +- JWT/basic auth routes in auth_router.py, users in users_router.py +- Authorization checks in core/permissions.py ### Security Controls +Defense-in-depth across network, API, data, secrets, and compliance. | Layer | Control | Implementation | |-------|---------|----------------| | **Network** | TLS 1.3 | End-to-end encryption | | **API** | OAuth 2.0 | Databricks SSO | | **Data** | AES-256 | Encryption at rest | -| **Secrets** | Vault | HashiCorp Vault | -| **Compliance** | SOC2 | Audit trails | --- ### Storage Strategy +Where different data types live and why. | Data Type | Storage | Purpose | |-----------|---------|---------| | **Transactional** | PostgreSQL | ACID compliance | -| **Session** | Redis | Fast cache | | **Vectors** | Databricks Vector | Semantic search | -| **Files** | S3/Azure Blob | Document storage | -| **Logs** | CloudWatch/Datadog | Observability | +| **Logs** | MLFlow traces | Observability | --- ### Observability -- Central log manager: `core/logger.py` (writes to `LOG_DIR`) -- API/SQL logging toggles (`LOG_LEVEL`, `SQL_DEBUG`) +Logs, traces, metrics, and how to access them. +- Central log manager: core/logger.py (writes to LOG_DIR) +- API/SQL logging toggles (LOG_LEVEL, SQL_DEBUG) - Execution logs/traces persisted and queryable via dedicated routes/services ### Configuration flags (selected) -- `DOCS_ENABLED`: enables `/api-docs`, `/api-redoc`, `/api-openapi.json` -- `AUTO_SEED_DATABASE`: async background seeders post DB init -- `DATABASE_TYPE`: `postgres` (default) or `sqlite` with `SQLITE_DB_PATH` +Important toggles that affect developer and runtime experience. +- DOCS_ENABLED: enables /docs +- AUTO_SEED_DATABASE: async background seeders post DB init +- DATABASE_TYPE: sqlite with SQLITE_DB_PATH diff --git a/src/docs/BUSINESS_USER_GUIDE.md b/src/docs/BUSINESS_USER_GUIDE.md deleted file mode 100644 index 6e1b5fc7..00000000 --- a/src/docs/BUSINESS_USER_GUIDE.md +++ /dev/null @@ -1,207 +0,0 @@ -# Kasal Business User Guide - -> **AI Workers for Your Business** - No coding required - ---- - -## What is Kasal? - -**Your AI workforce that never sleeps.** Kasal orchestrates AI agents to handle complex workflows automatically, 24/7. - -### Perfect For: -- **Customer Support** → 70% faster response times -- **Content Creation** → 5x more output -- **Data Analysis** → Real-time insights -- **Process Automation** → Zero human errors - -### ROI Calculator -| Your Current Process | With Kasal | Savings | -|---------------------|------------|---------| -| 10 hours/week manual work | 1 hour setup + monitoring | **9 hours/week** | -| $50/hour employee cost | $0.10/hour AI cost | **$2,500/month** | -| 3-day turnaround | Same-day results | **66% faster** | - ---- - -## Quick Start (15 Minutes) - -### Step 1: Choose Your Use Case -Click the template that matches your need: - -| **Customer Support** | **Content Marketing** | **Data Analysis** | -|-------------------------|-------------------------|---------------------| -| Auto-reply to emails | Generate blog posts | Analyze spreadsheets | -| Handle FAQs | Create social media | Build reports | -| Route to right team | SEO optimization | Find patterns | - -### Step 2: Configure Your AI Team -Simply describe what you want: -``` -"I need to respond to customer emails within 1 hour -with professional, helpful answers" -``` - -### Step 3: Connect Your Tools -One-click integrations: -- **Email**: Gmail, Outlook -- **Chat**: Slack, Teams -- **Data**: Excel, Google Sheets -- **CRM**: Salesforce, HubSpot - -### Step 4: Launch -Press **Start** and watch your AI team work! - ---- - -## Success Metrics - -### Real Customer Results - -**TechCorp** -> "Reduced support tickets by 60% in first month" -- **Before**: 200 tickets/day, 24hr response -- **After**: 80 tickets/day, 1hr response -- **Saved**: $120K annually - -**E-Shop Plus** -> "Generated 500 product descriptions in 2 hours" -- **Before**: 1 writer, 20 descriptions/day -- **After**: AI crew, 250 descriptions/hour -- **Result**: Launched 10x faster - -**DataDriven Inc** -> "Daily reports that took 3 hours now take 3 minutes" -- **Before**: Manual Excel analysis -- **After**: Automated insights -- **Impact**: Real-time decisions - ---- - -## Control Center - -### Dashboard Overview -![Dashboard](dashboard-preview.png) - -**Key Metrics at a Glance:** -- **Active Workflows**: See what's running -- **Completed Tasks**: Track progress -- **Performance**: Response times -- **Cost Savings**: ROI tracking - -### Managing Your AI Workers - -**Start/Stop Controls** -- **Run Continuously**: 24/7 automation -- **Business Hours**: 9-5 operation -- **Pause**: Instant stop - -**Quality Controls** -- **Review Mode**: Approve before sending -- **Auto Mode**: Fully autonomous -- **Alert Mode**: Notify on exceptions - ---- - -## Advanced Features - -### Multi-Agent Collaboration -Your AI workers can work as a team: - -**Marketing Campaign Example:** -1. **Researcher** → Finds trending topics -2. **Writer** → Creates content -3. **Designer** → Generates visuals -4. **Analyst** → Measures performance - -All working together, automatically. - -### Custom Workflows -Build complex processes visually: -- Drag & drop workflow designer -- No coding required -- Test before deploying - -### Integration Marketplace -- 100+ pre-built connectors -- API access to any system -- Custom integrations available - ---- - -## Common Questions - -
-How much does it cost? - -**Starter**: $99/month -- 5 AI agents -- 1,000 tasks/month -- Email support - -**Professional**: $499/month -- Unlimited agents -- 10,000 tasks/month -- Priority support - -**Enterprise**: Custom pricing -- Unlimited everything -- Dedicated support -- Custom integrations -
- -
-Is my data secure? - -**Enterprise Security** -- SOC2 Type II certified -- GDPR compliant -- End-to-end encryption -- Your data never trains AI models -
- -
-What if something goes wrong? - -**Built-in Safety** -- Automatic error handling -- Human escalation rules -- Full audit trail -- Instant rollback -
- -
-How fast can I get started? - -**Same Day Setup** -- 15-minute quick start -- Pre-built templates -- Free onboarding call -- 30-day money-back guarantee -
- ---- - -## Ready to Start? - -
- -### [Book a 15-Min Demo](https://kasal.ai/demo) -### [Calculate Your ROI](https://kasal.ai/calculator) -### [Start Free Trial](https://kasal.ai/trial) - -**No credit card required • Cancel anytime** - -
- ---- - -## Need Help? - -- **Chat**: Available 24/7 in-app -- **Email**: support@kasal.ai -- **Phone**: 1-800-KASAL-AI -- **Resources**: [Video Tutorials](https://kasal.ai/videos) - ---- - -*Kasal - Where AI Meets Business Excellence* \ No newline at end of file diff --git a/src/docs/CODE_STRUCTURE_GUIDE.md b/src/docs/CODE_STRUCTURE_GUIDE.md index 0c8232d9..ca4a44e7 100644 --- a/src/docs/CODE_STRUCTURE_GUIDE.md +++ b/src/docs/CODE_STRUCTURE_GUIDE.md @@ -1,104 +1,213 @@ ## Code Structure +A fast, skimmable map of the repository to help you find the right place quickly. -### Top-level -- `README.md`: Overview and documentation index -- `src/entrypoint.py`: App entry for Databricks Apps -- `src/build.py`: Frontend build pipeline and docs copy -- `src/deploy.py`: Deployment automation -- `src/manifest.yaml`, `src/app.yaml`: App metadata/config -- `src/docs/`: Documentation (guides, API reference, images) -- `src/frontend/`: React SPA -- `src/backend/`: FastAPI backend service - -### Backend (`src/backend/src`) -- `main.py`: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, seeds, API router registration -```273:301:/Users/anshu.roy/Documents/kasal/src/backend/src/main.py -# Initialize FastAPI app -app = FastAPI( - title=settings.PROJECT_NAME, - description=settings.PROJECT_DESCRIPTION, - version=settings.VERSION, - lifespan=lifespan, - docs_url="/api-docs" if settings.DOCS_ENABLED else None, - redoc_url="/api-redoc" if settings.DOCS_ENABLED else None, - openapi_url="/api-openapi.json" if settings.DOCS_ENABLED else None, - openapi_version="3.1.0" -) - -# Add user context middleware and include API -app.add_middleware(BaseHTTPMiddleware, dispatch=user_context_middleware) -app.include_router(api_router, prefix=settings.API_V1_STR) +### Repository layout ``` -- `api/`: HTTP route modules (per domain) - - Examples: `executions_router.py`, `execution_logs_router.py`, `execution_trace_router.py`, `engine_config_router.py`, `agents_router.py`, `crews_router.py`, `databricks_*_router.py`, `memory_backend_router.py`, `schemas_router.py`, `tools_router.py` -```48:66:/Users/anshu.roy/Documents/kasal/src/backend/src/api/__init__.py -# Create the main API router -api_router = APIRouter() - -# Include all the sub-routers -api_router.include_router(agents_router) -api_router.include_router(crews_router) -api_router.include_router(databricks_router) -api_router.include_router(databricks_knowledge_router) -api_router.include_router(flows_router) -api_router.include_router(healthcheck_router) -api_router.include_router(logs_router) -api_router.include_router(models_router) -api_router.include_router(databricks_secrets_router) -api_router.include_router(api_keys_router) -api_router.include_router(tasks_router) -api_router.include_router(templates_router) -api_router.include_router(group_tools_router) -api_router.include_router(mlflow_router) -``` -- `services/`: Business logic/orchestration and integrations - - Orchestration: `execution_service.py`, `crewai_execution_service.py`, `process_crew_executor.py`, `scheduler_service.py` - - Integrations: `databricks_*_service.py`, `mlflow_service.py` - - Observability/aux: `execution_logs_service.py`, `execution_trace_service.py`, `documentation_embedding_service.py` -- `repositories/`: Data access (SQL/external/vector/mlflow) - - Examples: `execution_repository.py`, `execution_history_repository.py`, `databricks_vector_index_repository.py`, `database_backup_repository.py` -- `models/`: SQLAlchemy ORM entities (tables) -- `schemas/`: Pydantic request/response models -- `db/`: DB setup and sessions - - `session.py`: async engine/session, SQLite lock retries, optional SQL logging - - `all_models.py`: model import aggregator -- `config/`: App configuration - - `settings.py`: env-driven settings and flags (e.g., `DOCS_ENABLED`, `AUTO_SEED_DATABASE`) - - `logging.py`: logging configuration -- `core/`: Cross-cutting concerns - - `llm_manager.py`: LLM routing/provider config - - `logger.py`: centralized logging manager - - `permissions.py`, `dependencies.py`, `base_service.py`, `base_repository.py`, `unit_of_work.py` - - `entity_extraction_fallback.py`: LLM memory fallback logic -- `engines/`: AI orchestration engines - - `crewai/`: `execution_runner.py`, `crew_preparation.py`, `flow_preparation.py`, callbacks, memory, tools, guardrails - - `engine_factory.py`: engine selection -- `utils/`: Utilities - - `user_context.py`: Databricks header parsing, group context, middleware - - `databricks_url_utils.py`, `databricks_auth.py`, rate limiting, crypto, prompts -- `seeds/`, `scripts/`, `dependencies/`: Seeders, scripts, DI helpers - -### Frontend (`src/frontend`) -- React + TypeScript (CRA + Craco) -- `src/config/api/ApiConfig.ts`: API base URL selection and Axios client -```1:16:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -import axios from 'axios'; - -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), -}; - -export const apiClient = axios.create({ - baseURL: config.apiUrl, - headers: { - 'Content-Type': 'application/json', - }, -}); + +├── README.md +└── src/ + ├── backend/ # FastAPI backend + ├── frontend/ # React + TypeScript frontend + ├── docs/ # Markdown docs (copied to /docs in the app) + ├── build.py # Frontend build + docs copy + ├── deploy.py # Deployment utilities + └── manifest.yaml # App metadata ``` -- `src/api/*Service.ts`: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) -- `src/components/`, `src/app/`, `src/store/`, `src/types/`, `src/utils/`, `src/hooks/`, `src/theme/` -- `public/` assets; `craco.config.js`, `tsconfig.json` \ No newline at end of file + +### Backend (src/backend/src) +- main.py: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, API router registration +- api/: HTTP routers per domain (agents, crews, executions, tools, models, engine-config, etc.) +- services/: Business logic & orchestration + - Orchestration: execution_service.py, crewai_execution_service.py, process_crew_executor.py, scheduler_service.py + - Integrations: databricks_*_service.py, mlflow_service.py + - Observability: execution_logs_service.py, execution_trace_service.py, documentation_embedding_service.py +- repositories/: Data access (SQL/external/vector/mlflow) +- models/: SQLAlchemy entities +- schemas/: Pydantic request/response DTOs +- db/: Sessions and Alembic integration (session.py, all_models.py) +- config/: Settings & logging (settings.py, logging.py) +- core/: Cross-cutting utilities (llm_manager.py, logger.py, permissions.py, unit_of_work.py) +- engines/: AI engine integration (CrewAI prep/runner, memory, tools, guardrails) +- utils/: Helpers (user_context.py, databricks_url_utils.py, etc.) +- seeds/, scripts/, dependencies/: Seeders, scripts, DI helpers + +### Frontend (src/frontend) +- CRA + TypeScript app +- src/config/api/ApiConfig.ts: API base URL selection and Axios client +- src/api/*Service.ts: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) +- src/components/: UI components & views +- src/store/: State management +- src/hooks/: Reusable logic +- src/types/: Shared TS types +- src/utils/, src/theme/: Utilities and theme +- public/: Static assets (docs copied to /docs here) + +### Key entry points +- Backend app starts in main.py (includes api_router with prefix from settings) +- Frontend docs are served from /docs (markdown files copied there at build) + +### Routers (where to look) +Common examples under src/backend/src/api/: +- agents_router.py, crews_router.py, executions_router.py, execution_logs_router.py, execution_trace_router.py +- engine_config_router.py, models_router.py, tools_router.py, schemas_router.py +- databricks_*_router.py (secrets, knowledge, connection) + +### Database & migrations +- DB sessions configured in src/backend/src/db/session.py +- Alembic configuration via alembic.ini (root) and migrations/ (root) +- Models aggregated in src/backend/src/db/all_models.py + +### Configuration & logging +- src/backend/src/config/settings.py: env-driven settings (CORS, DB URIs, docs flags, seeding) +- src/backend/src/config/logging.py and src/backend/src/core/logger.py: centralized logging + +### Core & engines +- src/backend/src/core/llm_manager.py: provider/model selection, streaming options +- src/backend/src/engines/crewai/*: crew preparation, execution runner, callbacks, memory/tool adapters + +### Tips: how to trace a feature +1) Start at the router file for the endpoint. +2) Open the called service and scan business logic. +3) Inspect repository methods and related models. +4) Check Pydantic schemas for request/response contracts. +5) Search for engine usage under engines/crewai if orchestration is involved. + +--- + +## Development conventions (back end) + +### Layering philosophy +- API (FastAPI routers) → Services (business logic) → Repositories (data access) → DB +- Keep routers thin (validation, auth), services cohesive (orchestration/transactions), repositories I/O-only. + +### Naming & structure +- Routers: _router.py (e.g., agents_router.py) +- Services: _service.py +- Repositories: _repository.py +- Models: singular file per entity (e.g., agent.py) +- Schemas: mirror model names (e.g., schemas/agent.py) + +### Request/response contracts +- Define input/output Pydantic models in src/backend/src/schemas/* +- Routers return explicit response models (response_model=...) where practical +- Prefer DTOs over ORM entities at boundaries + +### Transactions & Unit of Work +- Encapsulate write operations inside service methods +- Use the UnitOfWork pattern for multi-repository transactions (core/unit_of_work.py) +- Repositories should not commit; services decide transactional scope + +### Error handling +- Raise HTTPException in routers for user input errors +- Services raise domain errors; routers translate to HTTP +- Avoid broad try/except; log and rethrow with context + +--- + +## Back end deep‑dive (files that matter) + +### App bootstrap +- main.py: lifespan init (logging, DB init, seeders), CORS, user context middleware, include api_router +- config/settings.py: environment-driven settings (DB URIs, docs toggles, seeding) + +### API surface (selected) +- api/__init__.py: composes all routers +- api/executions_router.py: start/stop/get execution +- api/execution_logs_router.py, api/execution_trace_router.py: logs and trace endpoints +- api/engine_config_router.py, api/models_router.py, api/tools_router.py: engine and model config, tool registry + +### Services (selected) +- services/execution_service.py: high-level execution orchestration +- services/crewai_execution_service.py, engines/crewai/execution_runner.py: CrewAI integration points +- services/scheduler_service.py: background scheduling +- services/documentation_embedding_service.py: embeddings for better generation + +### Repositories (selected) +- repositories/execution_repository.py, execution_history_repository.py: persistence for runs +- repositories/databricks_*_repository.py: Databricks secrets, vector index, volumes + +### Database & sessions +- db/session.py: async engine/session, SQLite lock retries, SQL_DEBUG logging +- db/all_models.py: imports all models for Alembic + +### Observability +- core/logger.py: central logger manager (writes to LOG_DIR) +- services/execution_logs_service.py, execution_trace_service.py: persisted logs/trace + +--- + +## Configuration quick reference + +Defined in src/backend/src/config/settings.py: +- DATABASE_TYPE=postgres|sqlite (defaults to postgres) +- SQLITE_DB_PATH=./app.db when using SQLite +- POSTGRES_* envs for Postgres connection +- DOCS_ENABLED=true|false (exposes /api-docs, /api-redoc, /api-openapi.json) +- AUTO_SEED_DATABASE=true|false (background seeding after DB init) +- LOG_LEVEL=INFO|DEBUG +- SQL_DEBUG=true|false (emits SQL to logs for troubleshooting) + +Notes: +- USE_NULLPOOL is set early in main.py to avoid asyncpg pool issues +- Logs default under src/backend/src/logs/ + +--- + +## Engines & orchestration + +- Engine selection: src/backend/src/engines/engine_factory.py +- CrewAI integration lives under src/backend/src/engines/crewai/ + - crew_preparation.py: build agents, tools, memory for a run + - execution_runner.py: run loop, callbacks/guardrails + - trace_management.py: hook into tracing pipeline + +Memory/model caveat: +- Known limitation for specific Databricks models (Claude / GPT‑OSS) on entity extraction +- Automatic fallback to databricks-llama-4-maverick for memory entity extraction only + +--- + +## Front end deep‑dive + +### Docs viewer (this page) +- Markdown fetched from /docs/.md (copied from src/docs at build) +- Mermaid supported via fenced ```mermaid code blocks +- Images rendered responsively; prefer /docs/images/... or relative ./images/... +- Internal markdown links are intercepted to load other docs in‑app + +### API client +- src/frontend/src/config/api/ApiConfig.ts determines API base URL +- Default dev: http://localhost:8000/api/v1; override with REACT_APP_API_URL + +### UI organization +- src/components/: feature folders and shared components +- src/api/: type‑safe client wrappers by domain +- src/store/, src/hooks/, src/utils/, src/theme/ + +--- + +## End‑to‑end example (from API call to DB) + +1) Router (executions_router.py) accepts POST /executions with schema +2) Service (execution_service.py) validates logic, kicks off orchestration +3) Engine (engines/crewai/...) prepares crew and runs execution +4) Logs/Traces recorded via services and repositories +5) Repositories (execution_repository.py) persist status/history +6) Client polls GET /executions/{id} and GET /execution-logs/{id} + +--- + +## Anti‑patterns to avoid +- Business logic in routers (keep slim and delegate) +- Services directly returning ORM entities (use schemas/DTOs) +- Repositories committing transactions (services own commit/rollback) +- Ad‑hoc logging without the central logger (use core/logger.py) + +--- + +## Quick links +- Back end entrypoint: src/backend/src/main.py +- Compose routers: src/backend/src/api/__init__.py +- Settings: src/backend/src/config/settings.py +- Sessions: src/backend/src/db/session.py +- CrewAI runner: src/backend/src/engines/crewai/execution_runner.py diff --git a/src/docs/DEVELOPER_GUIDE.md b/src/docs/DEVELOPER_GUIDE.md index 093f54cf..e7c004fb 100644 --- a/src/docs/DEVELOPER_GUIDE.md +++ b/src/docs/DEVELOPER_GUIDE.md @@ -1,168 +1,145 @@ ## Developer Guide +Build, extend, and debug Kasal efficiently. This guide focuses on day-to-day workflows. + +### Components you'll touch +- **Frontend (React SPA)**: UI, designer, monitoring +- **API (FastAPI)**: REST endpoints and validation +- **Services**: Orchestration and business logic +- **Repositories**: DB and external I/O (Databricks, Vector, MLflow) +- **Engines (CrewAI)**: Agent/flow preparation and execution +- **Data & Storage**: SQLAlchemy models/sessions, embeddings, volumes ### Requirements +Tools and versions you need before running the stack. - Python 3.9+ - Node.js 18+ - Postgres (recommended) or SQLite for local dev - Databricks access if exercising Databricks features - -### Quick start -```bash -# Backend -cd src/backend -python -m venv .venv && source .venv/bin/activate -pip install -r ../requirements.txt -./run.sh # http://localhost:8000 (OpenAPI at /api-docs if enabled) - -# Frontend -cd ../frontend -npm install -npm start # http://localhost:3000 -``` - -Health check: -```bash -curl http://localhost:8000/health -# {"status":"healthy"} +## Developer Architecture Overview +This section gives developers a high-level view of the front end and back end. It explains core components and shows how to understand and trace them. + +## Backend Architecture +The backend uses FastAPI with a clean layered structure. It separates HTTP routing, business services, repositories, and SQLAlchemy models. + +### Core Components +- API Routers: src/backend/src/api/* map HTTP endpoints to service calls. +- Services: src/backend/src/services/* implement business logic and transactions. +- Repositories: src/backend/src/repositories/* handle database CRUD. +- Models/Schemas: src/backend/src/models/* and src/backend/src/schemas/* define persistence and I/O contracts. +- Core/Engines: src/backend/src/core/* and src/backend/src/engines/* integrate LLMs and execution flows. +- DB/Session: src/backend/src/db/* configures sessions and Alembic migrations. +- Config/Security: src/backend/src/config/* and src/backend/src/dependencies/* provide settings and auth. + +### Typical Request Flow +A request passes through router, service, repository, and database. LLM calls route through the LLM manager when needed. + +```mermaid +sequenceDiagram + participant C as Client + participant R as FastAPI Router + participant S as Service + participant U as UnitOfWork + participant Repo as Repository + participant DB as Database + participant L as LLM Manager + + C->>R: HTTP Request + R->>S: Validate and delegate + S->>U: Begin transaction + S->>Repo: Query/Command + Repo->>DB: SQLAlchemy operation + DB-->>Repo: Rows/Status + alt Needs LLM + S->>L: Generate/Score + L-->>S: LLM Result + end + S->>U: Commit + S-->>R: DTO/Schema + R-->>C: HTTP Response ``` -### Configuration -Backend settings: `src/backend/src/config/settings.py` -- Core: `DEBUG_MODE`, `LOG_LEVEL`, `DOCS_ENABLED`, `AUTO_SEED_DATABASE` -- Database: - - `DATABASE_TYPE=postgres|sqlite` (default: `postgres`) - - Postgres: `POSTGRES_SERVER`, `POSTGRES_PORT`, `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` - - SQLite: `SQLITE_DB_PATH=./app.db` - - SQL logging: `SQL_DEBUG=true|false` -- Notes: - - `USE_NULLPOOL` is set early in `main.py` to avoid asyncpg pool issues - - Logs written under `src/backend/src/logs/` - -Frontend API base URL (`REACT_APP_API_URL`) at build-time: -```bash -# Option A: dev default already points to http://localhost:8000/api/v1 -# Option B: override explicitly for a build (Unix/macOS) -REACT_APP_API_URL="http://localhost:8000/api/v1" npm run build - -# When using the top-level build script: -# The env var will propagate into the "npm run build" it runs -cd src -REACT_APP_API_URL="http://localhost:8000/api/v1" python build.py +### How to Understand Backend Components +- Start at the router file for the endpoint path. +- Open the service it calls and read business logic. +- Inspect repository methods and referenced models. +- Check schema types for request and response contracts. +- Review unit tests under `src/backend/tests` for examples. + +### Example: Minimal Endpoint Wiring +This shows a typical router, service, and repository connection. + +```python +# router.py +@router.get("/items/{item_id}", response_model=ItemOut) +async def get_item(item_id: UUID, service: ItemService = Depends(...)): + return await service.get_item(item_id) + +# services/item_service.py +class ItemService: + async def get_item(self, item_id: UUID) -> ItemOut: + with self.uow as uow: + item = uow.items.get(item_id) + return ItemOut.model_validate(item) + +# repositories/item_repository.py +class ItemRepository(BaseRepository[Item]): + ... ``` -### Conventions -- Routers (`api/*`): Validate with `schemas/*`, delegate to `services/*` -- Services: Business logic only; use repositories for I/O -- Repositories: All SQL/external I/O; don’t leak ORM to services -- Models: SQLAlchemy in `models/*`; Schemas: Pydantic in `schemas/*` - -### Add a new API resource (“widgets” example) -1) Model: `models/widget.py`; import in `db/all_models.py` -2) Schemas: `schemas/widget.py` (Create/Update/Read DTOs) -3) Repository: `repositories/widget_repository.py` -4) Service: `services/widget_service.py` -5) Router: `api/widgets_router.py` (validate → call service) -6) Register router in `api/__init__.py` -7) Frontend: add `src/frontend/src/api/widgets.ts` + components/views/state -8) Tests: `src/backend/tests/` - -### Add a new CrewAI tool -- Implement under `engines/crewai/tools/` (follow existing patterns) -- Expose configuration via service/router if user-configurable -- Ensure discovery/registration in the execution path (e.g., prep or service) - -### Executions and tracing -- Start executions via `executions_router.py` endpoints -- Services invoke engine flow (`engines/crewai/*`) -- Logs/traces: - - Execution logs via `execution_logs_*` - - Traces via `execution_trace_*` -```bash -# Kick off an execution -curl -X POST http://localhost:8000/api/v1/executions -H "Content-Type: application/json" -d '{...}' - -# Get execution status -curl http://localhost:8000/api/v1/executions/ - -# Fetch logs/trace -curl http://localhost:8000/api/v1/execution-logs/ -curl http://localhost:8000/api/v1/execution-trace/ +## Frontend Architecture +The frontend is a React + TypeScript application. It organizes UI components, API clients, state stores, hooks, and utilities. + +### Core Components +- API Clients: src/frontend/src/api/* wrap HTTP calls with typed methods. +- UI Components: src/frontend/src/components/* render views and dialogs. +- Hooks: src/frontend/src/hooks/* encapsulate logic and side effects. +- Stores: src/frontend/src/store/* manage app and workflow state. +- Types/Config: src/frontend/src/types/* and src/frontend/src/config/* provide typing and environment. +- Utils: src/frontend/src/utils/* offer reusable helpers. + +### UI Data Flow +Components call hooks, which use stores and API clients. Responses update state and re-render the UI. + +```mermaid +flowchart LR + A[UI Component] --> B[Hook] + B --> C[State Store] + B --> D[API Client] + D -->|HTTP| E[Backend API] + E -->|JSON| D + D --> C + C --> A ``` -### Background processing -- Scheduler: starts on DB-ready startup (`scheduler_service.py`) -- Embedding queue (SQLite): `embedding_queue_service.py` batches writes -- Cleanup on startup/shutdown: `execution_cleanup_service.py` - -### Database & migrations -- SQLite for quick local dev (`DATABASE_TYPE=sqlite`), Postgres for multi-user -- Alembic: -```bash -# after model changes -alembic revision --autogenerate -m "add widgets" -alembic upgrade head -``` - -### Auth, identity, tenancy -- Databricks headers parsed by `utils/user_context.py` -- Group-aware tenants; selected group passed in `group_id` header -- JWT/basic auth in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` - -### Logging & debugging -- App logs: `src/backend/src/logs/` (managed by `core/logger.py`) -- Verbose SQL: `export SQL_DEBUG=true` -- SQLite “database is locked”: mitigated via retry/backoff; reduce writers or use Postgres - -### Frontend notes -- Axios client and base URL: -```1:13:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), +### How to Understand Frontend Components +- Locate the component rendering the feature. +- Check its hook usage and props. +- Open the API client method it calls. +- Review the store slice it reads or writes. +- Inspect related types in src/frontend/src/types. + +### Example: Calling an API From a Component +A component loads items using an API service and updates local state. + +```ts +// api/ItemService.ts +export async function getItem(id: string) { + const res = await fetch(`/v1/items/${id}`, { headers: { Authorization: `Bearer ${token}` } }); + if (!res.ok) throw new Error("Failed"); + return (await res.json()) as Item; +} + +// components/Items/ItemView.tsx +const ItemView: React.FC<{ id: string }> = ({ id }) => { + const [item, setItem] = useState(null); + useEffect(() => { getItem(id).then(setItem); }, [id]); + return item ?
{item.name}
: Loading...; }; -export const apiClient = axios.create({ baseURL: config.apiUrl, headers: { 'Content-Type': 'application/json' } }); -``` - -### Testing -```bash -# Backend -python run_tests.py - -# Frontend -cd src/frontend -npm test ``` -### Production checklist -- Use Postgres (or managed DB), not SQLite -- Harden secrets/tokens; externalize (e.g., Databricks Secrets/Vault) -- Enforce TLS and CORS -- Monitor logs/traces; set alerts -- Review `DOCS_ENABLED`, `LOG_LEVEL`, `DEBUG_MODE` - -## Resources - -### Quick Links -- [API Playground](/api/docs) -- [Video Tutorials](https://kasal.ai/videos) -- [Discord Community](https://discord.gg/kasal) -- [Report Issues](https://github.com/kasal/issues) - -### Code Examples -- [Basic Agent Setup](https://github.com/kasal/examples/basic) -- [Multi-Agent Collaboration](https://github.com/kasal/examples/multi-agent) -- [Custom Tools](https://github.com/kasal/examples/tools) -- [Production Deployment](https://github.com/kasal/examples/deploy) - -### Support -- **Chat**: Available in-app 24/7 -- **Email**: dev@kasal.ai -- **Slack**: #kasal-developers - ---- +## End-to-End flow +This ties front end and back end with shared contracts. It helps new developers trace a feature quickly. -*Build smarter, ship faster with Kasal* \ No newline at end of file +- Frontend Component → Hook → Store/API Client → Backend Router → Service → Repository → DB. +- Shared types and response shapes live in frontend types and backend schemas. +- Tests in src/backend/tests and frontend __tests__ show usage patterns. \ No newline at end of file diff --git a/src/docs/README.md b/src/docs/README.md index 515063a6..f6cb427d 100644 --- a/src/docs/README.md +++ b/src/docs/README.md @@ -1,31 +1,19 @@ -![Kasal Logo](././/images/logo.png) -# 🟦 Kasal Documentation Hub + +# Kasal Documentation Hub **Enterprise AI Agent Orchestration Platform** --- ## Quick Start +Find the right documentation for your role and get productive fast. Choose your documentation based on your role: -## Business Documentation - -### [Business User Guide](./BUSINESS_USER_GUIDE.md) -**For Leaders, Managers, and Business Users** - -Get started with Kasal without any technical knowledge. Learn how to create AI workflows, calculate ROI, and manage your AI workforce. - -**Key Topics:** -- ROI Calculator and Cost Analysis -- Success Stories and Case Studies -- 15-Minute Quick Setup Guide -- Managing AI Workflows Without Code - ---- ## Technical Documentation +Developer-focused guides, code structure, and API reference. ### [Code structure Guide](./CODE_STRUCTURE_GUIDE.md) **For Software Engineers and Developers** @@ -62,6 +50,7 @@ Complete REST API documentation with examples, WebSocket events, and SDK librari --- ## Architecture Documentation +System design, patterns, and enterprise integration guidance. ### [Solution Architecture Guide](./ARCHITECTURE_GUIDE.md) **For Solution Architects and Technical Leaders** @@ -77,6 +66,7 @@ Understand Kasal's architecture, design patterns, and enterprise integration str --- ## Getting Started in 30 Seconds +One command sequence to run both backend and frontend locally. ```bash # Clone the repository @@ -97,6 +87,7 @@ Access the application at `http://localhost:3000` --- ## Key Features +What Kasal provides out of the box to build, operate, and govern AI workflows. | Feature | Description | |---------|-------------| @@ -110,11 +101,11 @@ Access the application at `http://localhost:3000` --- ## Documentation Structure +How this folder is organized and where to find topics. ``` docs/ ├── README.md # This file - Documentation hub -├── BUSINESS_USER_GUIDE.md # Business user documentation ├── CODE_STRUCTURE_GUIDE.md # Code structure documentation ├── DEVELOPER_GUIDE.md # Developer documentation ├── ARCHITECTURE_GUIDE.md # Architecture documentation @@ -128,6 +119,7 @@ docs/ --- ## Technology Stack +Core frameworks and platforms used across the project. - **Backend**: FastAPI, SQLAlchemy 2.0, Python 3.9+ - **Frontend**: React 18, TypeScript, Material-UI @@ -138,6 +130,7 @@ docs/ --- ## Support and Resources +Where to get help and how to contribute. ### Getting Help @@ -156,9 +149,10 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](https: --- ## Version Information +Project and documentation version details. - **Current Version**: 2.0.0 -- **Documentation Updated**: January 2025 +- **Documentation Updated**: October 2025 - **Minimum Python Version**: 3.9 - **Minimum Node Version**: 16.0 diff --git a/src/docs/WHY_KASAL.md b/src/docs/WHY_KASAL.md index 49088752..15c0c095 100644 --- a/src/docs/WHY_KASAL.md +++ b/src/docs/WHY_KASAL.md @@ -22,6 +22,49 @@ Transform your Databricks environment into an AI orchestration powerhouse. Build - **Extensible AI Engine**: CrewAI integration today, engine abstraction for future engines. - **Governance & Security**: Group-aware multi-tenant model, API keys, permissions, and auditability. +## Components at a glance +A quick tour of the building blocks—what each part does and why it matters. + +### Frontend (React SPA) +- **What it does**: Visual designer for agents, tasks, and flows; live monitoring UI. +- **Why it matters**: Non-technical users can build and operate AI workflows without touching code. + +### API (FastAPI) +- **What it does**: Validates requests, exposes REST endpoints, and routes calls to services. +- **Why it matters**: Clear, versioned contracts between UI/automation and backend logic. + +### Services (Business Logic) +- **What it does**: Implements orchestration, validation, scheduling, and domain logic. +- **Why it matters**: Keeps HTTP thin and domain logic testable and reusable. + +### Repositories (Data Access) +- **What it does**: Encapsulates SQL and external I/O (Databricks APIs, Vector Search, MLflow). +- **Why it matters**: Swappable persistence and integrations without leaking into services. + +### Engines (CrewAI Orchestration) +- **What it does**: Prepares crews, runs executions, handles callbacks/guardrails, manages memory. +- **Why it matters**: Pluggable execution engine today (CrewAI) and extensible for future engines. + +### Data & Storage +- **What it does**: Async SQLAlchemy sessions, models/schemas, vector indexes, volumes. +- **Why it matters**: Reliable persistence with optional vector search and document storage. + +### Scheduler & Background Jobs +- **What it does**: Recurring runs, long tasks, and background queues (e.g., embedding batching). +- **Why it matters**: Production-ready operations beyond single request/response. + +### Observability +- **What it does**: Structured logs, execution logs, traces, history, health checks. +- **Why it matters**: Debug fast, audit runs, and understand system behavior end-to-end. + +### Security & Governance +- **What it does**: Group-aware multi-tenancy, JWT/Databricks headers, centralized permissions. +- **Why it matters**: Safely share across teams while isolating data and enforcing roles. + +### Databricks Integrations +- **What it does**: OAuth, Secrets, SQL Warehouses, Unity Catalog, Volumes, Vector Search, MLflow. +- **Why it matters**: Build where your data and models already live with first-class support. + --- ## What Makes Kasal Different diff --git a/src/docs/images/Kasal-overview.png b/src/docs/images/Kasal-overview.png new file mode 100644 index 00000000..df9aa78e Binary files /dev/null and b/src/docs/images/Kasal-overview.png differ diff --git a/src/docs/images/logo.png b/src/docs/images/logo.png new file mode 100644 index 00000000..2c5bf58d Binary files /dev/null and b/src/docs/images/logo.png differ diff --git a/src/frontend/package.json b/src/frontend/package.json index 3be7981a..6f2950b5 100644 --- a/src/frontend/package.json +++ b/src/frontend/package.json @@ -28,6 +28,7 @@ "jspdf-autotable": "^4.0.0", "lodash.throttle": "^4.1.1", "marked": "^12.0.0", + "mermaid": "^10.9.1", "notistack": "^3.0.1", "prism-react-renderer": "^2.3.0", "react": "^18.2.0", diff --git a/src/frontend/public/Kasal-overview.png b/src/frontend/public/Kasal-overview.png new file mode 100644 index 00000000..df9aa78e Binary files /dev/null and b/src/frontend/public/Kasal-overview.png differ diff --git a/src/frontend/public/docs/API_REFERENCE.md b/src/frontend/public/docs/API_REFERENCE.md index 7fbdc73f..7b9a6b4f 100644 --- a/src/frontend/public/docs/API_REFERENCE.md +++ b/src/frontend/public/docs/API_REFERENCE.md @@ -5,6 +5,7 @@ --- ## Getting Started +Base URLs, authentication, and rate limits you need before calling endpoints. ### Base URL ``` @@ -35,6 +36,7 @@ curl -X GET https://api.example.com/v1/crews \ --- ## Authentication Endpoints +Login, refresh, and logout flows to manage tokens. ### POST /auth/login **Login with credentials** @@ -77,6 +79,7 @@ Response: 204 No Content --- ## Crew Management +Create and manage multi-agent crews and their configurations. ### GET /crews **List all crews** @@ -160,6 +163,7 @@ Response: 204 No Content --- ## Agent Management +Create and list individual agents with roles, models, and tools. ### GET /agents **List all agents** @@ -201,6 +205,7 @@ Response: 201 Created --- ## Execution Management +Start executions, get status, retrieve traces, and stop runs. ### POST /executions **Start crew execution** @@ -264,6 +269,7 @@ Response: 200 OK --- ## Task Management +Create and list tasks assigned to agents. ### GET /tasks **List tasks** @@ -302,6 +308,7 @@ Response: 201 Created --- ## Tool Management +Discover built-in tools and register custom tools. ### GET /tools **List available tools** @@ -344,6 +351,7 @@ Response: 201 Created --- ## Memory Management +Fetch and clear short/long-term memory for a crew. ### GET /memory/{crew_id} **Get crew memory** @@ -378,40 +386,10 @@ Response: 204 No Content --- -## Analytics -### GET /analytics/usage -**Get usage statistics** -```json -Response: 200 OK -{ - "period": "2024-01", - "executions": 1543, - "tokens_used": 2450000, - "success_rate": 0.98, - "avg_duration": 245 -} -``` -### GET /analytics/performance -**Get performance metrics** -```json -Response: 200 OK -{ - "crews": [ - { - "crew_id": "crew_abc123", - "executions": 500, - "success_rate": 0.99, - "avg_duration": 180 - } - ] -} -``` - ---- - -## WebSocket Events +## 🔵 WebSocket Events +Real-time updates for task lifecycle, errors, and progress. ### Connection ```javascript @@ -454,7 +432,8 @@ ws.onopen = () => { --- -## Error Codes +## 🔷 Error Codes +Standardized error responses and meanings. | Code | Message | Description | |------|---------|-------------| @@ -482,7 +461,8 @@ ws.onopen = () => { --- -## Testing +## 🔹 Testing +Sandbox, Postman collection, and OpenAPI spec. ### Sandbox Environment ```bash @@ -493,55 +473,6 @@ curl -X POST https://sandbox-api.kasal.ai/v1/crews \ -d @crew.json ``` -### Postman Collection -[Download Collection](https://api.kasal.ai/postman/kasal-api-v1.json) - -### OpenAPI Spec -[View OpenAPI 3.0 Spec](https://api.kasal.ai/openapi.json) - ---- - -## SDKs & Libraries - -### Python -```python -pip install kasal-sdk - -from kasal import KasalClient - -client = KasalClient(api_key="your_key") -crew = client.crews.create(name="My Crew") -result = client.execute(crew.id, inputs={}) -``` - -### JavaScript/TypeScript -```javascript -npm install @kasal/sdk - -import { Kasal } from '@kasal/sdk'; - -const kasal = new Kasal({ apiKey: 'your_key' }); -const crew = await kasal.crews.create({ name: 'My Crew' }); -const result = await kasal.execute(crew.id, {}); -``` - -### Go -```go -import "github.com/kasal/kasal-go" - -client := kasal.NewClient("your_key") -crew, _ := client.Crews.Create("My Crew") -result, _ := client.Execute(crew.ID, inputs) -``` - ---- - -## API Support - -- **Status Page**: [status.kasal.ai](https://status.kasal.ai) -- **API Console**: [console.kasal.ai](https://console.kasal.ai) -- **Developer Forum**: [forum.kasal.ai](https://forum.kasal.ai) -- **Email**: api-support@kasal.ai --- diff --git a/src/frontend/public/docs/ARCHITECTURE_GUIDE.md b/src/frontend/public/docs/ARCHITECTURE_GUIDE.md index c2a54fdb..ca1a02f9 100644 --- a/src/frontend/public/docs/ARCHITECTURE_GUIDE.md +++ b/src/frontend/public/docs/ARCHITECTURE_GUIDE.md @@ -1,11 +1,12 @@ -# Kasal Solution Architecture +# 🔵 Kasal Solution Architecture > **Enterprise AI Orchestration Platform** - Scalable, secure, cloud-native -![Kasal Overvew](././/images/Kasal-overview.png) +![Kasal Overvew](/Kasal-overview.png) *Visual workflow designer for creating AI agent collaborations* --- ## System Overview +What the platform aims to achieve and core design principles. ### Platform Vision **Transform business workflows with autonomous AI agents** - Zero infrastructure complexity @@ -22,6 +23,7 @@ --- ## High-Level Architecture +A big-picture view of the client, application, AI, and data layers. ```mermaid graph TB @@ -71,81 +73,87 @@ graph TB --- ## Architecture Pattern +The layered approach and how requests flow through components. ### High-level - Layered architecture: - Frontend (React SPA) → API (FastAPI) → Services → Repositories → Database - Async-first (async SQLAlchemy, background tasks, queues) -- Config via environment (`src/backend/src/config/settings.py`) -- Pluggable orchestration engine (`src/backend/src/engines/` with CrewAI) +- Config via environment (src/backend/src/config/settings.py) +- Pluggable orchestration engine (src/backend/src/engines/ with CrewAI) ### Request lifecycle (CRUD path) -1) Router in `api/` receives request, validates using `schemas/` -2) Router calls `services/` for business logic -3) Service uses `repositories/` for DB/external I/O -4) Data persisted via `db/session.py` +From HTTP request to response: validation, business logic, and persistence. +1) Router in api/ receives request, validates using schemas/ +2) Router calls services/ for business logic +3) Service uses repositories/ for DB/external I/O +4) Data persisted via db/session.py 5) Response serialized with Pydantic schemas ### Orchestration lifecycle (AI execution) -- Entry via `executions_router.py` → `execution_service.py` -- Service prepares agents/tools/memory and selects engine (`engines/engine_factory.py`) +How executions are prepared, run, and observed using the engine. +- Entry via executions_router.py → execution_service.py +- Service prepares agents/tools/memory and selects engine (engines/engine_factory.py) - CrewAI path: - - Prep: `engines/crewai/crew_preparation.py` and `flow_preparation.py` - - Run: `engines/crewai/execution_runner.py` with callbacks/guardrails - - Observability: `execution_logs_service.py`, `execution_trace_service.py` -- Persist status/history: `execution_repository.py`, `execution_history_repository.py` + - Prep: engines/crewai/crew_preparation.py and flow_preparation.py + - Run: engines/crewai/execution_runner.py with callbacks/guardrails + - Observability: execution_logs_service.py, execution_trace_service.py +- Persist status/history: execution_repository.py, execution_history_repository.py ### Background processing -- Scheduler at startup: `scheduler_service.py` -- Embedding queue (SQLite): `embedding_queue_service.py` (batches writes) -- Startup/shutdown cleanup: `execution_cleanup_service.py` +Schedulers and queues for recurring and long-running tasks. +- Scheduler at startup: scheduler_service.py +- Embedding queue (SQLite): embedding_queue_service.py (batches writes) +- Startup/shutdown cleanup: execution_cleanup_service.py ### Data modeling -- ORM in `models/*` mirrors `schemas/*` +ORM entities, Pydantic schemas, and repository boundaries. +- ORM in models/* mirrors schemas/* - Repositories encapsulate all SQL/external calls (Databricks APIs, Vector Search, MLflow) -- `db/session.py`: +- db/session.py: - Async engine and session factory - SQLite lock retry w/ backoff - - Optional SQL logging via `SQL_DEBUG=true` + - Optional SQL logging via SQL_DEBUG=true ### Auth, identity, and tenancy -- Databricks Apps headers parsed by `utils/user_context.py` +User context, group isolation, and authorization controls. +- Databricks Apps headers parsed by utils/user_context.py - Group-aware multi-tenant context propagated via middleware -- JWT/basic auth routes in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` +- JWT/basic auth routes in auth_router.py, users in users_router.py +- Authorization checks in core/permissions.py ### Security Controls +Defense-in-depth across network, API, data, secrets, and compliance. | Layer | Control | Implementation | |-------|---------|----------------| | **Network** | TLS 1.3 | End-to-end encryption | | **API** | OAuth 2.0 | Databricks SSO | | **Data** | AES-256 | Encryption at rest | -| **Secrets** | Vault | HashiCorp Vault | -| **Compliance** | SOC2 | Audit trails | --- ### Storage Strategy +Where different data types live and why. | Data Type | Storage | Purpose | |-----------|---------|---------| | **Transactional** | PostgreSQL | ACID compliance | -| **Session** | Redis | Fast cache | | **Vectors** | Databricks Vector | Semantic search | -| **Files** | S3/Azure Blob | Document storage | -| **Logs** | CloudWatch/Datadog | Observability | +| **Logs** | MLFlow traces | Observability | --- ### Observability -- Central log manager: `core/logger.py` (writes to `LOG_DIR`) -- API/SQL logging toggles (`LOG_LEVEL`, `SQL_DEBUG`) +Logs, traces, metrics, and how to access them. +- Central log manager: core/logger.py (writes to LOG_DIR) +- API/SQL logging toggles (LOG_LEVEL, SQL_DEBUG) - Execution logs/traces persisted and queryable via dedicated routes/services ### Configuration flags (selected) -- `DOCS_ENABLED`: enables `/api-docs`, `/api-redoc`, `/api-openapi.json` -- `AUTO_SEED_DATABASE`: async background seeders post DB init -- `DATABASE_TYPE`: `postgres` (default) or `sqlite` with `SQLITE_DB_PATH` +Important toggles that affect developer and runtime experience. +- DOCS_ENABLED: enables /docs +- AUTO_SEED_DATABASE: async background seeders post DB init +- DATABASE_TYPE: sqlite with SQLITE_DB_PATH diff --git a/src/frontend/public/docs/BUSINESS_USER_GUIDE.md b/src/frontend/public/docs/BUSINESS_USER_GUIDE.md deleted file mode 100644 index 6e1b5fc7..00000000 --- a/src/frontend/public/docs/BUSINESS_USER_GUIDE.md +++ /dev/null @@ -1,207 +0,0 @@ -# Kasal Business User Guide - -> **AI Workers for Your Business** - No coding required - ---- - -## What is Kasal? - -**Your AI workforce that never sleeps.** Kasal orchestrates AI agents to handle complex workflows automatically, 24/7. - -### Perfect For: -- **Customer Support** → 70% faster response times -- **Content Creation** → 5x more output -- **Data Analysis** → Real-time insights -- **Process Automation** → Zero human errors - -### ROI Calculator -| Your Current Process | With Kasal | Savings | -|---------------------|------------|---------| -| 10 hours/week manual work | 1 hour setup + monitoring | **9 hours/week** | -| $50/hour employee cost | $0.10/hour AI cost | **$2,500/month** | -| 3-day turnaround | Same-day results | **66% faster** | - ---- - -## Quick Start (15 Minutes) - -### Step 1: Choose Your Use Case -Click the template that matches your need: - -| **Customer Support** | **Content Marketing** | **Data Analysis** | -|-------------------------|-------------------------|---------------------| -| Auto-reply to emails | Generate blog posts | Analyze spreadsheets | -| Handle FAQs | Create social media | Build reports | -| Route to right team | SEO optimization | Find patterns | - -### Step 2: Configure Your AI Team -Simply describe what you want: -``` -"I need to respond to customer emails within 1 hour -with professional, helpful answers" -``` - -### Step 3: Connect Your Tools -One-click integrations: -- **Email**: Gmail, Outlook -- **Chat**: Slack, Teams -- **Data**: Excel, Google Sheets -- **CRM**: Salesforce, HubSpot - -### Step 4: Launch -Press **Start** and watch your AI team work! - ---- - -## Success Metrics - -### Real Customer Results - -**TechCorp** -> "Reduced support tickets by 60% in first month" -- **Before**: 200 tickets/day, 24hr response -- **After**: 80 tickets/day, 1hr response -- **Saved**: $120K annually - -**E-Shop Plus** -> "Generated 500 product descriptions in 2 hours" -- **Before**: 1 writer, 20 descriptions/day -- **After**: AI crew, 250 descriptions/hour -- **Result**: Launched 10x faster - -**DataDriven Inc** -> "Daily reports that took 3 hours now take 3 minutes" -- **Before**: Manual Excel analysis -- **After**: Automated insights -- **Impact**: Real-time decisions - ---- - -## Control Center - -### Dashboard Overview -![Dashboard](dashboard-preview.png) - -**Key Metrics at a Glance:** -- **Active Workflows**: See what's running -- **Completed Tasks**: Track progress -- **Performance**: Response times -- **Cost Savings**: ROI tracking - -### Managing Your AI Workers - -**Start/Stop Controls** -- **Run Continuously**: 24/7 automation -- **Business Hours**: 9-5 operation -- **Pause**: Instant stop - -**Quality Controls** -- **Review Mode**: Approve before sending -- **Auto Mode**: Fully autonomous -- **Alert Mode**: Notify on exceptions - ---- - -## Advanced Features - -### Multi-Agent Collaboration -Your AI workers can work as a team: - -**Marketing Campaign Example:** -1. **Researcher** → Finds trending topics -2. **Writer** → Creates content -3. **Designer** → Generates visuals -4. **Analyst** → Measures performance - -All working together, automatically. - -### Custom Workflows -Build complex processes visually: -- Drag & drop workflow designer -- No coding required -- Test before deploying - -### Integration Marketplace -- 100+ pre-built connectors -- API access to any system -- Custom integrations available - ---- - -## Common Questions - -
-How much does it cost? - -**Starter**: $99/month -- 5 AI agents -- 1,000 tasks/month -- Email support - -**Professional**: $499/month -- Unlimited agents -- 10,000 tasks/month -- Priority support - -**Enterprise**: Custom pricing -- Unlimited everything -- Dedicated support -- Custom integrations -
- -
-Is my data secure? - -**Enterprise Security** -- SOC2 Type II certified -- GDPR compliant -- End-to-end encryption -- Your data never trains AI models -
- -
-What if something goes wrong? - -**Built-in Safety** -- Automatic error handling -- Human escalation rules -- Full audit trail -- Instant rollback -
- -
-How fast can I get started? - -**Same Day Setup** -- 15-minute quick start -- Pre-built templates -- Free onboarding call -- 30-day money-back guarantee -
- ---- - -## Ready to Start? - -
- -### [Book a 15-Min Demo](https://kasal.ai/demo) -### [Calculate Your ROI](https://kasal.ai/calculator) -### [Start Free Trial](https://kasal.ai/trial) - -**No credit card required • Cancel anytime** - -
- ---- - -## Need Help? - -- **Chat**: Available 24/7 in-app -- **Email**: support@kasal.ai -- **Phone**: 1-800-KASAL-AI -- **Resources**: [Video Tutorials](https://kasal.ai/videos) - ---- - -*Kasal - Where AI Meets Business Excellence* \ No newline at end of file diff --git a/src/frontend/public/docs/CODE_STRUCTURE_GUIDE.md b/src/frontend/public/docs/CODE_STRUCTURE_GUIDE.md index 0c8232d9..ca4a44e7 100644 --- a/src/frontend/public/docs/CODE_STRUCTURE_GUIDE.md +++ b/src/frontend/public/docs/CODE_STRUCTURE_GUIDE.md @@ -1,104 +1,213 @@ ## Code Structure +A fast, skimmable map of the repository to help you find the right place quickly. -### Top-level -- `README.md`: Overview and documentation index -- `src/entrypoint.py`: App entry for Databricks Apps -- `src/build.py`: Frontend build pipeline and docs copy -- `src/deploy.py`: Deployment automation -- `src/manifest.yaml`, `src/app.yaml`: App metadata/config -- `src/docs/`: Documentation (guides, API reference, images) -- `src/frontend/`: React SPA -- `src/backend/`: FastAPI backend service - -### Backend (`src/backend/src`) -- `main.py`: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, seeds, API router registration -```273:301:/Users/anshu.roy/Documents/kasal/src/backend/src/main.py -# Initialize FastAPI app -app = FastAPI( - title=settings.PROJECT_NAME, - description=settings.PROJECT_DESCRIPTION, - version=settings.VERSION, - lifespan=lifespan, - docs_url="/api-docs" if settings.DOCS_ENABLED else None, - redoc_url="/api-redoc" if settings.DOCS_ENABLED else None, - openapi_url="/api-openapi.json" if settings.DOCS_ENABLED else None, - openapi_version="3.1.0" -) - -# Add user context middleware and include API -app.add_middleware(BaseHTTPMiddleware, dispatch=user_context_middleware) -app.include_router(api_router, prefix=settings.API_V1_STR) +### Repository layout ``` -- `api/`: HTTP route modules (per domain) - - Examples: `executions_router.py`, `execution_logs_router.py`, `execution_trace_router.py`, `engine_config_router.py`, `agents_router.py`, `crews_router.py`, `databricks_*_router.py`, `memory_backend_router.py`, `schemas_router.py`, `tools_router.py` -```48:66:/Users/anshu.roy/Documents/kasal/src/backend/src/api/__init__.py -# Create the main API router -api_router = APIRouter() - -# Include all the sub-routers -api_router.include_router(agents_router) -api_router.include_router(crews_router) -api_router.include_router(databricks_router) -api_router.include_router(databricks_knowledge_router) -api_router.include_router(flows_router) -api_router.include_router(healthcheck_router) -api_router.include_router(logs_router) -api_router.include_router(models_router) -api_router.include_router(databricks_secrets_router) -api_router.include_router(api_keys_router) -api_router.include_router(tasks_router) -api_router.include_router(templates_router) -api_router.include_router(group_tools_router) -api_router.include_router(mlflow_router) -``` -- `services/`: Business logic/orchestration and integrations - - Orchestration: `execution_service.py`, `crewai_execution_service.py`, `process_crew_executor.py`, `scheduler_service.py` - - Integrations: `databricks_*_service.py`, `mlflow_service.py` - - Observability/aux: `execution_logs_service.py`, `execution_trace_service.py`, `documentation_embedding_service.py` -- `repositories/`: Data access (SQL/external/vector/mlflow) - - Examples: `execution_repository.py`, `execution_history_repository.py`, `databricks_vector_index_repository.py`, `database_backup_repository.py` -- `models/`: SQLAlchemy ORM entities (tables) -- `schemas/`: Pydantic request/response models -- `db/`: DB setup and sessions - - `session.py`: async engine/session, SQLite lock retries, optional SQL logging - - `all_models.py`: model import aggregator -- `config/`: App configuration - - `settings.py`: env-driven settings and flags (e.g., `DOCS_ENABLED`, `AUTO_SEED_DATABASE`) - - `logging.py`: logging configuration -- `core/`: Cross-cutting concerns - - `llm_manager.py`: LLM routing/provider config - - `logger.py`: centralized logging manager - - `permissions.py`, `dependencies.py`, `base_service.py`, `base_repository.py`, `unit_of_work.py` - - `entity_extraction_fallback.py`: LLM memory fallback logic -- `engines/`: AI orchestration engines - - `crewai/`: `execution_runner.py`, `crew_preparation.py`, `flow_preparation.py`, callbacks, memory, tools, guardrails - - `engine_factory.py`: engine selection -- `utils/`: Utilities - - `user_context.py`: Databricks header parsing, group context, middleware - - `databricks_url_utils.py`, `databricks_auth.py`, rate limiting, crypto, prompts -- `seeds/`, `scripts/`, `dependencies/`: Seeders, scripts, DI helpers - -### Frontend (`src/frontend`) -- React + TypeScript (CRA + Craco) -- `src/config/api/ApiConfig.ts`: API base URL selection and Axios client -```1:16:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -import axios from 'axios'; - -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), -}; - -export const apiClient = axios.create({ - baseURL: config.apiUrl, - headers: { - 'Content-Type': 'application/json', - }, -}); + +├── README.md +└── src/ + ├── backend/ # FastAPI backend + ├── frontend/ # React + TypeScript frontend + ├── docs/ # Markdown docs (copied to /docs in the app) + ├── build.py # Frontend build + docs copy + ├── deploy.py # Deployment utilities + └── manifest.yaml # App metadata ``` -- `src/api/*Service.ts`: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) -- `src/components/`, `src/app/`, `src/store/`, `src/types/`, `src/utils/`, `src/hooks/`, `src/theme/` -- `public/` assets; `craco.config.js`, `tsconfig.json` \ No newline at end of file + +### Backend (src/backend/src) +- main.py: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, API router registration +- api/: HTTP routers per domain (agents, crews, executions, tools, models, engine-config, etc.) +- services/: Business logic & orchestration + - Orchestration: execution_service.py, crewai_execution_service.py, process_crew_executor.py, scheduler_service.py + - Integrations: databricks_*_service.py, mlflow_service.py + - Observability: execution_logs_service.py, execution_trace_service.py, documentation_embedding_service.py +- repositories/: Data access (SQL/external/vector/mlflow) +- models/: SQLAlchemy entities +- schemas/: Pydantic request/response DTOs +- db/: Sessions and Alembic integration (session.py, all_models.py) +- config/: Settings & logging (settings.py, logging.py) +- core/: Cross-cutting utilities (llm_manager.py, logger.py, permissions.py, unit_of_work.py) +- engines/: AI engine integration (CrewAI prep/runner, memory, tools, guardrails) +- utils/: Helpers (user_context.py, databricks_url_utils.py, etc.) +- seeds/, scripts/, dependencies/: Seeders, scripts, DI helpers + +### Frontend (src/frontend) +- CRA + TypeScript app +- src/config/api/ApiConfig.ts: API base URL selection and Axios client +- src/api/*Service.ts: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) +- src/components/: UI components & views +- src/store/: State management +- src/hooks/: Reusable logic +- src/types/: Shared TS types +- src/utils/, src/theme/: Utilities and theme +- public/: Static assets (docs copied to /docs here) + +### Key entry points +- Backend app starts in main.py (includes api_router with prefix from settings) +- Frontend docs are served from /docs (markdown files copied there at build) + +### Routers (where to look) +Common examples under src/backend/src/api/: +- agents_router.py, crews_router.py, executions_router.py, execution_logs_router.py, execution_trace_router.py +- engine_config_router.py, models_router.py, tools_router.py, schemas_router.py +- databricks_*_router.py (secrets, knowledge, connection) + +### Database & migrations +- DB sessions configured in src/backend/src/db/session.py +- Alembic configuration via alembic.ini (root) and migrations/ (root) +- Models aggregated in src/backend/src/db/all_models.py + +### Configuration & logging +- src/backend/src/config/settings.py: env-driven settings (CORS, DB URIs, docs flags, seeding) +- src/backend/src/config/logging.py and src/backend/src/core/logger.py: centralized logging + +### Core & engines +- src/backend/src/core/llm_manager.py: provider/model selection, streaming options +- src/backend/src/engines/crewai/*: crew preparation, execution runner, callbacks, memory/tool adapters + +### Tips: how to trace a feature +1) Start at the router file for the endpoint. +2) Open the called service and scan business logic. +3) Inspect repository methods and related models. +4) Check Pydantic schemas for request/response contracts. +5) Search for engine usage under engines/crewai if orchestration is involved. + +--- + +## Development conventions (back end) + +### Layering philosophy +- API (FastAPI routers) → Services (business logic) → Repositories (data access) → DB +- Keep routers thin (validation, auth), services cohesive (orchestration/transactions), repositories I/O-only. + +### Naming & structure +- Routers: _router.py (e.g., agents_router.py) +- Services: _service.py +- Repositories: _repository.py +- Models: singular file per entity (e.g., agent.py) +- Schemas: mirror model names (e.g., schemas/agent.py) + +### Request/response contracts +- Define input/output Pydantic models in src/backend/src/schemas/* +- Routers return explicit response models (response_model=...) where practical +- Prefer DTOs over ORM entities at boundaries + +### Transactions & Unit of Work +- Encapsulate write operations inside service methods +- Use the UnitOfWork pattern for multi-repository transactions (core/unit_of_work.py) +- Repositories should not commit; services decide transactional scope + +### Error handling +- Raise HTTPException in routers for user input errors +- Services raise domain errors; routers translate to HTTP +- Avoid broad try/except; log and rethrow with context + +--- + +## Back end deep‑dive (files that matter) + +### App bootstrap +- main.py: lifespan init (logging, DB init, seeders), CORS, user context middleware, include api_router +- config/settings.py: environment-driven settings (DB URIs, docs toggles, seeding) + +### API surface (selected) +- api/__init__.py: composes all routers +- api/executions_router.py: start/stop/get execution +- api/execution_logs_router.py, api/execution_trace_router.py: logs and trace endpoints +- api/engine_config_router.py, api/models_router.py, api/tools_router.py: engine and model config, tool registry + +### Services (selected) +- services/execution_service.py: high-level execution orchestration +- services/crewai_execution_service.py, engines/crewai/execution_runner.py: CrewAI integration points +- services/scheduler_service.py: background scheduling +- services/documentation_embedding_service.py: embeddings for better generation + +### Repositories (selected) +- repositories/execution_repository.py, execution_history_repository.py: persistence for runs +- repositories/databricks_*_repository.py: Databricks secrets, vector index, volumes + +### Database & sessions +- db/session.py: async engine/session, SQLite lock retries, SQL_DEBUG logging +- db/all_models.py: imports all models for Alembic + +### Observability +- core/logger.py: central logger manager (writes to LOG_DIR) +- services/execution_logs_service.py, execution_trace_service.py: persisted logs/trace + +--- + +## Configuration quick reference + +Defined in src/backend/src/config/settings.py: +- DATABASE_TYPE=postgres|sqlite (defaults to postgres) +- SQLITE_DB_PATH=./app.db when using SQLite +- POSTGRES_* envs for Postgres connection +- DOCS_ENABLED=true|false (exposes /api-docs, /api-redoc, /api-openapi.json) +- AUTO_SEED_DATABASE=true|false (background seeding after DB init) +- LOG_LEVEL=INFO|DEBUG +- SQL_DEBUG=true|false (emits SQL to logs for troubleshooting) + +Notes: +- USE_NULLPOOL is set early in main.py to avoid asyncpg pool issues +- Logs default under src/backend/src/logs/ + +--- + +## Engines & orchestration + +- Engine selection: src/backend/src/engines/engine_factory.py +- CrewAI integration lives under src/backend/src/engines/crewai/ + - crew_preparation.py: build agents, tools, memory for a run + - execution_runner.py: run loop, callbacks/guardrails + - trace_management.py: hook into tracing pipeline + +Memory/model caveat: +- Known limitation for specific Databricks models (Claude / GPT‑OSS) on entity extraction +- Automatic fallback to databricks-llama-4-maverick for memory entity extraction only + +--- + +## Front end deep‑dive + +### Docs viewer (this page) +- Markdown fetched from /docs/.md (copied from src/docs at build) +- Mermaid supported via fenced ```mermaid code blocks +- Images rendered responsively; prefer /docs/images/... or relative ./images/... +- Internal markdown links are intercepted to load other docs in‑app + +### API client +- src/frontend/src/config/api/ApiConfig.ts determines API base URL +- Default dev: http://localhost:8000/api/v1; override with REACT_APP_API_URL + +### UI organization +- src/components/: feature folders and shared components +- src/api/: type‑safe client wrappers by domain +- src/store/, src/hooks/, src/utils/, src/theme/ + +--- + +## End‑to‑end example (from API call to DB) + +1) Router (executions_router.py) accepts POST /executions with schema +2) Service (execution_service.py) validates logic, kicks off orchestration +3) Engine (engines/crewai/...) prepares crew and runs execution +4) Logs/Traces recorded via services and repositories +5) Repositories (execution_repository.py) persist status/history +6) Client polls GET /executions/{id} and GET /execution-logs/{id} + +--- + +## Anti‑patterns to avoid +- Business logic in routers (keep slim and delegate) +- Services directly returning ORM entities (use schemas/DTOs) +- Repositories committing transactions (services own commit/rollback) +- Ad‑hoc logging without the central logger (use core/logger.py) + +--- + +## Quick links +- Back end entrypoint: src/backend/src/main.py +- Compose routers: src/backend/src/api/__init__.py +- Settings: src/backend/src/config/settings.py +- Sessions: src/backend/src/db/session.py +- CrewAI runner: src/backend/src/engines/crewai/execution_runner.py diff --git a/src/frontend/public/docs/DEVELOPER_GUIDE.md b/src/frontend/public/docs/DEVELOPER_GUIDE.md index 093f54cf..e7c004fb 100644 --- a/src/frontend/public/docs/DEVELOPER_GUIDE.md +++ b/src/frontend/public/docs/DEVELOPER_GUIDE.md @@ -1,168 +1,145 @@ ## Developer Guide +Build, extend, and debug Kasal efficiently. This guide focuses on day-to-day workflows. + +### Components you'll touch +- **Frontend (React SPA)**: UI, designer, monitoring +- **API (FastAPI)**: REST endpoints and validation +- **Services**: Orchestration and business logic +- **Repositories**: DB and external I/O (Databricks, Vector, MLflow) +- **Engines (CrewAI)**: Agent/flow preparation and execution +- **Data & Storage**: SQLAlchemy models/sessions, embeddings, volumes ### Requirements +Tools and versions you need before running the stack. - Python 3.9+ - Node.js 18+ - Postgres (recommended) or SQLite for local dev - Databricks access if exercising Databricks features - -### Quick start -```bash -# Backend -cd src/backend -python -m venv .venv && source .venv/bin/activate -pip install -r ../requirements.txt -./run.sh # http://localhost:8000 (OpenAPI at /api-docs if enabled) - -# Frontend -cd ../frontend -npm install -npm start # http://localhost:3000 -``` - -Health check: -```bash -curl http://localhost:8000/health -# {"status":"healthy"} +## Developer Architecture Overview +This section gives developers a high-level view of the front end and back end. It explains core components and shows how to understand and trace them. + +## Backend Architecture +The backend uses FastAPI with a clean layered structure. It separates HTTP routing, business services, repositories, and SQLAlchemy models. + +### Core Components +- API Routers: src/backend/src/api/* map HTTP endpoints to service calls. +- Services: src/backend/src/services/* implement business logic and transactions. +- Repositories: src/backend/src/repositories/* handle database CRUD. +- Models/Schemas: src/backend/src/models/* and src/backend/src/schemas/* define persistence and I/O contracts. +- Core/Engines: src/backend/src/core/* and src/backend/src/engines/* integrate LLMs and execution flows. +- DB/Session: src/backend/src/db/* configures sessions and Alembic migrations. +- Config/Security: src/backend/src/config/* and src/backend/src/dependencies/* provide settings and auth. + +### Typical Request Flow +A request passes through router, service, repository, and database. LLM calls route through the LLM manager when needed. + +```mermaid +sequenceDiagram + participant C as Client + participant R as FastAPI Router + participant S as Service + participant U as UnitOfWork + participant Repo as Repository + participant DB as Database + participant L as LLM Manager + + C->>R: HTTP Request + R->>S: Validate and delegate + S->>U: Begin transaction + S->>Repo: Query/Command + Repo->>DB: SQLAlchemy operation + DB-->>Repo: Rows/Status + alt Needs LLM + S->>L: Generate/Score + L-->>S: LLM Result + end + S->>U: Commit + S-->>R: DTO/Schema + R-->>C: HTTP Response ``` -### Configuration -Backend settings: `src/backend/src/config/settings.py` -- Core: `DEBUG_MODE`, `LOG_LEVEL`, `DOCS_ENABLED`, `AUTO_SEED_DATABASE` -- Database: - - `DATABASE_TYPE=postgres|sqlite` (default: `postgres`) - - Postgres: `POSTGRES_SERVER`, `POSTGRES_PORT`, `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` - - SQLite: `SQLITE_DB_PATH=./app.db` - - SQL logging: `SQL_DEBUG=true|false` -- Notes: - - `USE_NULLPOOL` is set early in `main.py` to avoid asyncpg pool issues - - Logs written under `src/backend/src/logs/` - -Frontend API base URL (`REACT_APP_API_URL`) at build-time: -```bash -# Option A: dev default already points to http://localhost:8000/api/v1 -# Option B: override explicitly for a build (Unix/macOS) -REACT_APP_API_URL="http://localhost:8000/api/v1" npm run build - -# When using the top-level build script: -# The env var will propagate into the "npm run build" it runs -cd src -REACT_APP_API_URL="http://localhost:8000/api/v1" python build.py +### How to Understand Backend Components +- Start at the router file for the endpoint path. +- Open the service it calls and read business logic. +- Inspect repository methods and referenced models. +- Check schema types for request and response contracts. +- Review unit tests under `src/backend/tests` for examples. + +### Example: Minimal Endpoint Wiring +This shows a typical router, service, and repository connection. + +```python +# router.py +@router.get("/items/{item_id}", response_model=ItemOut) +async def get_item(item_id: UUID, service: ItemService = Depends(...)): + return await service.get_item(item_id) + +# services/item_service.py +class ItemService: + async def get_item(self, item_id: UUID) -> ItemOut: + with self.uow as uow: + item = uow.items.get(item_id) + return ItemOut.model_validate(item) + +# repositories/item_repository.py +class ItemRepository(BaseRepository[Item]): + ... ``` -### Conventions -- Routers (`api/*`): Validate with `schemas/*`, delegate to `services/*` -- Services: Business logic only; use repositories for I/O -- Repositories: All SQL/external I/O; don’t leak ORM to services -- Models: SQLAlchemy in `models/*`; Schemas: Pydantic in `schemas/*` - -### Add a new API resource (“widgets” example) -1) Model: `models/widget.py`; import in `db/all_models.py` -2) Schemas: `schemas/widget.py` (Create/Update/Read DTOs) -3) Repository: `repositories/widget_repository.py` -4) Service: `services/widget_service.py` -5) Router: `api/widgets_router.py` (validate → call service) -6) Register router in `api/__init__.py` -7) Frontend: add `src/frontend/src/api/widgets.ts` + components/views/state -8) Tests: `src/backend/tests/` - -### Add a new CrewAI tool -- Implement under `engines/crewai/tools/` (follow existing patterns) -- Expose configuration via service/router if user-configurable -- Ensure discovery/registration in the execution path (e.g., prep or service) - -### Executions and tracing -- Start executions via `executions_router.py` endpoints -- Services invoke engine flow (`engines/crewai/*`) -- Logs/traces: - - Execution logs via `execution_logs_*` - - Traces via `execution_trace_*` -```bash -# Kick off an execution -curl -X POST http://localhost:8000/api/v1/executions -H "Content-Type: application/json" -d '{...}' - -# Get execution status -curl http://localhost:8000/api/v1/executions/ - -# Fetch logs/trace -curl http://localhost:8000/api/v1/execution-logs/ -curl http://localhost:8000/api/v1/execution-trace/ +## Frontend Architecture +The frontend is a React + TypeScript application. It organizes UI components, API clients, state stores, hooks, and utilities. + +### Core Components +- API Clients: src/frontend/src/api/* wrap HTTP calls with typed methods. +- UI Components: src/frontend/src/components/* render views and dialogs. +- Hooks: src/frontend/src/hooks/* encapsulate logic and side effects. +- Stores: src/frontend/src/store/* manage app and workflow state. +- Types/Config: src/frontend/src/types/* and src/frontend/src/config/* provide typing and environment. +- Utils: src/frontend/src/utils/* offer reusable helpers. + +### UI Data Flow +Components call hooks, which use stores and API clients. Responses update state and re-render the UI. + +```mermaid +flowchart LR + A[UI Component] --> B[Hook] + B --> C[State Store] + B --> D[API Client] + D -->|HTTP| E[Backend API] + E -->|JSON| D + D --> C + C --> A ``` -### Background processing -- Scheduler: starts on DB-ready startup (`scheduler_service.py`) -- Embedding queue (SQLite): `embedding_queue_service.py` batches writes -- Cleanup on startup/shutdown: `execution_cleanup_service.py` - -### Database & migrations -- SQLite for quick local dev (`DATABASE_TYPE=sqlite`), Postgres for multi-user -- Alembic: -```bash -# after model changes -alembic revision --autogenerate -m "add widgets" -alembic upgrade head -``` - -### Auth, identity, tenancy -- Databricks headers parsed by `utils/user_context.py` -- Group-aware tenants; selected group passed in `group_id` header -- JWT/basic auth in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` - -### Logging & debugging -- App logs: `src/backend/src/logs/` (managed by `core/logger.py`) -- Verbose SQL: `export SQL_DEBUG=true` -- SQLite “database is locked”: mitigated via retry/backoff; reduce writers or use Postgres - -### Frontend notes -- Axios client and base URL: -```1:13:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), +### How to Understand Frontend Components +- Locate the component rendering the feature. +- Check its hook usage and props. +- Open the API client method it calls. +- Review the store slice it reads or writes. +- Inspect related types in src/frontend/src/types. + +### Example: Calling an API From a Component +A component loads items using an API service and updates local state. + +```ts +// api/ItemService.ts +export async function getItem(id: string) { + const res = await fetch(`/v1/items/${id}`, { headers: { Authorization: `Bearer ${token}` } }); + if (!res.ok) throw new Error("Failed"); + return (await res.json()) as Item; +} + +// components/Items/ItemView.tsx +const ItemView: React.FC<{ id: string }> = ({ id }) => { + const [item, setItem] = useState(null); + useEffect(() => { getItem(id).then(setItem); }, [id]); + return item ?
{item.name}
: Loading...; }; -export const apiClient = axios.create({ baseURL: config.apiUrl, headers: { 'Content-Type': 'application/json' } }); -``` - -### Testing -```bash -# Backend -python run_tests.py - -# Frontend -cd src/frontend -npm test ``` -### Production checklist -- Use Postgres (or managed DB), not SQLite -- Harden secrets/tokens; externalize (e.g., Databricks Secrets/Vault) -- Enforce TLS and CORS -- Monitor logs/traces; set alerts -- Review `DOCS_ENABLED`, `LOG_LEVEL`, `DEBUG_MODE` - -## Resources - -### Quick Links -- [API Playground](/api/docs) -- [Video Tutorials](https://kasal.ai/videos) -- [Discord Community](https://discord.gg/kasal) -- [Report Issues](https://github.com/kasal/issues) - -### Code Examples -- [Basic Agent Setup](https://github.com/kasal/examples/basic) -- [Multi-Agent Collaboration](https://github.com/kasal/examples/multi-agent) -- [Custom Tools](https://github.com/kasal/examples/tools) -- [Production Deployment](https://github.com/kasal/examples/deploy) - -### Support -- **Chat**: Available in-app 24/7 -- **Email**: dev@kasal.ai -- **Slack**: #kasal-developers - ---- +## End-to-End flow +This ties front end and back end with shared contracts. It helps new developers trace a feature quickly. -*Build smarter, ship faster with Kasal* \ No newline at end of file +- Frontend Component → Hook → Store/API Client → Backend Router → Service → Repository → DB. +- Shared types and response shapes live in frontend types and backend schemas. +- Tests in src/backend/tests and frontend __tests__ show usage patterns. \ No newline at end of file diff --git a/src/frontend/public/docs/README.md b/src/frontend/public/docs/README.md index 515063a6..f6cb427d 100644 --- a/src/frontend/public/docs/README.md +++ b/src/frontend/public/docs/README.md @@ -1,31 +1,19 @@ -![Kasal Logo](././/images/logo.png) -# 🟦 Kasal Documentation Hub + +# Kasal Documentation Hub **Enterprise AI Agent Orchestration Platform** --- ## Quick Start +Find the right documentation for your role and get productive fast. Choose your documentation based on your role: -## Business Documentation - -### [Business User Guide](./BUSINESS_USER_GUIDE.md) -**For Leaders, Managers, and Business Users** - -Get started with Kasal without any technical knowledge. Learn how to create AI workflows, calculate ROI, and manage your AI workforce. - -**Key Topics:** -- ROI Calculator and Cost Analysis -- Success Stories and Case Studies -- 15-Minute Quick Setup Guide -- Managing AI Workflows Without Code - ---- ## Technical Documentation +Developer-focused guides, code structure, and API reference. ### [Code structure Guide](./CODE_STRUCTURE_GUIDE.md) **For Software Engineers and Developers** @@ -62,6 +50,7 @@ Complete REST API documentation with examples, WebSocket events, and SDK librari --- ## Architecture Documentation +System design, patterns, and enterprise integration guidance. ### [Solution Architecture Guide](./ARCHITECTURE_GUIDE.md) **For Solution Architects and Technical Leaders** @@ -77,6 +66,7 @@ Understand Kasal's architecture, design patterns, and enterprise integration str --- ## Getting Started in 30 Seconds +One command sequence to run both backend and frontend locally. ```bash # Clone the repository @@ -97,6 +87,7 @@ Access the application at `http://localhost:3000` --- ## Key Features +What Kasal provides out of the box to build, operate, and govern AI workflows. | Feature | Description | |---------|-------------| @@ -110,11 +101,11 @@ Access the application at `http://localhost:3000` --- ## Documentation Structure +How this folder is organized and where to find topics. ``` docs/ ├── README.md # This file - Documentation hub -├── BUSINESS_USER_GUIDE.md # Business user documentation ├── CODE_STRUCTURE_GUIDE.md # Code structure documentation ├── DEVELOPER_GUIDE.md # Developer documentation ├── ARCHITECTURE_GUIDE.md # Architecture documentation @@ -128,6 +119,7 @@ docs/ --- ## Technology Stack +Core frameworks and platforms used across the project. - **Backend**: FastAPI, SQLAlchemy 2.0, Python 3.9+ - **Frontend**: React 18, TypeScript, Material-UI @@ -138,6 +130,7 @@ docs/ --- ## Support and Resources +Where to get help and how to contribute. ### Getting Help @@ -156,9 +149,10 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](https: --- ## Version Information +Project and documentation version details. - **Current Version**: 2.0.0 -- **Documentation Updated**: January 2025 +- **Documentation Updated**: October 2025 - **Minimum Python Version**: 3.9 - **Minimum Node Version**: 16.0 diff --git a/src/frontend/public/docs/WHY_KASAL.md b/src/frontend/public/docs/WHY_KASAL.md index 49088752..15c0c095 100644 --- a/src/frontend/public/docs/WHY_KASAL.md +++ b/src/frontend/public/docs/WHY_KASAL.md @@ -22,6 +22,49 @@ Transform your Databricks environment into an AI orchestration powerhouse. Build - **Extensible AI Engine**: CrewAI integration today, engine abstraction for future engines. - **Governance & Security**: Group-aware multi-tenant model, API keys, permissions, and auditability. +## Components at a glance +A quick tour of the building blocks—what each part does and why it matters. + +### Frontend (React SPA) +- **What it does**: Visual designer for agents, tasks, and flows; live monitoring UI. +- **Why it matters**: Non-technical users can build and operate AI workflows without touching code. + +### API (FastAPI) +- **What it does**: Validates requests, exposes REST endpoints, and routes calls to services. +- **Why it matters**: Clear, versioned contracts between UI/automation and backend logic. + +### Services (Business Logic) +- **What it does**: Implements orchestration, validation, scheduling, and domain logic. +- **Why it matters**: Keeps HTTP thin and domain logic testable and reusable. + +### Repositories (Data Access) +- **What it does**: Encapsulates SQL and external I/O (Databricks APIs, Vector Search, MLflow). +- **Why it matters**: Swappable persistence and integrations without leaking into services. + +### Engines (CrewAI Orchestration) +- **What it does**: Prepares crews, runs executions, handles callbacks/guardrails, manages memory. +- **Why it matters**: Pluggable execution engine today (CrewAI) and extensible for future engines. + +### Data & Storage +- **What it does**: Async SQLAlchemy sessions, models/schemas, vector indexes, volumes. +- **Why it matters**: Reliable persistence with optional vector search and document storage. + +### Scheduler & Background Jobs +- **What it does**: Recurring runs, long tasks, and background queues (e.g., embedding batching). +- **Why it matters**: Production-ready operations beyond single request/response. + +### Observability +- **What it does**: Structured logs, execution logs, traces, history, health checks. +- **Why it matters**: Debug fast, audit runs, and understand system behavior end-to-end. + +### Security & Governance +- **What it does**: Group-aware multi-tenancy, JWT/Databricks headers, centralized permissions. +- **Why it matters**: Safely share across teams while isolating data and enforcing roles. + +### Databricks Integrations +- **What it does**: OAuth, Secrets, SQL Warehouses, Unity Catalog, Volumes, Vector Search, MLflow. +- **Why it matters**: Build where your data and models already live with first-class support. + --- ## What Makes Kasal Different diff --git a/src/frontend/src/components/Documentation/Documentation.tsx b/src/frontend/src/components/Documentation/Documentation.tsx index a3e186f0..141ddf56 100644 --- a/src/frontend/src/components/Documentation/Documentation.tsx +++ b/src/frontend/src/components/Documentation/Documentation.tsx @@ -19,12 +19,12 @@ import { ExpandLess, ExpandMore, Menu as MenuIcon, - GitHub as GitHubIcon, Home as HomeIcon } from '@mui/icons-material'; import ReactMarkdown from 'react-markdown'; import remarkGfm from 'remark-gfm'; import { useNavigate } from 'react-router-dom'; +import mermaid from 'mermaid'; interface DocSection { label: string; @@ -32,16 +32,11 @@ interface DocSection { } const docSections: DocSection[] = [ + { - label: 'Overview', - items: [ - { label: 'Documentation Hub', file: 'README' }, - ], - }, - { - label: 'Business', + label: 'Architecture', items: [ - { label: 'Business User Guide', file: 'BUSINESS_USER_GUIDE' }, + { label: 'Solution Architecture', file: 'ARCHITECTURE_GUIDE' }, ], }, { @@ -49,15 +44,11 @@ const docSections: DocSection[] = [ items: [ { label: 'Code Structure Guide', file: 'CODE_STRUCTURE_GUIDE' }, { label: 'Developer Guide', file: 'DEVELOPER_GUIDE' }, + { label: 'Code structure', file: 'CODE_STRUCTURE_GUIDE' }, { label: 'API Reference', file: 'API_REFERENCE' }, ], }, - { - label: 'Architecture', - items: [ - { label: 'Solution Architecture', file: 'ARCHITECTURE_GUIDE' }, - ], - }, + ]; const Documentation: React.FC = () => { @@ -77,6 +68,21 @@ const Documentation: React.FC = () => { loadDocument(currentDoc); }, [currentDoc]); + // Initialize Mermaid diagrams whenever the markdown content changes + useEffect(() => { + if (!loading && docContent) { + try { + mermaid.initialize({ startOnLoad: false, securityLevel: 'loose' }); + // Run after the next paint to ensure markdown is in the DOM + window.requestAnimationFrame(() => { + mermaid.run(); + }); + } catch (e) { + // no-op: if mermaid fails, the raw code block will still be shown + } + } + }, [docContent, loading]); + const loadDocument = async (filename: string) => { setLoading(true); try { @@ -179,14 +185,6 @@ const Documentation: React.FC = () => { navigate('/workflow')}> - - - @@ -260,30 +258,129 @@ const Documentation: React.FC = () => { {children} ), - code: ({ children, ...props }: { children?: React.ReactNode; inline?: boolean }) => ( - props.inline ? ( + img: (props) => ( + + ), + a: ({ href, children, ...props }: { href?: string; children?: React.ReactNode }) => { + if (!href) return {children}; + + // Intra-page anchors (e.g., #section) + if (href.startsWith('#')) { + return ( + + {children} + + ); + } + + const isExternal = /^https?:\/\//i.test(href); + const isMd = href.toLowerCase().endsWith('.md'); + + // Normalize internal doc links to our loader (expects filename without .md) + if (!isExternal && isMd) { + // Strip common prefixes like /docs/ or src/docs/ + const normalized = href + .replace(/^\/?src\/docs\//i, '') + .replace(/^\/?docs\//i, ''); + const base = (normalized.split('/').pop() || normalized).replace(/\.md$/i, ''); + return ( + { + e.preventDefault(); + handleDocSelect(base); + }} + sx={{ + color: theme.palette.primary.main, + textDecoration: 'underline', + cursor: 'pointer', + '&:hover': { color: theme.palette.primary.dark }, + }} + > + {children} + + ); + } + + // External links open in new tab + return ( {children} - ) : ( + ); + }, + code: ({ children, inline, className, ...props }: { children?: React.ReactNode; inline?: boolean; className?: string }) => { + const languageMatch = /language-(\w+)/.exec(className || ''); + const language = languageMatch ? languageMatch[1] : undefined; + + // Render Mermaid diagrams as
blocks + if (!inline && language === 'mermaid') { + const diagram = String(children || '').replace(/\n$/, ''); + return ( + + {diagram} + + ); + } + + // Default code rendering (inline vs block) + if (inline) { + return ( + + {children} + + ); + } + + return ( { > {children} - ) - ), + ); + }, }} > {docContent} diff --git a/src/frontend_static/Kasal-overview.png b/src/frontend_static/Kasal-overview.png new file mode 100644 index 00000000..df9aa78e Binary files /dev/null and b/src/frontend_static/Kasal-overview.png differ diff --git a/src/frontend_static/docs/API_REFERENCE.md b/src/frontend_static/docs/API_REFERENCE.md index 7fbdc73f..7b9a6b4f 100644 --- a/src/frontend_static/docs/API_REFERENCE.md +++ b/src/frontend_static/docs/API_REFERENCE.md @@ -5,6 +5,7 @@ --- ## Getting Started +Base URLs, authentication, and rate limits you need before calling endpoints. ### Base URL ``` @@ -35,6 +36,7 @@ curl -X GET https://api.example.com/v1/crews \ --- ## Authentication Endpoints +Login, refresh, and logout flows to manage tokens. ### POST /auth/login **Login with credentials** @@ -77,6 +79,7 @@ Response: 204 No Content --- ## Crew Management +Create and manage multi-agent crews and their configurations. ### GET /crews **List all crews** @@ -160,6 +163,7 @@ Response: 204 No Content --- ## Agent Management +Create and list individual agents with roles, models, and tools. ### GET /agents **List all agents** @@ -201,6 +205,7 @@ Response: 201 Created --- ## Execution Management +Start executions, get status, retrieve traces, and stop runs. ### POST /executions **Start crew execution** @@ -264,6 +269,7 @@ Response: 200 OK --- ## Task Management +Create and list tasks assigned to agents. ### GET /tasks **List tasks** @@ -302,6 +308,7 @@ Response: 201 Created --- ## Tool Management +Discover built-in tools and register custom tools. ### GET /tools **List available tools** @@ -344,6 +351,7 @@ Response: 201 Created --- ## Memory Management +Fetch and clear short/long-term memory for a crew. ### GET /memory/{crew_id} **Get crew memory** @@ -378,40 +386,10 @@ Response: 204 No Content --- -## Analytics -### GET /analytics/usage -**Get usage statistics** -```json -Response: 200 OK -{ - "period": "2024-01", - "executions": 1543, - "tokens_used": 2450000, - "success_rate": 0.98, - "avg_duration": 245 -} -``` -### GET /analytics/performance -**Get performance metrics** -```json -Response: 200 OK -{ - "crews": [ - { - "crew_id": "crew_abc123", - "executions": 500, - "success_rate": 0.99, - "avg_duration": 180 - } - ] -} -``` - ---- - -## WebSocket Events +## 🔵 WebSocket Events +Real-time updates for task lifecycle, errors, and progress. ### Connection ```javascript @@ -454,7 +432,8 @@ ws.onopen = () => { --- -## Error Codes +## 🔷 Error Codes +Standardized error responses and meanings. | Code | Message | Description | |------|---------|-------------| @@ -482,7 +461,8 @@ ws.onopen = () => { --- -## Testing +## 🔹 Testing +Sandbox, Postman collection, and OpenAPI spec. ### Sandbox Environment ```bash @@ -493,55 +473,6 @@ curl -X POST https://sandbox-api.kasal.ai/v1/crews \ -d @crew.json ``` -### Postman Collection -[Download Collection](https://api.kasal.ai/postman/kasal-api-v1.json) - -### OpenAPI Spec -[View OpenAPI 3.0 Spec](https://api.kasal.ai/openapi.json) - ---- - -## SDKs & Libraries - -### Python -```python -pip install kasal-sdk - -from kasal import KasalClient - -client = KasalClient(api_key="your_key") -crew = client.crews.create(name="My Crew") -result = client.execute(crew.id, inputs={}) -``` - -### JavaScript/TypeScript -```javascript -npm install @kasal/sdk - -import { Kasal } from '@kasal/sdk'; - -const kasal = new Kasal({ apiKey: 'your_key' }); -const crew = await kasal.crews.create({ name: 'My Crew' }); -const result = await kasal.execute(crew.id, {}); -``` - -### Go -```go -import "github.com/kasal/kasal-go" - -client := kasal.NewClient("your_key") -crew, _ := client.Crews.Create("My Crew") -result, _ := client.Execute(crew.ID, inputs) -``` - ---- - -## API Support - -- **Status Page**: [status.kasal.ai](https://status.kasal.ai) -- **API Console**: [console.kasal.ai](https://console.kasal.ai) -- **Developer Forum**: [forum.kasal.ai](https://forum.kasal.ai) -- **Email**: api-support@kasal.ai --- diff --git a/src/frontend_static/docs/ARCHITECTURE_GUIDE.md b/src/frontend_static/docs/ARCHITECTURE_GUIDE.md index c2a54fdb..f1186d73 100644 --- a/src/frontend_static/docs/ARCHITECTURE_GUIDE.md +++ b/src/frontend_static/docs/ARCHITECTURE_GUIDE.md @@ -1,11 +1,12 @@ -# Kasal Solution Architecture +# 🔵 Kasal Solution Architecture > **Enterprise AI Orchestration Platform** - Scalable, secure, cloud-native -![Kasal Overvew](././/images/Kasal-overview.png) +![Kasal Overvew](/src/docs/images/Kasal-overview.png) *Visual workflow designer for creating AI agent collaborations* --- ## System Overview +What the platform aims to achieve and core design principles. ### Platform Vision **Transform business workflows with autonomous AI agents** - Zero infrastructure complexity @@ -22,6 +23,7 @@ --- ## High-Level Architecture +A big-picture view of the client, application, AI, and data layers. ```mermaid graph TB @@ -71,81 +73,87 @@ graph TB --- ## Architecture Pattern +The layered approach and how requests flow through components. ### High-level - Layered architecture: - Frontend (React SPA) → API (FastAPI) → Services → Repositories → Database - Async-first (async SQLAlchemy, background tasks, queues) -- Config via environment (`src/backend/src/config/settings.py`) -- Pluggable orchestration engine (`src/backend/src/engines/` with CrewAI) +- Config via environment (src/backend/src/config/settings.py) +- Pluggable orchestration engine (src/backend/src/engines/ with CrewAI) ### Request lifecycle (CRUD path) -1) Router in `api/` receives request, validates using `schemas/` -2) Router calls `services/` for business logic -3) Service uses `repositories/` for DB/external I/O -4) Data persisted via `db/session.py` +From HTTP request to response: validation, business logic, and persistence. +1) Router in api/ receives request, validates using schemas/ +2) Router calls services/ for business logic +3) Service uses repositories/ for DB/external I/O +4) Data persisted via db/session.py 5) Response serialized with Pydantic schemas ### Orchestration lifecycle (AI execution) -- Entry via `executions_router.py` → `execution_service.py` -- Service prepares agents/tools/memory and selects engine (`engines/engine_factory.py`) +How executions are prepared, run, and observed using the engine. +- Entry via executions_router.py → execution_service.py +- Service prepares agents/tools/memory and selects engine (engines/engine_factory.py) - CrewAI path: - - Prep: `engines/crewai/crew_preparation.py` and `flow_preparation.py` - - Run: `engines/crewai/execution_runner.py` with callbacks/guardrails - - Observability: `execution_logs_service.py`, `execution_trace_service.py` -- Persist status/history: `execution_repository.py`, `execution_history_repository.py` + - Prep: engines/crewai/crew_preparation.py and flow_preparation.py + - Run: engines/crewai/execution_runner.py with callbacks/guardrails + - Observability: execution_logs_service.py, execution_trace_service.py +- Persist status/history: execution_repository.py, execution_history_repository.py ### Background processing -- Scheduler at startup: `scheduler_service.py` -- Embedding queue (SQLite): `embedding_queue_service.py` (batches writes) -- Startup/shutdown cleanup: `execution_cleanup_service.py` +Schedulers and queues for recurring and long-running tasks. +- Scheduler at startup: scheduler_service.py +- Embedding queue (SQLite): embedding_queue_service.py (batches writes) +- Startup/shutdown cleanup: execution_cleanup_service.py ### Data modeling -- ORM in `models/*` mirrors `schemas/*` +ORM entities, Pydantic schemas, and repository boundaries. +- ORM in models/* mirrors schemas/* - Repositories encapsulate all SQL/external calls (Databricks APIs, Vector Search, MLflow) -- `db/session.py`: +- db/session.py: - Async engine and session factory - SQLite lock retry w/ backoff - - Optional SQL logging via `SQL_DEBUG=true` + - Optional SQL logging via SQL_DEBUG=true ### Auth, identity, and tenancy -- Databricks Apps headers parsed by `utils/user_context.py` +User context, group isolation, and authorization controls. +- Databricks Apps headers parsed by utils/user_context.py - Group-aware multi-tenant context propagated via middleware -- JWT/basic auth routes in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` +- JWT/basic auth routes in auth_router.py, users in users_router.py +- Authorization checks in core/permissions.py ### Security Controls +Defense-in-depth across network, API, data, secrets, and compliance. | Layer | Control | Implementation | |-------|---------|----------------| | **Network** | TLS 1.3 | End-to-end encryption | | **API** | OAuth 2.0 | Databricks SSO | | **Data** | AES-256 | Encryption at rest | -| **Secrets** | Vault | HashiCorp Vault | -| **Compliance** | SOC2 | Audit trails | --- ### Storage Strategy +Where different data types live and why. | Data Type | Storage | Purpose | |-----------|---------|---------| | **Transactional** | PostgreSQL | ACID compliance | -| **Session** | Redis | Fast cache | | **Vectors** | Databricks Vector | Semantic search | -| **Files** | S3/Azure Blob | Document storage | -| **Logs** | CloudWatch/Datadog | Observability | +| **Logs** | MLFlow traces | Observability | --- ### Observability -- Central log manager: `core/logger.py` (writes to `LOG_DIR`) -- API/SQL logging toggles (`LOG_LEVEL`, `SQL_DEBUG`) +Logs, traces, metrics, and how to access them. +- Central log manager: core/logger.py (writes to LOG_DIR) +- API/SQL logging toggles (LOG_LEVEL, SQL_DEBUG) - Execution logs/traces persisted and queryable via dedicated routes/services ### Configuration flags (selected) -- `DOCS_ENABLED`: enables `/api-docs`, `/api-redoc`, `/api-openapi.json` -- `AUTO_SEED_DATABASE`: async background seeders post DB init -- `DATABASE_TYPE`: `postgres` (default) or `sqlite` with `SQLITE_DB_PATH` +Important toggles that affect developer and runtime experience. +- DOCS_ENABLED: enables /docs +- AUTO_SEED_DATABASE: async background seeders post DB init +- DATABASE_TYPE: sqlite with SQLITE_DB_PATH diff --git a/src/frontend_static/docs/BUSINESS_USER_GUIDE.md b/src/frontend_static/docs/BUSINESS_USER_GUIDE.md deleted file mode 100644 index 6e1b5fc7..00000000 --- a/src/frontend_static/docs/BUSINESS_USER_GUIDE.md +++ /dev/null @@ -1,207 +0,0 @@ -# Kasal Business User Guide - -> **AI Workers for Your Business** - No coding required - ---- - -## What is Kasal? - -**Your AI workforce that never sleeps.** Kasal orchestrates AI agents to handle complex workflows automatically, 24/7. - -### Perfect For: -- **Customer Support** → 70% faster response times -- **Content Creation** → 5x more output -- **Data Analysis** → Real-time insights -- **Process Automation** → Zero human errors - -### ROI Calculator -| Your Current Process | With Kasal | Savings | -|---------------------|------------|---------| -| 10 hours/week manual work | 1 hour setup + monitoring | **9 hours/week** | -| $50/hour employee cost | $0.10/hour AI cost | **$2,500/month** | -| 3-day turnaround | Same-day results | **66% faster** | - ---- - -## Quick Start (15 Minutes) - -### Step 1: Choose Your Use Case -Click the template that matches your need: - -| **Customer Support** | **Content Marketing** | **Data Analysis** | -|-------------------------|-------------------------|---------------------| -| Auto-reply to emails | Generate blog posts | Analyze spreadsheets | -| Handle FAQs | Create social media | Build reports | -| Route to right team | SEO optimization | Find patterns | - -### Step 2: Configure Your AI Team -Simply describe what you want: -``` -"I need to respond to customer emails within 1 hour -with professional, helpful answers" -``` - -### Step 3: Connect Your Tools -One-click integrations: -- **Email**: Gmail, Outlook -- **Chat**: Slack, Teams -- **Data**: Excel, Google Sheets -- **CRM**: Salesforce, HubSpot - -### Step 4: Launch -Press **Start** and watch your AI team work! - ---- - -## Success Metrics - -### Real Customer Results - -**TechCorp** -> "Reduced support tickets by 60% in first month" -- **Before**: 200 tickets/day, 24hr response -- **After**: 80 tickets/day, 1hr response -- **Saved**: $120K annually - -**E-Shop Plus** -> "Generated 500 product descriptions in 2 hours" -- **Before**: 1 writer, 20 descriptions/day -- **After**: AI crew, 250 descriptions/hour -- **Result**: Launched 10x faster - -**DataDriven Inc** -> "Daily reports that took 3 hours now take 3 minutes" -- **Before**: Manual Excel analysis -- **After**: Automated insights -- **Impact**: Real-time decisions - ---- - -## Control Center - -### Dashboard Overview -![Dashboard](dashboard-preview.png) - -**Key Metrics at a Glance:** -- **Active Workflows**: See what's running -- **Completed Tasks**: Track progress -- **Performance**: Response times -- **Cost Savings**: ROI tracking - -### Managing Your AI Workers - -**Start/Stop Controls** -- **Run Continuously**: 24/7 automation -- **Business Hours**: 9-5 operation -- **Pause**: Instant stop - -**Quality Controls** -- **Review Mode**: Approve before sending -- **Auto Mode**: Fully autonomous -- **Alert Mode**: Notify on exceptions - ---- - -## Advanced Features - -### Multi-Agent Collaboration -Your AI workers can work as a team: - -**Marketing Campaign Example:** -1. **Researcher** → Finds trending topics -2. **Writer** → Creates content -3. **Designer** → Generates visuals -4. **Analyst** → Measures performance - -All working together, automatically. - -### Custom Workflows -Build complex processes visually: -- Drag & drop workflow designer -- No coding required -- Test before deploying - -### Integration Marketplace -- 100+ pre-built connectors -- API access to any system -- Custom integrations available - ---- - -## Common Questions - -
-How much does it cost? - -**Starter**: $99/month -- 5 AI agents -- 1,000 tasks/month -- Email support - -**Professional**: $499/month -- Unlimited agents -- 10,000 tasks/month -- Priority support - -**Enterprise**: Custom pricing -- Unlimited everything -- Dedicated support -- Custom integrations -
- -
-Is my data secure? - -**Enterprise Security** -- SOC2 Type II certified -- GDPR compliant -- End-to-end encryption -- Your data never trains AI models -
- -
-What if something goes wrong? - -**Built-in Safety** -- Automatic error handling -- Human escalation rules -- Full audit trail -- Instant rollback -
- -
-How fast can I get started? - -**Same Day Setup** -- 15-minute quick start -- Pre-built templates -- Free onboarding call -- 30-day money-back guarantee -
- ---- - -## Ready to Start? - -
- -### [Book a 15-Min Demo](https://kasal.ai/demo) -### [Calculate Your ROI](https://kasal.ai/calculator) -### [Start Free Trial](https://kasal.ai/trial) - -**No credit card required • Cancel anytime** - -
- ---- - -## Need Help? - -- **Chat**: Available 24/7 in-app -- **Email**: support@kasal.ai -- **Phone**: 1-800-KASAL-AI -- **Resources**: [Video Tutorials](https://kasal.ai/videos) - ---- - -*Kasal - Where AI Meets Business Excellence* \ No newline at end of file diff --git a/src/frontend_static/docs/CODE_STRUCTURE_GUIDE.md b/src/frontend_static/docs/CODE_STRUCTURE_GUIDE.md index 0c8232d9..ca4a44e7 100644 --- a/src/frontend_static/docs/CODE_STRUCTURE_GUIDE.md +++ b/src/frontend_static/docs/CODE_STRUCTURE_GUIDE.md @@ -1,104 +1,213 @@ ## Code Structure +A fast, skimmable map of the repository to help you find the right place quickly. -### Top-level -- `README.md`: Overview and documentation index -- `src/entrypoint.py`: App entry for Databricks Apps -- `src/build.py`: Frontend build pipeline and docs copy -- `src/deploy.py`: Deployment automation -- `src/manifest.yaml`, `src/app.yaml`: App metadata/config -- `src/docs/`: Documentation (guides, API reference, images) -- `src/frontend/`: React SPA -- `src/backend/`: FastAPI backend service - -### Backend (`src/backend/src`) -- `main.py`: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, seeds, API router registration -```273:301:/Users/anshu.roy/Documents/kasal/src/backend/src/main.py -# Initialize FastAPI app -app = FastAPI( - title=settings.PROJECT_NAME, - description=settings.PROJECT_DESCRIPTION, - version=settings.VERSION, - lifespan=lifespan, - docs_url="/api-docs" if settings.DOCS_ENABLED else None, - redoc_url="/api-redoc" if settings.DOCS_ENABLED else None, - openapi_url="/api-openapi.json" if settings.DOCS_ENABLED else None, - openapi_version="3.1.0" -) - -# Add user context middleware and include API -app.add_middleware(BaseHTTPMiddleware, dispatch=user_context_middleware) -app.include_router(api_router, prefix=settings.API_V1_STR) +### Repository layout ``` -- `api/`: HTTP route modules (per domain) - - Examples: `executions_router.py`, `execution_logs_router.py`, `execution_trace_router.py`, `engine_config_router.py`, `agents_router.py`, `crews_router.py`, `databricks_*_router.py`, `memory_backend_router.py`, `schemas_router.py`, `tools_router.py` -```48:66:/Users/anshu.roy/Documents/kasal/src/backend/src/api/__init__.py -# Create the main API router -api_router = APIRouter() - -# Include all the sub-routers -api_router.include_router(agents_router) -api_router.include_router(crews_router) -api_router.include_router(databricks_router) -api_router.include_router(databricks_knowledge_router) -api_router.include_router(flows_router) -api_router.include_router(healthcheck_router) -api_router.include_router(logs_router) -api_router.include_router(models_router) -api_router.include_router(databricks_secrets_router) -api_router.include_router(api_keys_router) -api_router.include_router(tasks_router) -api_router.include_router(templates_router) -api_router.include_router(group_tools_router) -api_router.include_router(mlflow_router) -``` -- `services/`: Business logic/orchestration and integrations - - Orchestration: `execution_service.py`, `crewai_execution_service.py`, `process_crew_executor.py`, `scheduler_service.py` - - Integrations: `databricks_*_service.py`, `mlflow_service.py` - - Observability/aux: `execution_logs_service.py`, `execution_trace_service.py`, `documentation_embedding_service.py` -- `repositories/`: Data access (SQL/external/vector/mlflow) - - Examples: `execution_repository.py`, `execution_history_repository.py`, `databricks_vector_index_repository.py`, `database_backup_repository.py` -- `models/`: SQLAlchemy ORM entities (tables) -- `schemas/`: Pydantic request/response models -- `db/`: DB setup and sessions - - `session.py`: async engine/session, SQLite lock retries, optional SQL logging - - `all_models.py`: model import aggregator -- `config/`: App configuration - - `settings.py`: env-driven settings and flags (e.g., `DOCS_ENABLED`, `AUTO_SEED_DATABASE`) - - `logging.py`: logging configuration -- `core/`: Cross-cutting concerns - - `llm_manager.py`: LLM routing/provider config - - `logger.py`: centralized logging manager - - `permissions.py`, `dependencies.py`, `base_service.py`, `base_repository.py`, `unit_of_work.py` - - `entity_extraction_fallback.py`: LLM memory fallback logic -- `engines/`: AI orchestration engines - - `crewai/`: `execution_runner.py`, `crew_preparation.py`, `flow_preparation.py`, callbacks, memory, tools, guardrails - - `engine_factory.py`: engine selection -- `utils/`: Utilities - - `user_context.py`: Databricks header parsing, group context, middleware - - `databricks_url_utils.py`, `databricks_auth.py`, rate limiting, crypto, prompts -- `seeds/`, `scripts/`, `dependencies/`: Seeders, scripts, DI helpers - -### Frontend (`src/frontend`) -- React + TypeScript (CRA + Craco) -- `src/config/api/ApiConfig.ts`: API base URL selection and Axios client -```1:16:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -import axios from 'axios'; - -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), -}; - -export const apiClient = axios.create({ - baseURL: config.apiUrl, - headers: { - 'Content-Type': 'application/json', - }, -}); + +├── README.md +└── src/ + ├── backend/ # FastAPI backend + ├── frontend/ # React + TypeScript frontend + ├── docs/ # Markdown docs (copied to /docs in the app) + ├── build.py # Frontend build + docs copy + ├── deploy.py # Deployment utilities + └── manifest.yaml # App metadata ``` -- `src/api/*Service.ts`: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) -- `src/components/`, `src/app/`, `src/store/`, `src/types/`, `src/utils/`, `src/hooks/`, `src/theme/` -- `public/` assets; `craco.config.js`, `tsconfig.json` \ No newline at end of file + +### Backend (src/backend/src) +- main.py: FastAPI app bootstrap, CORS, middleware, startup/shutdown, scheduler, API router registration +- api/: HTTP routers per domain (agents, crews, executions, tools, models, engine-config, etc.) +- services/: Business logic & orchestration + - Orchestration: execution_service.py, crewai_execution_service.py, process_crew_executor.py, scheduler_service.py + - Integrations: databricks_*_service.py, mlflow_service.py + - Observability: execution_logs_service.py, execution_trace_service.py, documentation_embedding_service.py +- repositories/: Data access (SQL/external/vector/mlflow) +- models/: SQLAlchemy entities +- schemas/: Pydantic request/response DTOs +- db/: Sessions and Alembic integration (session.py, all_models.py) +- config/: Settings & logging (settings.py, logging.py) +- core/: Cross-cutting utilities (llm_manager.py, logger.py, permissions.py, unit_of_work.py) +- engines/: AI engine integration (CrewAI prep/runner, memory, tools, guardrails) +- utils/: Helpers (user_context.py, databricks_url_utils.py, etc.) +- seeds/, scripts/, dependencies/: Seeders, scripts, DI helpers + +### Frontend (src/frontend) +- CRA + TypeScript app +- src/config/api/ApiConfig.ts: API base URL selection and Axios client +- src/api/*Service.ts: API clients per domain (Agents, Crews, Executions, Models, Tools, etc.) +- src/components/: UI components & views +- src/store/: State management +- src/hooks/: Reusable logic +- src/types/: Shared TS types +- src/utils/, src/theme/: Utilities and theme +- public/: Static assets (docs copied to /docs here) + +### Key entry points +- Backend app starts in main.py (includes api_router with prefix from settings) +- Frontend docs are served from /docs (markdown files copied there at build) + +### Routers (where to look) +Common examples under src/backend/src/api/: +- agents_router.py, crews_router.py, executions_router.py, execution_logs_router.py, execution_trace_router.py +- engine_config_router.py, models_router.py, tools_router.py, schemas_router.py +- databricks_*_router.py (secrets, knowledge, connection) + +### Database & migrations +- DB sessions configured in src/backend/src/db/session.py +- Alembic configuration via alembic.ini (root) and migrations/ (root) +- Models aggregated in src/backend/src/db/all_models.py + +### Configuration & logging +- src/backend/src/config/settings.py: env-driven settings (CORS, DB URIs, docs flags, seeding) +- src/backend/src/config/logging.py and src/backend/src/core/logger.py: centralized logging + +### Core & engines +- src/backend/src/core/llm_manager.py: provider/model selection, streaming options +- src/backend/src/engines/crewai/*: crew preparation, execution runner, callbacks, memory/tool adapters + +### Tips: how to trace a feature +1) Start at the router file for the endpoint. +2) Open the called service and scan business logic. +3) Inspect repository methods and related models. +4) Check Pydantic schemas for request/response contracts. +5) Search for engine usage under engines/crewai if orchestration is involved. + +--- + +## Development conventions (back end) + +### Layering philosophy +- API (FastAPI routers) → Services (business logic) → Repositories (data access) → DB +- Keep routers thin (validation, auth), services cohesive (orchestration/transactions), repositories I/O-only. + +### Naming & structure +- Routers: _router.py (e.g., agents_router.py) +- Services: _service.py +- Repositories: _repository.py +- Models: singular file per entity (e.g., agent.py) +- Schemas: mirror model names (e.g., schemas/agent.py) + +### Request/response contracts +- Define input/output Pydantic models in src/backend/src/schemas/* +- Routers return explicit response models (response_model=...) where practical +- Prefer DTOs over ORM entities at boundaries + +### Transactions & Unit of Work +- Encapsulate write operations inside service methods +- Use the UnitOfWork pattern for multi-repository transactions (core/unit_of_work.py) +- Repositories should not commit; services decide transactional scope + +### Error handling +- Raise HTTPException in routers for user input errors +- Services raise domain errors; routers translate to HTTP +- Avoid broad try/except; log and rethrow with context + +--- + +## Back end deep‑dive (files that matter) + +### App bootstrap +- main.py: lifespan init (logging, DB init, seeders), CORS, user context middleware, include api_router +- config/settings.py: environment-driven settings (DB URIs, docs toggles, seeding) + +### API surface (selected) +- api/__init__.py: composes all routers +- api/executions_router.py: start/stop/get execution +- api/execution_logs_router.py, api/execution_trace_router.py: logs and trace endpoints +- api/engine_config_router.py, api/models_router.py, api/tools_router.py: engine and model config, tool registry + +### Services (selected) +- services/execution_service.py: high-level execution orchestration +- services/crewai_execution_service.py, engines/crewai/execution_runner.py: CrewAI integration points +- services/scheduler_service.py: background scheduling +- services/documentation_embedding_service.py: embeddings for better generation + +### Repositories (selected) +- repositories/execution_repository.py, execution_history_repository.py: persistence for runs +- repositories/databricks_*_repository.py: Databricks secrets, vector index, volumes + +### Database & sessions +- db/session.py: async engine/session, SQLite lock retries, SQL_DEBUG logging +- db/all_models.py: imports all models for Alembic + +### Observability +- core/logger.py: central logger manager (writes to LOG_DIR) +- services/execution_logs_service.py, execution_trace_service.py: persisted logs/trace + +--- + +## Configuration quick reference + +Defined in src/backend/src/config/settings.py: +- DATABASE_TYPE=postgres|sqlite (defaults to postgres) +- SQLITE_DB_PATH=./app.db when using SQLite +- POSTGRES_* envs for Postgres connection +- DOCS_ENABLED=true|false (exposes /api-docs, /api-redoc, /api-openapi.json) +- AUTO_SEED_DATABASE=true|false (background seeding after DB init) +- LOG_LEVEL=INFO|DEBUG +- SQL_DEBUG=true|false (emits SQL to logs for troubleshooting) + +Notes: +- USE_NULLPOOL is set early in main.py to avoid asyncpg pool issues +- Logs default under src/backend/src/logs/ + +--- + +## Engines & orchestration + +- Engine selection: src/backend/src/engines/engine_factory.py +- CrewAI integration lives under src/backend/src/engines/crewai/ + - crew_preparation.py: build agents, tools, memory for a run + - execution_runner.py: run loop, callbacks/guardrails + - trace_management.py: hook into tracing pipeline + +Memory/model caveat: +- Known limitation for specific Databricks models (Claude / GPT‑OSS) on entity extraction +- Automatic fallback to databricks-llama-4-maverick for memory entity extraction only + +--- + +## Front end deep‑dive + +### Docs viewer (this page) +- Markdown fetched from /docs/.md (copied from src/docs at build) +- Mermaid supported via fenced ```mermaid code blocks +- Images rendered responsively; prefer /docs/images/... or relative ./images/... +- Internal markdown links are intercepted to load other docs in‑app + +### API client +- src/frontend/src/config/api/ApiConfig.ts determines API base URL +- Default dev: http://localhost:8000/api/v1; override with REACT_APP_API_URL + +### UI organization +- src/components/: feature folders and shared components +- src/api/: type‑safe client wrappers by domain +- src/store/, src/hooks/, src/utils/, src/theme/ + +--- + +## End‑to‑end example (from API call to DB) + +1) Router (executions_router.py) accepts POST /executions with schema +2) Service (execution_service.py) validates logic, kicks off orchestration +3) Engine (engines/crewai/...) prepares crew and runs execution +4) Logs/Traces recorded via services and repositories +5) Repositories (execution_repository.py) persist status/history +6) Client polls GET /executions/{id} and GET /execution-logs/{id} + +--- + +## Anti‑patterns to avoid +- Business logic in routers (keep slim and delegate) +- Services directly returning ORM entities (use schemas/DTOs) +- Repositories committing transactions (services own commit/rollback) +- Ad‑hoc logging without the central logger (use core/logger.py) + +--- + +## Quick links +- Back end entrypoint: src/backend/src/main.py +- Compose routers: src/backend/src/api/__init__.py +- Settings: src/backend/src/config/settings.py +- Sessions: src/backend/src/db/session.py +- CrewAI runner: src/backend/src/engines/crewai/execution_runner.py diff --git a/src/frontend_static/docs/DEVELOPER_GUIDE.md b/src/frontend_static/docs/DEVELOPER_GUIDE.md index 093f54cf..e7c004fb 100644 --- a/src/frontend_static/docs/DEVELOPER_GUIDE.md +++ b/src/frontend_static/docs/DEVELOPER_GUIDE.md @@ -1,168 +1,145 @@ ## Developer Guide +Build, extend, and debug Kasal efficiently. This guide focuses on day-to-day workflows. + +### Components you'll touch +- **Frontend (React SPA)**: UI, designer, monitoring +- **API (FastAPI)**: REST endpoints and validation +- **Services**: Orchestration and business logic +- **Repositories**: DB and external I/O (Databricks, Vector, MLflow) +- **Engines (CrewAI)**: Agent/flow preparation and execution +- **Data & Storage**: SQLAlchemy models/sessions, embeddings, volumes ### Requirements +Tools and versions you need before running the stack. - Python 3.9+ - Node.js 18+ - Postgres (recommended) or SQLite for local dev - Databricks access if exercising Databricks features - -### Quick start -```bash -# Backend -cd src/backend -python -m venv .venv && source .venv/bin/activate -pip install -r ../requirements.txt -./run.sh # http://localhost:8000 (OpenAPI at /api-docs if enabled) - -# Frontend -cd ../frontend -npm install -npm start # http://localhost:3000 -``` - -Health check: -```bash -curl http://localhost:8000/health -# {"status":"healthy"} +## Developer Architecture Overview +This section gives developers a high-level view of the front end and back end. It explains core components and shows how to understand and trace them. + +## Backend Architecture +The backend uses FastAPI with a clean layered structure. It separates HTTP routing, business services, repositories, and SQLAlchemy models. + +### Core Components +- API Routers: src/backend/src/api/* map HTTP endpoints to service calls. +- Services: src/backend/src/services/* implement business logic and transactions. +- Repositories: src/backend/src/repositories/* handle database CRUD. +- Models/Schemas: src/backend/src/models/* and src/backend/src/schemas/* define persistence and I/O contracts. +- Core/Engines: src/backend/src/core/* and src/backend/src/engines/* integrate LLMs and execution flows. +- DB/Session: src/backend/src/db/* configures sessions and Alembic migrations. +- Config/Security: src/backend/src/config/* and src/backend/src/dependencies/* provide settings and auth. + +### Typical Request Flow +A request passes through router, service, repository, and database. LLM calls route through the LLM manager when needed. + +```mermaid +sequenceDiagram + participant C as Client + participant R as FastAPI Router + participant S as Service + participant U as UnitOfWork + participant Repo as Repository + participant DB as Database + participant L as LLM Manager + + C->>R: HTTP Request + R->>S: Validate and delegate + S->>U: Begin transaction + S->>Repo: Query/Command + Repo->>DB: SQLAlchemy operation + DB-->>Repo: Rows/Status + alt Needs LLM + S->>L: Generate/Score + L-->>S: LLM Result + end + S->>U: Commit + S-->>R: DTO/Schema + R-->>C: HTTP Response ``` -### Configuration -Backend settings: `src/backend/src/config/settings.py` -- Core: `DEBUG_MODE`, `LOG_LEVEL`, `DOCS_ENABLED`, `AUTO_SEED_DATABASE` -- Database: - - `DATABASE_TYPE=postgres|sqlite` (default: `postgres`) - - Postgres: `POSTGRES_SERVER`, `POSTGRES_PORT`, `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` - - SQLite: `SQLITE_DB_PATH=./app.db` - - SQL logging: `SQL_DEBUG=true|false` -- Notes: - - `USE_NULLPOOL` is set early in `main.py` to avoid asyncpg pool issues - - Logs written under `src/backend/src/logs/` - -Frontend API base URL (`REACT_APP_API_URL`) at build-time: -```bash -# Option A: dev default already points to http://localhost:8000/api/v1 -# Option B: override explicitly for a build (Unix/macOS) -REACT_APP_API_URL="http://localhost:8000/api/v1" npm run build - -# When using the top-level build script: -# The env var will propagate into the "npm run build" it runs -cd src -REACT_APP_API_URL="http://localhost:8000/api/v1" python build.py +### How to Understand Backend Components +- Start at the router file for the endpoint path. +- Open the service it calls and read business logic. +- Inspect repository methods and referenced models. +- Check schema types for request and response contracts. +- Review unit tests under `src/backend/tests` for examples. + +### Example: Minimal Endpoint Wiring +This shows a typical router, service, and repository connection. + +```python +# router.py +@router.get("/items/{item_id}", response_model=ItemOut) +async def get_item(item_id: UUID, service: ItemService = Depends(...)): + return await service.get_item(item_id) + +# services/item_service.py +class ItemService: + async def get_item(self, item_id: UUID) -> ItemOut: + with self.uow as uow: + item = uow.items.get(item_id) + return ItemOut.model_validate(item) + +# repositories/item_repository.py +class ItemRepository(BaseRepository[Item]): + ... ``` -### Conventions -- Routers (`api/*`): Validate with `schemas/*`, delegate to `services/*` -- Services: Business logic only; use repositories for I/O -- Repositories: All SQL/external I/O; don’t leak ORM to services -- Models: SQLAlchemy in `models/*`; Schemas: Pydantic in `schemas/*` - -### Add a new API resource (“widgets” example) -1) Model: `models/widget.py`; import in `db/all_models.py` -2) Schemas: `schemas/widget.py` (Create/Update/Read DTOs) -3) Repository: `repositories/widget_repository.py` -4) Service: `services/widget_service.py` -5) Router: `api/widgets_router.py` (validate → call service) -6) Register router in `api/__init__.py` -7) Frontend: add `src/frontend/src/api/widgets.ts` + components/views/state -8) Tests: `src/backend/tests/` - -### Add a new CrewAI tool -- Implement under `engines/crewai/tools/` (follow existing patterns) -- Expose configuration via service/router if user-configurable -- Ensure discovery/registration in the execution path (e.g., prep or service) - -### Executions and tracing -- Start executions via `executions_router.py` endpoints -- Services invoke engine flow (`engines/crewai/*`) -- Logs/traces: - - Execution logs via `execution_logs_*` - - Traces via `execution_trace_*` -```bash -# Kick off an execution -curl -X POST http://localhost:8000/api/v1/executions -H "Content-Type: application/json" -d '{...}' - -# Get execution status -curl http://localhost:8000/api/v1/executions/ - -# Fetch logs/trace -curl http://localhost:8000/api/v1/execution-logs/ -curl http://localhost:8000/api/v1/execution-trace/ +## Frontend Architecture +The frontend is a React + TypeScript application. It organizes UI components, API clients, state stores, hooks, and utilities. + +### Core Components +- API Clients: src/frontend/src/api/* wrap HTTP calls with typed methods. +- UI Components: src/frontend/src/components/* render views and dialogs. +- Hooks: src/frontend/src/hooks/* encapsulate logic and side effects. +- Stores: src/frontend/src/store/* manage app and workflow state. +- Types/Config: src/frontend/src/types/* and src/frontend/src/config/* provide typing and environment. +- Utils: src/frontend/src/utils/* offer reusable helpers. + +### UI Data Flow +Components call hooks, which use stores and API clients. Responses update state and re-render the UI. + +```mermaid +flowchart LR + A[UI Component] --> B[Hook] + B --> C[State Store] + B --> D[API Client] + D -->|HTTP| E[Backend API] + E -->|JSON| D + D --> C + C --> A ``` -### Background processing -- Scheduler: starts on DB-ready startup (`scheduler_service.py`) -- Embedding queue (SQLite): `embedding_queue_service.py` batches writes -- Cleanup on startup/shutdown: `execution_cleanup_service.py` - -### Database & migrations -- SQLite for quick local dev (`DATABASE_TYPE=sqlite`), Postgres for multi-user -- Alembic: -```bash -# after model changes -alembic revision --autogenerate -m "add widgets" -alembic upgrade head -``` - -### Auth, identity, tenancy -- Databricks headers parsed by `utils/user_context.py` -- Group-aware tenants; selected group passed in `group_id` header -- JWT/basic auth in `auth_router.py`, users in `users_router.py` -- Authorization checks in `core/permissions.py` - -### Logging & debugging -- App logs: `src/backend/src/logs/` (managed by `core/logger.py`) -- Verbose SQL: `export SQL_DEBUG=true` -- SQLite “database is locked”: mitigated via retry/backoff; reduce writers or use Postgres - -### Frontend notes -- Axios client and base URL: -```1:13:/Users/anshu.roy/Documents/kasal/src/frontend/src/config/api/ApiConfig.ts -export const config = { - apiUrl: - process.env.REACT_APP_API_URL || - (process.env.NODE_ENV === 'development' - ? 'http://localhost:8000/api/v1' - : '/api/v1'), +### How to Understand Frontend Components +- Locate the component rendering the feature. +- Check its hook usage and props. +- Open the API client method it calls. +- Review the store slice it reads or writes. +- Inspect related types in src/frontend/src/types. + +### Example: Calling an API From a Component +A component loads items using an API service and updates local state. + +```ts +// api/ItemService.ts +export async function getItem(id: string) { + const res = await fetch(`/v1/items/${id}`, { headers: { Authorization: `Bearer ${token}` } }); + if (!res.ok) throw new Error("Failed"); + return (await res.json()) as Item; +} + +// components/Items/ItemView.tsx +const ItemView: React.FC<{ id: string }> = ({ id }) => { + const [item, setItem] = useState(null); + useEffect(() => { getItem(id).then(setItem); }, [id]); + return item ?
{item.name}
: Loading...; }; -export const apiClient = axios.create({ baseURL: config.apiUrl, headers: { 'Content-Type': 'application/json' } }); -``` - -### Testing -```bash -# Backend -python run_tests.py - -# Frontend -cd src/frontend -npm test ``` -### Production checklist -- Use Postgres (or managed DB), not SQLite -- Harden secrets/tokens; externalize (e.g., Databricks Secrets/Vault) -- Enforce TLS and CORS -- Monitor logs/traces; set alerts -- Review `DOCS_ENABLED`, `LOG_LEVEL`, `DEBUG_MODE` - -## Resources - -### Quick Links -- [API Playground](/api/docs) -- [Video Tutorials](https://kasal.ai/videos) -- [Discord Community](https://discord.gg/kasal) -- [Report Issues](https://github.com/kasal/issues) - -### Code Examples -- [Basic Agent Setup](https://github.com/kasal/examples/basic) -- [Multi-Agent Collaboration](https://github.com/kasal/examples/multi-agent) -- [Custom Tools](https://github.com/kasal/examples/tools) -- [Production Deployment](https://github.com/kasal/examples/deploy) - -### Support -- **Chat**: Available in-app 24/7 -- **Email**: dev@kasal.ai -- **Slack**: #kasal-developers - ---- +## End-to-End flow +This ties front end and back end with shared contracts. It helps new developers trace a feature quickly. -*Build smarter, ship faster with Kasal* \ No newline at end of file +- Frontend Component → Hook → Store/API Client → Backend Router → Service → Repository → DB. +- Shared types and response shapes live in frontend types and backend schemas. +- Tests in src/backend/tests and frontend __tests__ show usage patterns. \ No newline at end of file diff --git a/src/frontend_static/docs/README.md b/src/frontend_static/docs/README.md index 515063a6..f6cb427d 100644 --- a/src/frontend_static/docs/README.md +++ b/src/frontend_static/docs/README.md @@ -1,31 +1,19 @@ -![Kasal Logo](././/images/logo.png) -# 🟦 Kasal Documentation Hub + +# Kasal Documentation Hub **Enterprise AI Agent Orchestration Platform** --- ## Quick Start +Find the right documentation for your role and get productive fast. Choose your documentation based on your role: -## Business Documentation - -### [Business User Guide](./BUSINESS_USER_GUIDE.md) -**For Leaders, Managers, and Business Users** - -Get started with Kasal without any technical knowledge. Learn how to create AI workflows, calculate ROI, and manage your AI workforce. - -**Key Topics:** -- ROI Calculator and Cost Analysis -- Success Stories and Case Studies -- 15-Minute Quick Setup Guide -- Managing AI Workflows Without Code - ---- ## Technical Documentation +Developer-focused guides, code structure, and API reference. ### [Code structure Guide](./CODE_STRUCTURE_GUIDE.md) **For Software Engineers and Developers** @@ -62,6 +50,7 @@ Complete REST API documentation with examples, WebSocket events, and SDK librari --- ## Architecture Documentation +System design, patterns, and enterprise integration guidance. ### [Solution Architecture Guide](./ARCHITECTURE_GUIDE.md) **For Solution Architects and Technical Leaders** @@ -77,6 +66,7 @@ Understand Kasal's architecture, design patterns, and enterprise integration str --- ## Getting Started in 30 Seconds +One command sequence to run both backend and frontend locally. ```bash # Clone the repository @@ -97,6 +87,7 @@ Access the application at `http://localhost:3000` --- ## Key Features +What Kasal provides out of the box to build, operate, and govern AI workflows. | Feature | Description | |---------|-------------| @@ -110,11 +101,11 @@ Access the application at `http://localhost:3000` --- ## Documentation Structure +How this folder is organized and where to find topics. ``` docs/ ├── README.md # This file - Documentation hub -├── BUSINESS_USER_GUIDE.md # Business user documentation ├── CODE_STRUCTURE_GUIDE.md # Code structure documentation ├── DEVELOPER_GUIDE.md # Developer documentation ├── ARCHITECTURE_GUIDE.md # Architecture documentation @@ -128,6 +119,7 @@ docs/ --- ## Technology Stack +Core frameworks and platforms used across the project. - **Backend**: FastAPI, SQLAlchemy 2.0, Python 3.9+ - **Frontend**: React 18, TypeScript, Material-UI @@ -138,6 +130,7 @@ docs/ --- ## Support and Resources +Where to get help and how to contribute. ### Getting Help @@ -156,9 +149,10 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](https: --- ## Version Information +Project and documentation version details. - **Current Version**: 2.0.0 -- **Documentation Updated**: January 2025 +- **Documentation Updated**: October 2025 - **Minimum Python Version**: 3.9 - **Minimum Node Version**: 16.0