diff --git a/docs/import-enhancement.md b/docs/import-enhancement.md index ab810ee..9c9918c 100644 --- a/docs/import-enhancement.md +++ b/docs/import-enhancement.md @@ -5,421 +5,117 @@ Assisted by CursorAI # StepZen Import Enhancement System -This document outlines the two-phase approach for integrating StepZen CLI import commands with the VS Code extension and providing intelligent schema enhancements. +This document outlines the roadmap for enhancing StepZen's import capabilities in VS Code, building on the solid foundation of CLI integration that's now in place. -## Overview +## Current State -The import enhancement system consists of two distinct phases: +The extension now includes a complete CLI import integration system with: -1. **Phase 1: CLI Import Integration** - Seamless UI for all StepZen import commands -2. **Phase 2: Functional Enhancements** - Intelligent post-import schema improvements +- **All Import Types Supported**: cURL, OpenAPI, GraphQL, and Database imports +- **Robust Architecture**: Service-based design with command builders for each import type +- **Shell Safety**: Proper escaping for passwords and special characters +- **Type Safety**: Comprehensive TypeScript interfaces and validation +- **Test Coverage**: 227 passing tests with comprehensive coverage -## Phase 1: CLI Import Integration (✅ COMPLETED) +## Next Steps -### Architecture +### Phase 1: UI Polish & User Experience -The system uses a generalized service-based architecture that follows the extension's established patterns: +#### 1. Enhanced User Interfaces -``` -ImportService (Orchestrator) -├── CurlCommandBuilder -├── OpenApiCommandBuilder -├── GraphQLCommandBuilder -└── DatabaseCommandBuilder -``` +**Current**: Basic command implementations exist +**Goal**: Rich, intuitive VS Code interfaces -**Key Components:** +- **cURL Import UI** -- **`ImportService`** - Central orchestrator that handles all import types -- **Command Builders** - Type-specific logic for building CLI arguments -- **Type Definitions** - Comprehensive TypeScript interfaces for all configurations -- **VS Code Commands** - User-friendly interfaces for each import type + - Smart cURL command parsing and preview + - Visual header/authentication configuration + - Real-time validation and suggestions -### Supported Import Types +- **Database Import UI** -#### 1. cURL Import (`stepzen import curl`) + - Connection testing before import + - Database schema preview + - Table/view selection interface -- **Command**: `stepzen.importCurl` -- **Features**: - - Smart cURL command parsing - - Auto-detection of secret headers - - Path parameter configuration - - Schema and query name generation - - **Request body support** - Handles `--data`, `-d`, `--data-raw`, `--data-ascii`, `--data-binary` flags - - **HTTP method detection** - Supports GET, POST, PUT, PATCH, DELETE methods - - **Header parsing** - Supports both `-H` and `--header` flags +- **GraphQL Import UI** + - Endpoint introspection and validation + - Schema preview before import + - Authentication method selection -#### 2. OpenAPI Import (`stepzen import openapi`) +#### 2. Import Experience Improvements -- **Command**: `stepzen.importOpenapi` -- **Features**: - - File browser for local specs - - URL input for remote specs - - Automatic schema name generation - - Support for YAML and JSON formats +- **Progress Indicators**: Show import progress and status +- **Better Error Messages**: Actionable error descriptions with suggestions +- **Import Preview**: Show what will be generated before executing +- **Success Actions**: Quick access to generated files and next steps -#### 3. GraphQL Import (`stepzen import graphql`) +#### 3. Workspace Integration -- **Command**: `stepzen.importGraphql` -- **Features**: - - Multiple authentication methods (Bearer, API Key, Basic Auth) - - Type prefix configuration - - Endpoint validation - - Secret management +- **Project Detection**: Smarter StepZen project discovery +- **File Organization**: Intelligent placement of generated schemas +- **Conflict Resolution**: Handle naming conflicts gracefully -#### 4. Database Import (`stepzen import {postgresql|mysql|db2|oracle|snowflake|presto}`) +### Phase 2: Intelligent Schema Enhancements -- **Command**: `stepzen.importDatabase` -- **Features**: - - Support for all StepZen database types - - Connection string or individual parameters - - Auto-linking with `@materializer` - - Database-specific options (e.g., Snowflake warehouse) +Once the import UX is polished, add intelligent post-import schema improvements: -### Implementation Status +#### 1. Automatic Improvements -- ✅ **Type Definitions** - Complete TypeScript interfaces (`src/types/import.ts`) -- ✅ **Import Service** - Generalized service with command builders (`src/services/importService.ts`) -- ✅ **Command Implementations** - All four import commands implemented -- ✅ **Unit Tests** - Comprehensive test coverage (226 passing tests) -- ✅ **Service Registry Integration** - ImportService registered and available -- ✅ **Command Registration** - All commands registered in `extension.ts` -- ✅ **Error Handling** - Robust validation and error reporting -- ✅ **Architecture Compliance** - Follows established extension patterns -- ✅ **Project Resolution** - Automatic StepZen project directory detection with multi-project support -- ⏳ **UI Implementation** - Need to create VS Code command interfaces -- ⏳ **cURL Parsing Logic** - Need to implement actual cURL command parsing -- ⏳ **Integration Testing** - End-to-end testing with CLI -- ⏳ **User Documentation** - User-facing guides and examples +- **Field Name Normalization**: Convert `snake_case` to `camelCase` +- **Query Name Optimization**: Fix verb-based names (`getUser` → `user`) +- **Type Documentation**: Generate descriptions from source metadata +- **Pagination Addition**: Convert lists to GraphQL Cursor Connections -### Testing Strategy +#### 2. Relationship Detection -#### Unit Tests +- **Foreign Key Analysis**: Detect relationships in database schemas +- **Cross-Source Linking**: Connect data across different import sources +- **Materializer Suggestions**: Recommend `@materializer` directives -- **Import Service Tests** (`src/test/unit/services/importService.test.ts`) +#### 3. Enhancement UI - - CLI argument building for all import types - - Configuration validation - - Error handling - - Type detection logic +- **Enhancement Preview**: Show proposed changes before applying +- **Selective Application**: Choose which enhancements to apply +- **Rollback Support**: Undo enhancements if needed -- **Command Tests** (`src/test/unit/commands/importCurl.test.ts`) - - cURL parsing logic - - URL validation - - Schema name generation - - Edge case handling +## Implementation Priorities -#### Integration Tests (TODO) +### High Priority (Next Sprint) -- End-to-end command execution -- CLI integration validation -- File system operations -- Error scenarios +1. **cURL Command Parsing**: Complete the smart cURL parsing logic +2. **Database Connection Testing**: Validate connections before import +3. **Import Progress UI**: Show real-time import status -### Architecture Alignment +### Medium Priority -The implementation follows the extension's established patterns: +1. **Schema Preview**: Show generated schema before import +2. **Error Recovery**: Better handling of failed imports +3. **Import History**: Track and replay previous imports -- **Service Registry** - `ImportService` registered in service registry -- **Error Handling** - Uses `handleError()` and `ValidationError` -- **Logging** - Integrated with `services.logger` -- **Command Structure** - Follows established command patterns -- **TypeScript Strict Mode** - Full type safety -- **Project Resolution** - Uses `ProjectResolver` service for automatic directory detection +### Future Considerations -### Project Resolution +1. **Custom Import Types**: Plugin system for new import sources +2. **Batch Operations**: Import multiple sources at once +3. **Schema Versioning**: Track schema changes over time -The import system automatically resolves the correct StepZen project directory using the established `ProjectResolver` service: +## Architecture Notes -#### Single Project Workspace +The current implementation follows established extension patterns: -- Automatically detects the StepZen project directory -- Searches upward from the active file or workspace root -- Executes import commands in the correct project directory +- **Service Registry**: All import functionality accessible via `services.import` +- **Command Builders**: Extensible pattern for new import types +- **Error Handling**: Consistent error reporting and logging +- **Type Safety**: Full TypeScript coverage with proper validation -#### Multi-Project Workspace +New features should continue following these patterns for consistency and maintainability. -- Scans all workspace folders for StepZen projects -- Prompts user to select the target project when multiple are found -- Caches the selection for subsequent operations -- Supports multi-root VS Code workspaces +## Success Metrics -#### Error Handling - -- Clear error messages when no StepZen project is found -- Graceful handling of user cancellation during project selection -- Automatic fallback to workspace scanning when active file context is unavailable - -This ensures that import commands always execute in the correct directory, regardless of workspace complexity. - -## Current Implementation Details - -### File Structure - -``` -src/ -├── types/ -│ └── import.ts # Complete type definitions for all import configs -├── services/ -│ ├── importService.ts # Core ImportService with command builders -│ └── index.ts # Service registry integration -├── commands/ -│ ├── importCurl.ts # cURL import command (placeholder) -│ ├── importOpenapi.ts # OpenAPI import command (placeholder) -│ ├── importGraphql.ts # GraphQL import command (placeholder) -│ └── importDatabase.ts # Database import command (placeholder) -└── test/unit/ - ├── services/ - │ └── importService.test.ts # Comprehensive ImportService tests - └── commands/ - └── importCurl.test.ts # cURL parsing logic tests -``` - -### Key Implementation Features - -#### Type System (`src/types/import.ts`) - -- **Discriminated Unions** - Proper TypeScript type discrimination -- **Comprehensive Interfaces** - All CLI options covered -- **Validation Support** - Built-in validation helpers -- **Extensibility** - Easy to add new import types - -#### Import Service (`src/services/importService.ts`) - -- **Command Builder Pattern** - Extensible architecture -- **Automatic Type Detection** - Smart configuration analysis -- **Robust Validation** - Comprehensive input validation -- **Error Handling** - Proper error bubbling and reporting - -#### Service Integration - -- **Dependency Injection** - Uses service registry pattern -- **Logging Integration** - Comprehensive logging throughout -- **Error Handling** - Follows extension error patterns -- **Testing Support** - Fully mockable for testing - -### Command Registration - -All import commands are registered in `src/extension.ts`: - -```typescript -// Import commands -vscode.commands.registerCommand('stepzen.importCurl', importCurl), -vscode.commands.registerCommand('stepzen.importOpenapi', importOpenapi), -vscode.commands.registerCommand('stepzen.importGraphql', importGraphql), -vscode.commands.registerCommand('stepzen.importDatabase', importDatabase), -``` - -## Phase 2: Functional Enhancements (🔄 NEXT PHASE) - -### Enhancement Categories - -#### 1. Add Pagination - -- **Target**: All directive types (`@rest`, `@dbquery`, `@graphql`) -- **Implementation**: Convert list fields to GraphQL Cursor Connections -- **Benefits**: Consistent pagination across all data sources - -#### 2. Improve Field Names - -- **Target**: All generated types -- **Implementation**: - - Convert `snake_case` to `camelCase` - - Fix verb-based query names (e.g., `getUser` → `user`) - - Standardize naming conventions -- **Benefits**: Better GraphQL conventions - -#### 3. Add Documentation - -- **Target**: All types and fields -- **Implementation**: - - Generate descriptions from database comments - - Add field documentation from OpenAPI descriptions - - Create meaningful type descriptions -- **Benefits**: Self-documenting schemas - -#### 4. Connect Related Data - -- **Target**: Object types with relationships -- **Implementation**: - - Add `@materializer` directives for foreign keys - - Create nested object relationships - - Link related entities across data sources -- **Benefits**: Rich, connected data graphs - -### Enhancement Architecture - -``` -EnhancementService -├── PaginationEnhancer -├── FieldNameEnhancer -├── DocumentationEnhancer -└── RelationshipEnhancer -``` - -**Integration Points:** - -- **Schema Analysis** - Leverage existing `SchemaIndexService` -- **File Modification** - Use established file editing patterns -- **User Interface** - Follow command structure patterns - -## Development Roadmap - -### Phase 1: CLI Import Integration (✅ COMPLETED) - -**Status**: Foundation complete with comprehensive test coverage - -**Completed Components**: - -- ✅ Complete type system for all import configurations -- ✅ Generalized ImportService with command builder pattern -- ✅ All four import command implementations (cURL, OpenAPI, GraphQL, Database) -- ✅ Robust validation and error handling -- ✅ Service registry integration -- ✅ 226 passing unit tests with 95%+ coverage -- ✅ Architecture compliance with extension patterns - -### Immediate Next Steps (Phase 1 Completion) - -#### 1. UI Implementation (High Priority) - -- **Create VS Code Command Interfaces** - - Input forms for each import type - - File/URL pickers for OpenAPI specs - - Authentication configuration UI - - Progress indicators and feedback - -#### 2. cURL Parsing Logic (High Priority) - -- **Implement Smart cURL Parsing** - - Extract URLs, headers, and authentication - - Auto-detect secret headers (Authorization, API keys) - - Generate meaningful schema and query names - - Handle complex cURL commands with multiple options - -#### 3. Integration Testing (Medium Priority) - -- **End-to-End CLI Testing** - - Test with actual StepZen CLI commands - - Validate generated arguments - - Error scenario testing - - File system operations - -#### 4. User Experience Polish (Medium Priority) - -- **Enhanced User Feedback** - - Better error messages with actionable suggestions - - Progress indicators for long-running imports - - Success notifications with next steps - - Workspace trust validation - -### Phase 2: Functional Enhancements (🔄 NEXT PHASE) - -#### Planning Phase (Immediate) - -1. **Schema Analysis Enhancement** - - - Extend `SchemaIndexService` for enhancement detection - - Identify enhancement opportunities in imported schemas - - Build enhancement recommendation engine - -2. **Enhancement UI Design** - - Design user interface for functional enhancements - - Create enhancement preview and confirmation flows - - Implement enhancement progress tracking - -#### Implementation Phase (Future) - -1. **Core Enhancement Engine** - - - Build safe schema modification patterns - - Implement enhancement algorithms - - Add rollback and undo capabilities - -2. **Enhancement Categories** - - Add Pagination (GraphQL Cursor Connections) - - Improve Field Names (camelCase, proper conventions) - - Add Documentation (from comments and descriptions) - - Connect Related Data (@materializer relationships) - -## Testing Coverage - -### Current Test Coverage (✅ COMPREHENSIVE) - -- ✅ **Import Service** - 100% coverage (`src/test/unit/services/importService.test.ts`) - - - All command builders tested (cURL, OpenAPI, GraphQL, Database) - - Configuration validation for all import types - - Error scenarios and edge cases - - Type detection and argument building - - CLI integration mocking - -- ✅ **Command Logic** - 95% coverage (`src/test/unit/commands/importCurl.test.ts`) - - - URL parsing and validation - - cURL command parsing logic - - Schema name generation algorithms - - Query name generation with parameter handling - - Header extraction and secret detection - - Edge cases and malformed input handling - -- ✅ **Integration Tests** - Service registry integration - - ImportService properly registered and accessible - - Dependency injection working correctly - - Error handling integration with extension patterns - -**Total Test Count**: 226 passing tests -**Coverage**: 95%+ across all import functionality - -### Planned Test Coverage - -- ⏳ **UI Integration Tests** - VS Code command testing - - - Command palette integration - - Input validation and user feedback - - File picker and URL input testing - - Authentication flow testing - -- ⏳ **End-to-End Tests** - CLI execution - - - Actual StepZen CLI integration - - File system operations - - Generated schema validation - - Error recovery scenarios - -- ⏳ **Enhancement Tests** - Schema modification testing (Phase 2) - - Safe schema editing patterns - - Enhancement algorithm validation - - Rollback and undo functionality - -## Documentation - -### User Documentation (TODO) - -- **Getting Started Guide** - How to use import commands -- **Configuration Reference** - All available options -- **Troubleshooting Guide** - Common issues and solutions -- **Examples** - Real-world import scenarios - -### Developer Documentation - -- **Architecture Guide** - System design and patterns -- **Extension Guide** - Adding new import types -- **Testing Guide** - Running and writing tests - -## Security Considerations - -### Credential Handling - -- **Secrets Detection** - Auto-detect authentication headers -- **Secure Storage** - Use VS Code secret storage for credentials -- **Non-Interactive Mode** - Prevent credential prompts in CLI - -### Validation - -- **Input Validation** - Validate all user inputs -- **URL Validation** - Ensure valid endpoints -- **Connection Validation** - Test database connections safely +- **User Adoption**: Track usage of import commands +- **Error Reduction**: Measure import success rates +- **Time Savings**: Compare manual vs. automated import workflows +- **Schema Quality**: Assess generated schema adherence to GraphQL best practices ---