Product Requirements Document: OpenAPI Test Generation

Overview

Feature Name: OpenAPI Test Generation
Version: 1.0
Target Release: Next minor version
Owner: Manny

Summary

Enable Doc Detective to automatically generate HTTP API tests from OpenAPI 3.x specifications provided as direct input files. This feature transforms OpenAPI definitions into executable test specifications that leverage Doc Detective's existing httpRequest action capabilities.

Background & Motivation

Currently, Doc Detective can reference OpenAPI definitions for individual httpRequest steps, but cannot automatically generate comprehensive test suites from OpenAPI specifications. This manual process limits test coverage and requires significant developer effort to maintain API tests as specifications evolve.

By automatically generating tests from OpenAPI definitions, teams can:

• Achieve comprehensive API test coverage with minimal manual effort
• Maintain test synchronization with API specification changes
• Leverage contract testing for API validation
• Reduce time-to-market for API testing initiatives

Goals & Success Criteria

Primary Goals

1. Automatic Test Generation: Transform OpenAPI 3.x files into executable Doc Detective test specifications
2. Safety-First Approach: Only generate tests for operations deemed safe to execute automatically
3. Dependency Management: Support operation dependencies through x-doc-detective extensions
4. Seamless Integration: Work within existing Doc Detective detection and resolution workflows

Success Criteria

• OpenAPI 3.x files can be provided as direct input to doc-detective command
• Generated tests execute successfully against live APIs
• Tests leverage all existing Doc Detective httpRequest capabilities
• Zero breaking changes to existing functionality

User Stories

Primary User Stories

1. As an API developer, I want to provide my OpenAPI spec as input to Doc Detective so that comprehensive API tests are automatically generated
2. As a QA engineer, I want unsafe operations (PUT/DELETE) to require explicit configuration so that automated tests don't accidentally modify production data
3. As a DevOps engineer, I want dependency management between API operations so that complex test scenarios (create → read → delete) can be automated

Secondary User Stories

1. As an API maintainer, I want root-level defaults in my OpenAPI spec so that common configuration applies to all operations
2. As a test author, I want operation-level overrides so that specific endpoints can have custom test behavior

Functional Requirements

Core Functionality

FR-1: OpenAPI File Detection

• Requirement: Detect OpenAPI 3.x files in input parameter
• Acceptance Criteria:
  • Files with .json, .yaml, .yml extensions containing valid OpenAPI 3.x structure are identified
  • OpenAPI detection bypasses existing fileTypes validation
  • Only OpenAPI 3.x specifications are supported (version starts with "3.")
  • Invalid OpenAPI files are logged and skipped

FR-2: Automatic Test Generation

• Requirement: Transform OpenAPI operations into Doc Detective test specifications
• Acceptance Criteria:
  • Each operation becomes a separate test with unique testId
  • Generated tests use httpRequest action with openApi.operationId reference
  • Test descriptions derived from operation summary or description
  • OpenAPI definition is loaded, dereferenced, and included in spec's openApi array

FR-3: Operation Safety Classification

• Requirement: Only generate tests for operations deemed safe to execute
• Acceptance Criteria:
  • GET operations are safe by default
  • POST operations are safe by default
  • PUT, PATCH, DELETE, HEAD, OPTIONS operations are unsafe by default
  • Unsafe operations are skipped unless explicitly marked safe
  • Safety decisions are logged at debug level

FR-4: Extension Property Support

• Requirement: Support x-doc-detective extension for operation configuration
• Acceptance Criteria:
  • x-doc-detective accepted at OpenAPI root level (defaults for all operations)
  • x-doc-detective accepted at operation level (overrides root defaults)
  • Supports safe boolean property to override default safety classification
  • Supports before array for prerequisite operations
  • Supports after array for cleanup operations
  • Supports all existing openApi schema properties: server, validateSchema, mockResponse, statusCodes, useExample, exampleKey, requestHeaders, responseHeaders

FR-5: Dependency Management

• Requirement: Execute prerequisite and cleanup operations
• Acceptance Criteria:
  • before operations are added as steps before main operation
  • after operations are added as steps after main operation
  • Dependencies referenced by operationId OR path+method combination
  • No nested dependencies supported (flat structure only)
  • Dependency resolution failures are logged and operation is skipped
  • Variables from before operations automatically propagate to subsequent steps

Integration Requirements

FR-6: Workflow Integration

• Requirement: Integrate with existing Doc Detective detection workflow
• Acceptance Criteria:
  • OpenAPI detection occurs in parseTests() function within utils.js
  • Generated specs follow same structure as manually authored specs
  • Generated tests go through normal resolution process (contexts, browsers, etc.)
  • No changes required to resolveTests() or execution logic

FR-7: Configuration Inheritance

• Requirement: Leverage existing Doc Detective configuration system
• Acceptance Criteria:
  • Generated tests respect global configuration settings
  • OpenAPI-specific configuration merges with global defaults
  • Existing loadDescription() function used for OpenAPI dereferencing
  • Environment variable substitution works in generated tests

Technical Requirements

TR-1: File Processing

• Modify isValidSourceFile() to identify OpenAPI 3.x files
• Add isOpenApi3File() helper function for OpenAPI detection
• Extend parseTests() to handle OpenAPI transformation

TR-2: Code Organization

• Add transformation functions to openapi.js:
  • transformOpenApiToSpec()
  • extractOperations()
  • transformOperationToTest()
  • createHttpRequestStep()
  • createDependencySteps()
  • isOperationSafe()
  • findOperationById()
  • findOperationByPath()

TR-3: Error Handling

• Graceful handling of malformed OpenAPI files
• Comprehensive logging for debugging generated tests
• Fallback behavior when dependencies cannot be resolved

TR-4: Performance Considerations

• Efficient operation extraction from large OpenAPI specifications
• Minimal memory overhead for dereferenced definitions
• Reasonable test generation time for complex APIs

Non-Functional Requirements

NFR-1: Backward Compatibility

• Zero breaking changes to existing Doc Detective functionality
• Existing test specifications and workflows remain unchanged
• Existing OpenAPI integration features remain functional

NFR-2: Maintainability

• Code follows existing Doc Detective patterns and conventions
• Comprehensive logging for troubleshooting
• Clear error messages for configuration issues

NFR-3: Extensibility

• Architecture supports future enhancements (retry logic, advanced dependency management)
• Clean separation between OpenAPI processing and core test logic

Implementation Phases

Phase 1: Core Implementation

1. OpenAPI file detection in isValidSourceFile()
2. Basic test generation in parseTests()
3. Operation safety classification
4. Simple httpRequest step generation

Phase 2: Extension Support

1. x-doc-detective property parsing
2. Root-level and operation-level configuration merging
3. OpenAPI schema property integration
4. Dependency management (before/after operations)

Phase 3: Polish & Testing

1. Comprehensive error handling and logging
2. Integration testing with real OpenAPI specifications
3. Documentation and examples
4. Performance optimization

Testing Strategy

Unit Tests

• OpenAPI file detection accuracy
• Test generation logic for various operation types
• Safety classification for different HTTP methods
• Extension property parsing and merging
• Dependency resolution logic

Integration Tests

• End-to-end test generation from sample OpenAPI files
• Generated test execution against mock APIs
• Variable propagation between dependent operations
• Error handling for malformed specifications

Test Data

• Valid OpenAPI 3.x specifications with various operation types
• Specifications with x-doc-detective extensions
• Malformed/invalid OpenAPI files for error testing
• Complex dependency scenarios

Dependencies & Assumptions

Dependencies

• Existing loadDescription() function for OpenAPI dereferencing
• Existing httpRequest action implementation
• Existing variable propagation system ($$response.body.*)
• doc-detective-common validation functions

Assumptions

• OpenAPI specifications are well-formed and valid
• Target APIs are accessible during test execution
• Users understand OpenAPI concepts and structure
• Existing OpenAPI integration patterns remain stable

Documentation Requirements

User Documentation

• Usage examples with sample OpenAPI files
• x-doc-detective extension property reference
• Safety classification explanation
• Dependency management patterns

Developer Documentation

• Implementation architecture overview
• Function documentation for new utilities
• Integration points with existing codebase
• Troubleshooting guide for common issues

Success Metrics

Adoption Metrics

• Number of OpenAPI files processed successfully
• Percentage of generated tests that execute without errors
• User adoption rate of x-doc-detective extensions

Quality Metrics

• Test coverage of generated test logic
• Performance benchmarks for large OpenAPI specifications
• Error rate for malformed input files

Risks & Mitigations

Technical Risks

• Risk: Complex OpenAPI specifications may generate too many tests
  • Mitigation: Implement filtering and safety checks
• Risk: Dependency cycles in before/after operations
  • Mitigation: Flat dependency structure prevents cycles
• Risk: Performance impact on large specifications
  • Mitigation: Efficient parsing and lazy evaluation

Product Risks

• Risk: Generated tests may not match user expectations
  • Mitigation: Clear documentation and extensive examples
• Risk: Breaking changes to existing OpenAPI integration
  • Mitigation: Comprehensive regression testing

Future Considerations

Potential Enhancements

• Support for OpenAPI 2.x (Swagger) specifications
• Advanced dependency management with cycle detection
• Custom test generation templates
• Integration with API mocking frameworks
• Parallel execution optimization for large test suites

Integration Opportunities

• CI/CD pipeline integration for contract testing
• API documentation generation from test results
• Integration with API gateway testing workflows