feat: Implement dataset worker API and worker management

[Sankara-Jefferson]

Dataset Worker API Implementation

Overview

This PR implements a new API for managing dataset workers in Singularity, providing endpoints for worker lifecycle management and status monitoring.

Key Changes

• Add new dataset worker API endpoints:

  • POST /api/dataset-worker/start: Start a worker with configurable parameters
  • POST /api/dataset-worker/stop: Stop a running worker
  • GET /api/dataset-worker/status: Get current worker status
  • GET /api/dataset-worker/logs: Retrieve worker logs

• Implement worker process management with:

  • Process creation and lifecycle management
  • Configuration handling (concurrency, scan/pack/dag settings)
  • Worker status tracking
  • Error handling and recovery

• Add comprehensive test coverage:

  • Unit tests for worker management
  • Integration tests for API endpoints
  • Mock implementations for testing

Testing

• Unit tests added for all new functionality
• Integration tests verify full worker lifecycle
• Mock command runner enables reliable testing without actual processes

Documentation

• Added DATASET_WORKER_API.md with detailed API documentation
• Included inline documentation for public interfaces

Implementation Details

• Uses dependency injection pattern for process management
• Thread-safe worker state handling
• Graceful shutdown support
• Configurable worker parameters

[ianconsolata]

What's the status of this draft PR? Is there still a plan to merge it?