Automatically invoke Docker when running pytest

CONTRIBUTING.md

@@ -25,27 +25,150 @@ pre-commit run --all-files
 
 ## Running tests
 
-To create a test environment, do the following:
-```
-pip install -e ".[dev]"
+### Prerequisites
+
+Before running tests, ensure you have:
+- **Docker installed and running** (for automatic test database management)
+- Test dependencies installed: `pip install -e ".[test]"` or `pip install -e ".[dev]"` for all development dependencies
+
+The `docker` Python package is required for the test framework to manage Docker containers automatically.
+
+### Installation
+
+To create a test environment:
+```bash
+pip install -e ".[dev]"  # All development dependencies (recommended)
 ```
 
 Or if you only need specific dependency groups:
-```
-pip install -e ".[test]"  # Just testing dependencies
-pip install -e ".[docs]"  # Just documentation dependencies
+```bash
+pip install -e ".[test]"      # Just testing dependencies
+pip install -e ".[docs]"      # Just documentation dependencies
+pip install -e ".[build]"     # Just build dependencies
+pip install -e ".[examples]"  # Just example/demo dependencies
 ```
 
-If you have Docker installed, you can run the tests as follows. Note that
-you should run the tests using both standard protocol and Data API (HTTP):
-```
+### Basic Testing
+
+The test framework provides **automatic Docker container management**. When you run tests without setting `SINGLESTOREDB_URL`, the framework will:
+
+1. Automatically start a SingleStore Docker container (`ghcr.io/singlestore-labs/singlestoredb-dev`)
+2. Allocate dynamic ports to avoid conflicts (MySQL, Data API, Studio)
+3. Wait for the container to be ready
+4. Run all tests against the container
+5. Clean up the container after tests complete
+
+#### Standard MySQL Protocol Tests
+```bash
+# Run all tests (auto-starts Docker container)
 pytest -v singlestoredb/tests
-USE_DATA_API=1 -v singlestoredb/tests
+
+# Run with coverage
+pytest -v --cov=singlestoredb --pyargs singlestoredb.tests
+
+# Run single test file
+pytest singlestoredb/tests/test_basics.py
+
+# Run without management API tests
+pytest -v -m 'not management' singlestoredb/tests
 ```
 
-If you need to run against a specific server version, you can specify
-the URL of that server:
+#### Data API Tests
+
+The SDK supports testing via SingleStore's **Data API** (port 9000) instead of the MySQL protocol (port 3306). This mode uses a **dual-URL system**:
+
+- `SINGLESTOREDB_URL`: Set to HTTP Data API endpoint (port 9000) for test operations
+- `SINGLESTOREDB_INIT_DB_URL`: Automatically set to MySQL endpoint (port 3306) for setup operations
+
+**Why dual URLs?** Some setup operations like `SET GLOBAL` and `USE database` commands don't work over the HTTP Data API, so they're routed through the MySQL protocol automatically.
+
+Enable HTTP Data API testing:
+```bash
+# Run tests via HTTP Data API
+USE_DATA_API=1 pytest -v singlestoredb/tests
+```
+
+**Known Limitations in HTTP Data API Mode:**
+- `USE database` command is not supported (some tests will be skipped)
+- Setup operations requiring `SET GLOBAL` are automatically routed to MySQL protocol
+
+#### Testing Against an Existing Server
+
+If you have a running SingleStore instance, you can test against it by setting `SINGLESTOREDB_URL`. The Docker container will not be started.
+
+```bash
+# Test against MySQL protocol
+SINGLESTOREDB_URL=user:password@host:3306 pytest -v singlestoredb/tests
+
+# Test against Data API
+SINGLESTOREDB_INIT_DB_URL=user:password@host:3306 \
+    SINGLESTOREDB_URL=http://user:password@host:9000 \
+    pytest -v singlestoredb/tests
 ```
-SINGLESTOREDB_URL=user:[email protected]:3306 pytest -v singlestoredb/tests
-SINGLESTOREDB_URL=http://user:[email protected]:8090 pytest -v singlestoredb/tests
+
+### Docker Container Details
+
+When the test framework starts a Docker container automatically:
+
+- **Container name**: `singlestoredb-test-{worker}-{uuid}` (supports parallel test execution)
+- **Port mappings**:
+  - MySQL protocol: Random available port → Container port 3306
+  - Data API (HTTP): Random available port → Container port 9000
+  - Studio: Random available port → Container port 8080
+- **License**: Uses `SINGLESTORE_LICENSE` environment variable if set, otherwise runs without license
+- **Cleanup**: Container is automatically removed after tests complete
+
+### Environment Variables
+
+The following environment variables control test behavior:
+
+- **`SINGLESTOREDB_URL`**: Database connection URL. If not set, a Docker container is started automatically.
+  - MySQL format: `user:password@host:3306`
+  - HTTP format: `http://user:password@host:9000`
+
+- **`USE_DATA_API`**: Set to `1`, `true`, or `on` to run tests via HTTP Data API instead of MySQL protocol.
+  - Automatically sets up the dual-URL system
+  - Example: `USE_DATA_API=1 pytest -v singlestoredb/tests`
+
+- **`SINGLESTOREDB_INIT_DB_URL`**: MySQL connection URL for setup operations (auto-set in HTTP Data API mode). Used for operations that require MySQL protocol even when testing via HTTP.
+
+- **`SINGLESTORE_LICENSE`**: Optional license key for Docker container. If not provided, container runs without a license.
+
+- **`SINGLESTOREDB_PURE_PYTHON`**: Set to `1` to disable C acceleration and test in pure Python mode.
+
+- **`SINGLESTOREDB_MANAGEMENT_TOKEN`**: Management API token for testing management features (mark tests with `@pytest.mark.management`).
+
+### Testing Best Practices
+
+1. **Test both protocols**: Always run tests with both MySQL protocol and HTTP Data API before submitting:
+   ```bash
+   pytest -v singlestoredb/tests
+   USE_DATA_API=1 pytest -v singlestoredb/tests
+   ```
+
+2. **Pure Python testing**: Test without C acceleration to ensure compatibility:
+   ```bash
+   SINGLESTOREDB_PURE_PYTHON=1 pytest -v singlestoredb/tests
+   ```
+
+3. **Management API tests**: These require a management token and are marked with `@pytest.mark.management`.
+
+### Examples
+
+```bash
+# Standard workflow - test both protocols
+pytest -v singlestoredb/tests
+USE_DATA_API=1 pytest -v singlestoredb/tests
+
+# Test single module with coverage
+pytest -v --cov=singlestoredb.connection singlestoredb/tests/test_connection.py
+
+# Test UDF functionality
+pytest singlestoredb/tests/test_udf.py
+
+# Test against specific server (skips Docker)
+SINGLESTOREDB_URL=admin:pass@localhost:3306 pytest -v singlestoredb/tests
+
+# Debug mode with verbose output
+pytest -vv -s singlestoredb/tests/test_basics.py
 ```

pyproject.toml

@@ -47,6 +47,7 @@ vectorstore = ["singlestore-vectorstore>=0.1.2"]
 test = [
     "coverage",
     "dash",
+    "docker",
     "fastapi",
     "ipython",
     "jupysql",

singlestoredb/pytest.py

@@ -86,21 +86,43 @@ def node_name() -> Iterator[str]:
 
 
 class _TestContainerManager():
-    """Manages the setup and teardown of a SingleStoreDB Dev Container"""
+    """Manages the setup and teardown of a SingleStoreDB Dev Container
+
+    If SINGLESTOREDB_URL environment variable is set, the manager will use
+    the existing server instead of starting a Docker container. This allows
+    tests to run against either an existing server or an automatically
+    managed Docker container.
+    """
 
     def __init__(self) -> None:
+        # Check if SINGLESTOREDB_URL is already set - if so, use existing server
+        self.existing_url = os.environ.get('SINGLESTOREDB_URL')
+        self.use_existing = self.existing_url is not None
+
+        if self.use_existing:
+            logger.info('Using existing SingleStore server from SINGLESTOREDB_URL')
+            self.url = self.existing_url
+            # No need to initialize Docker-related attributes
+            return
+
+        logger.info('SINGLESTOREDB_URL not set, will start Docker container')
+
         # Generate unique container name using UUID and worker ID
         worker = os.environ.get('PYTEST_XDIST_WORKER', 'master')
         unique_id = uuid.uuid4().hex[:8]
         self.container_name = f'singlestoredb-test-{worker}-{unique_id}'
 
         self.dev_image_name = 'ghcr.io/singlestore-labs/singlestoredb-dev'
 
-        assert 'SINGLESTORE_LICENSE' in os.environ, 'SINGLESTORE_LICENSE not set'
+        # Use SINGLESTORE_LICENSE from environment, or empty string as fallback
+        # Empty string works for the client SDK
+        license = os.environ.get('SINGLESTORE_LICENSE', '')
+        if not license:
+            logger.info('SINGLESTORE_LICENSE not set, using empty string')
 
         self.root_password = 'Q8r4D7yXR8oqn'
         self.environment_vars = {
-            'SINGLESTORE_LICENSE': None,
+            'SINGLESTORE_LICENSE': license,
             'ROOT_PASSWORD': f"\"{self.root_password}\"",
             'SINGLESTORE_SET_GLOBAL_DEFAULT_PARTITIONS_PER_LEAF': '1',
         }
@@ -111,12 +133,23 @@ def __init__(self) -> None:
         self.studio_port = _find_free_port()
         self.ports = [
             (self.mysql_port, '3306'),    # External port -> Internal port
-            (self.http_port, '8080'),
-            (self.studio_port, '9000'),
+            (self.studio_port, '8080'),   # Studio
+            (self.http_port, '9000'),     # Data API
         ]
 
         self.url = f'root:{self.root_password}@127.0.0.1:{self.mysql_port}'
 
+    @property
+    def http_connection_url(self) -> Optional[str]:
+        """HTTP connection URL for the SingleStoreDB server using Data API."""
+        if self.use_existing:
+            # If using existing server, HTTP URL not available from manager
+            return None
+        return (
+            f'singlestoredb+http://root:{self.root_password}@'
+            f'127.0.0.1:{self.http_port}'
+        )
+
     def _container_exists(self) -> bool:
         """Check if a container with this name already exists."""
         try:
@@ -169,11 +202,16 @@ def start(self) -> None:
             f'{self.http_port}, {self.studio_port}',
         )
         try:
-            license = os.environ['SINGLESTORE_LICENSE']
+            license = os.environ.get('SINGLESTORE_LICENSE', '')
             env = {
                 'SINGLESTORE_LICENSE': license,
             }
-            subprocess.check_call(command, shell=True, env=env)
+            # Capture output to avoid printing the container ID hash
+            subprocess.check_call(
+                command, shell=True, env=env,
+                stdout=subprocess.DEVNULL,
+            )
+
         except Exception as e:
             logger.exception(e)
             raise RuntimeError(
@@ -250,30 +288,51 @@ def stop(self) -> None:
         logger.info('Cleaning up SingleStore DB dev container')
         logger.debug('Stopping container')
         try:
-            subprocess.check_call(f'docker stop {self.container_name}', shell=True)
+            subprocess.check_call(
+                f'docker stop {self.container_name}',
+                shell=True,
+                stdout=subprocess.DEVNULL,
+            )
+
         except Exception as e:
             logger.exception(e)
             raise RuntimeError('Failed to stop container.') from e
 
         logger.debug('Removing container')
         try:
-            subprocess.check_call(f'docker rm {self.container_name}', shell=True)
+            subprocess.check_call(
+                f'docker rm {self.container_name}',
+                shell=True,
+                stdout=subprocess.DEVNULL,
+            )
+
         except Exception as e:
             logger.exception(e)
-            raise RuntimeError('Failed to stop container.') from e
+            raise RuntimeError('Failed to remove container.') from e
 
 
 @pytest.fixture(scope='session')
 def singlestoredb_test_container(
     execution_mode: ExecutionMode,
 ) -> Iterator[_TestContainerManager]:
-    """Sets up and tears down the test container"""
+    """Sets up and tears down the test container
+
+    If SINGLESTOREDB_URL is set in the environment, uses the existing server
+    and skips Docker container lifecycle management. Otherwise, automatically
+    starts a Docker container for testing.
+    """
 
     if not isinstance(execution_mode, ExecutionMode):
         raise TypeError(f"Invalid execution mode '{execution_mode}'")
 
     container_manager = _TestContainerManager()
 
+    # If using existing server, skip all Docker lifecycle management
+    if container_manager.use_existing:
+        logger.info('Using existing server, skipping Docker container lifecycle')
+        yield container_manager
+        return
+
     # In sequential operation do all the steps
     if execution_mode == ExecutionMode.SEQUENTIAL:
         logger.debug('Not distributed')