DoltLab Enterprise v2.3.12 - Availability, Reliability, and Scalability Overview

[coffeegoddd]

DoltLab Enterprise v2.3.12 - Availability, Reliability, and Scalability

DoltLab Enterprise v2.3.12 is deployed via Docker Compose on a single host, orchestrating multiple services that together power the platform. These services include:

• doltlabdb: the main Dolt server database instance.
• doltlabremoteapi: manages remote database operations (clone, push, fetch).
• doltlabapi: central API for user management, jobs, and coordination across services.
• doltlabfileserviceapi: handles disk-backed user-uploaded file storage.
• doltlabgraphql: GraphQL API consumed by the UI.
• doltlabui: React frontend web UI.
• doltlabenvoy: Envoy reverse proxy for routing internal service traffic.

Currently, the architecture introduces key limitations in availability, reliability, and scalability, largely due to its single-host nature and lack of redundancy.

Key Issues

Single Points of Failure

• Deployment: By default all services and data reside on a single host. If that host fails, the entire deployment goes down.
• Data Loss: Disk-backed Docker volumes present risk unless filesystem snapshots or external storage are used.

Data Persistence Strategies

• Default Configuration: All data resides on the host disk.
• Recommended Best Practice: RAID configuration and regular disk snapshots.
• Cloud-backed Storage Support:
  • doltlabremoteapi: Supports AWS S3 for remote database storage.
  • doltlabapi: Supports AWS S3 for user-uploaded files. When enabled, doltlabfileserviceapi is bypassed.
  • doltlabdb: Automated backups to cloud storage can be configured, or this service can be replaced with an externally hosted Dolt server or HostedDolt instance.

Service Replication & Availability

• No automatic service restart: Services do not restart on failure under current Docker Compose config.
• No replication or load balancing: All services are single-instance. Additionally, a failure in doltlabapi brings down the entire platform, and a failure in doltlabremoteapi prevents all remote operations.
• Multi-host mode lacks central proxy: No edge Envoy for intelligent routing or load balancing.

Service-Level Availability & Scalability

doltlabdb

• Single Dolt server instance.
• No clustering or failover.

doltlabremoteapi

• Handles all remote operations.
• Not independently scalable from the background tablefile serving process (unless cloud storage is used).

doltlabapi

• Central hub for API requests and job orchestration.
• No support for async-work offloading, which requires cloud-based message queue.
• Jobs always runs on the same host as doltlabapi, even in multi-host mode.

doltlabenvoy

• In single-host mode: central reverse proxy.
• In multi-host mode: no central edge proxy or intelligent routing.

[coffeegoddd]

As of DoltLab v2.4.0, DoltLab Enterprise multihost deployments are deployed and managed via Docker Swarm, which addresses a number of scaling issues present in earlier versions of DoltLab.

Now, service replication and availablitity is managed with Docker Swarm which will maintian the desired state of the deployment stack across nodes in the swarm cluster. There is no central proxy in DoltLab v2.4.0, since Docker Swarm load balances requests automatically to the doltlabenvoy replicas.

If cloud-backed storage is configured, all services except for doltlabdb can be scaled to multiple replicas. If cloud-backed storage is not configured, doltlabdb, doltlabfileserviceapi, and doltlabremoteapi are limited to a single replica instance to prevent concurrent writes resulting in data corruption.