-
Notifications
You must be signed in to change notification settings - Fork 7
ING-1216: Add tuneable documentation and basic project docs structure. #282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Documentation | ||
|
|
||
| Documentation here is a developer level view of concepts and how the system is implemented. | ||
| While system operators and end users may benefit from this information, there are other docs which are more relevant to those audiences. | ||
|
|
||
| In particular: | ||
| * For Kubernetes Operator/OpenShift deployments, see the [CAO Documentation on CNG](https://docs.couchbase.com/operator/current/concept-cloud-native-gateway.html) | ||
| * For Capella Data API, see the [Runbook for Data API for Provisioned Clusters](https://confluence.issues.couchbase.com/wiki/x/S4ACv) | ||
|
|
||
| ## Concepts | ||
|
|
||
| **Stellar Gateway** is a Couchbase Protostellar implementation that provides both a gRPC "protostellar" and a RESTful interface to Couchbase. It is deployed 1:1 with a cluster, but may be deployed different ways. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Stellar Gateway is a service that provides both a gRPC "protostellar" implementation and a RESTful interface to Couchbase. |
||
|
|
||
| ### Architecture Overview | ||
|
|
||
| The system uses a modular architecture where each service (data, service discovery, Data API) can be independently configured and scaled. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sentence is a slightly confusing statement, it seems to be mixing the concept of scalability of the cluster with the scalability of CNG itself. |
||
|
|
||
| ### Main Entry Points | ||
| - **`cmd/gateway/main.go`** - Main gateway service entry point with CLI interface | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These should be limited to just |
||
| - **`cmd/dev/main.go`** - Development environment setup | ||
| - **`cmd/bridge/main.go`** - Legacy bridge service (deprecated, never deployed) | ||
|
|
||
| ### Core Modules | ||
|
|
||
| #### **Gateway Core (`gateway/`)** | ||
| - **`gateway.go`** - Main Gateway struct and orchestration logic | ||
| - **`system/`** - System-level server management (gRPC, HTTP, Data API servers) | ||
| - **`clustering/`** - Cluster membership and service discovery | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. While "cluster membership and service discovery" is what this folder does, this statement is a bit confusing without mentioning the distinction between CNG service discovery and cluster service discovery. This particular folder may also be deprecated in the future, since we don't currently have a fully implemented implementation of CNG service discovery. |
||
| - **`topology/`** - Network topology management | ||
|
|
||
| #### **Protocol Implementations** | ||
| - **`dataimpl/`** - Protostellar data API service (KV, Query, Search) implementation | ||
| - **`sdimpl/`** - Service discovery implementation | ||
| - **`dapiimpl/`** - Data API v1 HTTP REST implementation with proxy services | ||
| - **`legacybridge/`** - Legacy protocol bridge for backwards compatibility | ||
|
|
||
| #### **Client** | ||
| - **`client/`** - Client front end for connecting to clusters | ||
| - Connection management, routing, topology watching | ||
|
|
||
| #### **Data API Specification** | ||
| - **`dataapiv1/`** - OpenAPI spec and generated code for REST Data API | ||
| - Auto-generated from `spec.yaml` using oapi-codegen | ||
|
|
||
| #### **Infrastructure Modules** | ||
| - **`auth/`** - Authentication (cbauth integration) | ||
| - **`ratelimiting/`** - Request rate limiting | ||
| - **`hooks/`** - Lifecycle hooks and barriers | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The hooks library enables hooking the GRPC implementation in various ways for testing. It's "testing hooks" rather than "lifecycle hooks". |
||
| - **`apiversion/`** - API versioning and gRPC interceptors | ||
|
|
||
| #### **Contrib/Utilities** | ||
| - **`contrib/`** - External components (etcd, goclustering, protostellar extensions) | ||
| - **`utils/`** - Utility packages (TLS, secrets management, networking) | ||
| - **`pkg/`** - Shared packages (metrics, interceptors, web API) | ||
|
|
||
| ### Public Interfaces | ||
|
|
||
| 1. **gRPC Protostellar API** - Main protocol interface (port 18098/18099) | ||
| 2. **Data API REST** - HTTP REST API (configurable port, spec in `dataapiv1/`) | ||
| 3. **Web/Metrics API** - Health checks and metrics (port 9091) | ||
| 4. **Client Library** - Go client in `client/` package | ||
|
|
||
| ### Key Features | ||
| - Multiple deployment modes (standalone, clustered) | ||
| - TLS support with configurable certificates | ||
| - Rate limiting and request throttling | ||
| - OpenTelemetry integration for observability | ||
| - Cloud provider secrets integration (AWS, Azure, GCP) | ||
| - Graceful shutdown and configuration reloading | ||
|
|
||
| ## Detailed Documentation | ||
|
|
||
| * [Tuneables](tuneables.md) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Tuneables | ||
|
|
||
| ## Rate Limiting | ||
|
|
||
| In a Capella deployment, we wish to have a rate limit as a simple method of avoiding accidental overload, resource exhaustion and pathological system failure. | ||
| Comprehensive DDoS prevention is more complex and outside the scope of what Data API can provide, but this feature provides some reasonable limit. | ||
|
|
||
| **Field:** `rate-limit` | ||
|
|
||
| **Format:** Integer, unsigned | ||
|
|
||
| **Default:** 0 (unlimited). Capella deployments default this to 10000. | ||
|
|
||
| **Description:** A per-process limit which, when exceeded, will return HTTP 429 errors. | ||
|
|
||
| This is service independent and resource usage/complexity of underlying services called may not align. For example, KV operations with options like durability, complex queries, and FTS requests may exhaust resources before this rate limit is hit. | ||
|
|
||
| **When to adjust:** | ||
| - If a deployment wants to be more conservative on Data API or CNG resource usage, this may be tuned lower which will reject requests to keep load lower. | ||
| - If a deployment wants to allow more resource usage, increase this to a higher level or to the value 0 to be unlimited. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This sentence starts a bit strange, missing a "The" perhaps?