Commit 8edd129
CLOUDP-392735: Search: support multiple mongots and envoy load balancer (#817)
<!-- start git-machete generated -->
# Based on PR #806
## Chain of upstream PRs as of 2026-03-23
* PR #806:
`master` <- `search/base`
* **PR #817 (THIS ONE)**:
`search/base` <- `search/multiple-mongot`
<!-- end git-machete generated -->
# Summary
This PR adds support for **multiple mongot instances** per MongoDB
source, with L7 load balancing and sharded cluster integration. It
builds on `search/base` (#806) and introduces three major capabilities:
1. **Multiple mongot replicas** with managed (Envoy) or unmanaged (BYO)
load balancing
2. **Sharded cluster support** for both operator-managed and external
MongoDB
3. **X509 client certificate authentication** for mongot-to-MongoDB sync
source connections
4. **JVM flags** for mongot process configuration
---
## API/CRD Changes
The `MongoDBSearch` CRD gains the following new fields:
```yaml
spec:
# New: Load balancer configuration (exactly one of managed/unmanaged must be set)
loadBalancer: # was: spec.lb with mode enum
managed: # operator-deployed Envoy proxy
externalHostname: "..." # SNI hostname, supports {shardName} placeholder
resourceRequirements: { ... } # Envoy container resources (default: 100m/128Mi req, 500m/512Mi lim)
deployment: { ... } # Envoy Deployment overrides (same convention as spec.statefulSet)
unmanaged: # BYO L7 load balancer
endpoint: "lb.example.com:27029" # supports {shardName} placeholder for sharded
# New: X509 client certificate auth for sync source (mutually exclusive with username/password)
source:
x509:
clientCertificateSecretRef:
name: "mongot-x509-cert" # Secret with tls.crt, tls.key (required), tls.keyFilePassword (optional)
# New: JVM flags for mongot
jvmFlags: ["-Xms2g", "-Xmx2g"]
# New: External sharded cluster source
source:
external:
shardedCluster:
router:
hosts: ["mongos-0.example.com:27017"]
shards:
- shardName: "shard-0"
hosts: ["shard0-rs0.example.com:27017"]
- shardName: "shard-1"
hosts: ["shard1-rs0.example.com:27017"]
```
Key structural change: `spec.lb.mode` (Managed/Unmanaged enum) was
replaced with `spec.loadBalancer.managed` /
`spec.loadBalancer.unmanaged` (mutually exclusive sub-objects, enforced
via CEL validation).
---
## Resource Naming Conventions
All search resource names include a hardcoded `-0-` cluster index to
reserve the naming scheme for future multi-cluster support.
### ReplicaSet (Non-Sharded) Resources
| Resource | Name Pattern |
|----------|-------------|
| StatefulSet | `{name}-search` |
| Headless Service | `{name}-search-svc` |
| ConfigMap | `{name}-search-config` |
| TLS Secret | `[{prefix}-]{name}-search-0-cert` |
| TLS Client Secret | `[{prefix}-]{name}-search-0-client-cert` |
| X509 Client Cert Secret | `{name}-x509-client-cert` |
| Proxy Service (Envoy) | `{name}-search-0-proxy-svc` |
| LB Deployment | `{name}-search-lb-0` |
| LB ConfigMap | `{name}-search-lb-0-config` |
| LB Server Cert | `[{prefix}-]{name}-search-lb-0-cert` |
| LB Client Cert | `[{prefix}-]{name}-search-lb-0-client-cert` |
### Sharded Resources (Per-Shard)
| Resource | Name Pattern |
|----------|-------------|
| StatefulSet | `{name}-search-0-{shard}` |
| Headless Service | `{name}-search-0-{shard}-svc` |
| ConfigMap | `{name}-search-0-{shard}-config` |
| TLS Secret | `[{prefix}-]{name}-search-0-{shard}-cert` |
| Proxy Service (Envoy) | `{name}-search-0-{shard}-proxy-svc` |
| LB Server Cert (per-shard) |
`[{prefix}-]{name}-search-lb-0-{shard}-cert` |
---
## Architecture
### Reconciliation Paths
The `MongoDBSearchReconcileHelper` dispatches to:
- **`reconcileNonSharded`**: Creates a single set of resources (1
StatefulSet, 1 Service, 1 ConfigMap). All mongot pods connect to the
same RS host seeds.
- **`reconcileSharded`**: Creates per-shard resources -- one
StatefulSet, Service, ConfigMap, and TLS secret per shard. Each shard's
mongot connects to that shard's mongod hosts. A `Router` section in the
mongot config points to mongos for query routing.
### MongoDBSearchEnvoyReconciler (New Controller)
A dedicated controller that manages the Envoy proxy infrastructure when
`spec.loadBalancer.managed` is set:
- **ReplicaSet**: 1 Envoy Deployment + 1 ConfigMap + 1 ClusterIP proxy
Service
- **Sharded**: 1 Envoy Deployment + 1 ConfigMap + N ClusterIP proxy
Services (one per shard)
For sharded clusters, all shard routes are multiplexed through a
**single Envoy deployment** using SNI-based routing. Each shard gets a
dedicated proxy Service that resolves to the same Envoy pods. Envoy
reads the SNI hostname from the TLS ClientHello to route traffic to the
correct shard's mongot backend.
The controller:
- Watches `MongoDBSearch`, `MongoDB`, and `MongoDBCommunity` resources
- Owns Deployment and ConfigMap resources
- Returns early with `Pending` status if no routes can be configured
- Updates LB sub-status on the MongoDBSearch CR at every return path
- Uses TLS 1.3 exclusively for Envoy-to-mongot connections
### X509 Client Certificate Authentication
When `spec.source.x509` is configured, mongot authenticates to the
MongoDB sync source using x509 client certificates instead of
username/password:
- **Mutually exclusive** with `spec.source.passwordSecretRef` and
`spec.source.username` (enforced by validation)
- **Requires TLS** to be enabled on the sync source
- The operator reads the user-provided Secret (containing `tls.crt`,
`tls.key`, and optionally `tls.keyFilePassword`), combines them into an
operator-managed Secret (`{name}-x509-client-cert`), and mounts it into
the mongot container
- Mongot config is modified to clear username/password fields and set
`authSource: $external` with `x509.tlsCertificateKeyFile` pointing to
the mounted cert
- For sharded clusters, x509 config is applied to both the ReplicaSet
sync source and the Router section
- Optional key password support: if `tls.keyFilePassword` is present in
the Secret, it is mounted separately and referenced via
`tlsCertificateKeyFilePasswordFile`
The `x509AuthResource` adapter implements the `TLSConfigurableResource`
interface, allowing reuse of the existing `tls.EnsureTLSSecret`
infrastructure.
### Endpoint Resolution
How `mongod.setParameter.mongotHost` resolves for each topology + LB
mode:
| Topology | LB Mode | mongotHost |
|----------|---------|------------|
| RS | None (single replica) |
`{name}-search-svc.{ns}.svc.{domain}:27028` |
| RS | Unmanaged | User-provided `spec.loadBalancer.unmanaged.endpoint`
|
| RS | Managed | `{name}-search-0-proxy-svc.{ns}.svc.{domain}:27029` |
| Sharded | None | `{name}-search-0-{shard}-svc.{ns}.svc.{domain}:27028`
(per shard) |
| Sharded | Unmanaged | Endpoint template with `{shardName}` substituted
per shard |
| Sharded | Managed |
`{name}-search-0-{shard}-proxy-svc.{ns}.svc.{domain}:27029` (per shard)
|
### Sharded Controller Integration
The `mongodbshardedcluster_controller` is extended to watch
MongoDBSearch resources via `lookupCorrespondingSearchResource()`. It
applies search config to each shard via
`applySearchParametersForShards()`:
- Per-shard mongod config points to each shard's own mongot endpoint
- Mongos config gets `mongotHost` pointing to the first shard's endpoint
---
## Test Coverage
### E2E Tests (Python)
| # | Topology | Source | Mongot Count | LB Mode | Test File |
|---|----------|--------|-------------|---------|-----------|
| 1 | RS | External | Single | None |
`search_replicaset_external_mongodb_single_mongot.py` |
| 2 | RS | External | Multi | Unmanaged |
`search_replicaset_external_mongodb_multi_mongot_unmanaged_lb.py` |
| 3 | RS | Internal | Multi | Unmanaged |
`search_replicaset_internal_mongodb_multi_mongot_unmanaged_lb.py` |
| 4 | RS | External | Multi | Managed |
`search_replicaset_external_mongodb_multi_mongot_managed_lb.py` |
| 5 | RS | Internal | Multi | Managed |
`search_replicaset_internal_mongodb_multi_mongot_managed_lb.py` |
| 6 | RS | External | - | Proxy Svc |
`search_replicaset_external_mongodb_proxy_service.py` |
| 7 | RS | - | - | X509 | `search_mongot_replicaset_x509_auth.py` |
| 8 | RS | Community | Multi | Managed |
`search_community_auto_embedding_multi_mongot.py` |
| 9 | Sharded | Internal | Single | None |
`search_sharded_internal_mongodb_single_mongot.py` |
| 10 | Sharded | External | Single | None |
`search_sharded_external_mongodb_single_mongot.py` |
| 11 | Sharded | Internal | Multi | Unmanaged |
`search_sharded_internal_mongodb_multi_mongot_unmanaged_lb.py` |
| 12 | Sharded | External | Multi | Unmanaged |
`search_sharded_external_mongodb_multi_mongot_unmanaged_lb.py` |
| 13 | Sharded | Internal | Multi | Managed |
`search_sharded_internal_mongodb_multi_mongot_managed_lb.py` |
| 14 | Sharded | External (Ent) | Multi | Managed |
`search_sharded_enterprise_external_mongod_managed_lb.py` |
### Unit Tests (Go)
| File | Lines | Coverage |
|------|-------|----------|
| `mongodbsearch_reconcile_helper_test.go` | 2,313 | Per-shard
mongod/mongos config, LB endpoint resolution, validation |
| `sharded_external_search_source_test.go` | 452 | External sharded
source, shard name validation, host lists |
| `enterprise_search_source_test.go` | 373 | Enterprise RS source, error
returns (no panic) |
| `envoy_config_builder_test.go` | 318 | Envoy JSON generation,
TLS/non-TLS, single/multi route |
| `mongodbsearch_validation_test.go` | 300 | Shard names, X509 auth
config |
| `mongodbsearchenvoy_controller_test.go` | 255 | Route building, pod
spec, deployment overrides |
| `mongodbsearch_validation_test.go` (api/) | 212 | JVM flags, LB
config, endpoint template |
| `search_construction_test.go` | 209 | StatefulSet construction, JVM
flags generation |
| `mongodbsearch_types_test.go` | 132 | Resource name generation |
---
## Other Changes
- **Error handling**: `panic()` calls in `HostSeeds()` implementations
replaced with `fmt.Errorf` returns across `enterprise_search_source.go`,
`external_search_source.go`, `community_search_source.go`
- **Keyfile handling**: Extracted `ensureKeyfileModification()` helper
to deduplicate keyfile blocks between `reconcileNonSharded` and
`reconcileSharded`
- **Endpoint resolution**: Extracted `mongotEndpointForShard()` helper
for consistent endpoint resolution across managed/unmanaged/direct modes
- **File renames**: `sharded_enterprise_search_source.go` renamed to
`sharded_internal_search_source.go` (struct
`ShardedInternalSearchSource`)
- **Changelog**: Consolidated all search changes into a single entry
(`20260318_feature_mongodbsearch_improvements.md`)
- **RBAC**: Added `deployments` permission to `apps` API group for Envoy
Deployment management
- **Helm chart**: New `MDB_ENVOY_IMAGE` env var for configuring the
Envoy container image
- **Readiness probe** for mongot containers
- **Auto-heap sizing**: JVM `-Xms`/`-Xmx` set to 50% of memory request
when not explicitly provided
- **Duplicate function consolidation**: `verify_rs_mongod_parameters`
consolidated into `replicaset_search_helper.py`
- **IsShardedEndpoint()**: Extracted helper method for checking sharded
endpoint configuration
- **Status updates**: `updateLBStatus()` patches LB sub-status on
MongoDBSearch CR at every reconcile return path
- **Test infrastructure**: `SearchDeploymentHelper`, `SearchTester`
factories, `search_resource_names` utilities, `ToolsPod` for in-cluster
mongorestore
---
## Known Limitations / Follow-ups
- **Envoy log level**: Currently hardcoded to `"info"` -- needs a
configurable field in `ManagedLBConfig`
- **`rs_search_helper.py`**: Should be consolidated into
`replicaset_search_helper.py` and removed
- **Reconcile loop tests**: `mongodbsearchenvoy_controller_test.go`
covers route building and pod spec but lacks full reconcile loop
integration tests
- **Test file naming**: Some test files don't follow a consistent naming
convention from the test matrix
## Checklist
- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you added changelog file?
- use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in [CONTRIBUTING.md](http://CONTRIBUTING.md) for more details
---------
Co-authored-by: Julien Benhaim <julien.benhaim@mongodb.com>
Co-authored-by: Vivek Singh <vsingh.ggits.2010@gmail.com>
Co-authored-by: Julien-Ben <33035980+Julien-Ben@users.noreply.github.com>
Co-authored-by: Anand <13899132+anandsyncs@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Vivek Singh <vivek.s@mongodb.com>1 parent c23b484 commit 8edd129
File tree
116 files changed
+12257
-1213
lines changed- api/v1
- mdb
- search
- user
- changelog
- config
- crd/bases
- manager
- rbac
- controllers
- operator
- searchcontroller
- docker/mongodb-kubernetes-tests
- kubetester
- tests
- common
- mongodb_tools_pod
- search
- multicluster
- replicaset
- search
- fixtures
- shardedcluster
- standalone
- docs/search
- 01-search-community-deploy/code_snippets
- 02-search-enterprise-deploy/code_snippets
- helm_chart
- crds
- templates
- mongodb-community-operator
- api/v1/common
- pkg
- mongot
- util/merge
- pkg
- kubectl-mongodb/common
- util
- public
- samples/multi-cluster-cli-gitops/resources/rbac
- scripts
- dev
- contexts
- variables
- prepare-multi-cluster
- evergreen/e2e
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
116 files changed
+12257
-1213
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1325 | 1325 | | |
1326 | 1326 | | |
1327 | 1327 | | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
1328 | 1333 | | |
1329 | 1334 | | |
1330 | 1335 | | |
1331 | 1336 | | |
1332 | 1337 | | |
1333 | | - | |
| 1338 | + | |
1334 | 1339 | | |
1335 | 1340 | | |
1336 | 1341 | | |
1337 | 1342 | | |
1338 | | - | |
| 1343 | + | |
1339 | 1344 | | |
1340 | 1345 | | |
1341 | 1346 | | |
| |||
1355 | 1360 | | |
1356 | 1361 | | |
1357 | 1362 | | |
1358 | | - | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
1359 | 1423 | | |
1360 | 1424 | | |
1361 | 1425 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
669 | 669 | | |
670 | 670 | | |
671 | 671 | | |
672 | | - | |
673 | | - | |
674 | | - | |
675 | | - | |
676 | | - | |
677 | 672 | | |
678 | 673 | | |
679 | 674 | | |
| |||
806 | 801 | | |
807 | 802 | | |
808 | 803 | | |
809 | | - | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
810 | 819 | | |
| 820 | + | |
| 821 | + | |
811 | 822 | | |
812 | 823 | | |
813 | | - | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
814 | 883 | | |
815 | 884 | | |
816 | 885 | | |
| |||
1241 | 1310 | | |
1242 | 1311 | | |
1243 | 1312 | | |
1244 | | - | |
1245 | | - | |
1246 | 1313 | | |
1247 | 1314 | | |
1248 | 1315 | | |
| |||
1361 | 1428 | | |
1362 | 1429 | | |
1363 | 1430 | | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
1364 | 1451 | | |
1365 | 1452 | | |
1366 | 1453 | | |
| |||
1470 | 1557 | | |
1471 | 1558 | | |
1472 | 1559 | | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
| 1571 | + | |
| 1572 | + | |
| 1573 | + | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
1473 | 1580 | | |
1474 | 1581 | | |
1475 | 1582 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
132 | 142 | | |
133 | 143 | | |
134 | 144 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| 9 | + | |
8 | 10 | | |
9 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
10 | 15 | | |
11 | 16 | | |
12 | 17 | | |
| |||
74 | 79 | | |
75 | 80 | | |
76 | 81 | | |
77 | | - | |
| 82 | + | |
78 | 83 | | |
79 | 84 | | |
80 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1678 | 1678 | | |
1679 | 1679 | | |
1680 | 1680 | | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
1681 | 1689 | | |
1682 | 1690 | | |
1683 | 1691 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
419 | 419 | | |
420 | 420 | | |
421 | 421 | | |
422 | | - | |
| 422 | + | |
423 | 423 | | |
424 | 424 | | |
425 | 425 | | |
| |||
0 commit comments