Skip to content

Commit f0e0327

Browse files
Nao-risclaude
Nao-ris
and
claude
authored
Add Docker-based replication testing environment with Sync Gateway & Couchbase Server (#25)
## Overview This PR adds infrastructure for testing Couchbase Lite replication scenarios with a full stack (Sync Gateway + Couchbase Server) in a local Docker environment. Previously, testing replication required using the full billeo-engine setup, which is complex and heavy. This provides a lightweight alternative for development and testing. ## What's included ### Docker Infrastructure (`examples/docker-conf/`) - **Couchbase Server** with auto-configuration (cluster, bucket, users) - **Sync Gateway** with customizable sync function - Simple update scripts for modifying sync function and database config without restarting containers ### Utility Functions (`examples/utils/`) Reusable helper functions for testing replication scenarios: - **SGW admin operations**: user management, sessions, document operations, database lifecycle (`_offline`, `_online`, `_compact`, `_resync`) - **CBS admin operations**: bucket compaction, document queries, tombstone management ### Example (`examples/ticket_70596.rs`) Demonstrates auto-purge behavior when documents are moved to inaccessible channels. ## Usage ```bash # Start the stack cd examples/docker-conf docker compose up # Run the example cargo run --features=enterprise --example ticket_70596 ``` See `examples/README.md` for detailed instructions and configuration options. ## Future use cases This infrastructure can be used to test: - Sync function logic changes - Performance testing with realistic server setup - Any replication edge cases --------- Co-authored-by: Claude <[email protected]>
1 parent 4f68fdc commit f0e0327

File tree

19 files changed

+695
-7
lines changed

19 files changed

+695
-7
lines changed

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,4 @@ jobs:
2727
- name: Run tests with Couchbase Lite C leak check
2828
run: LEAK_CHECK=y cargo test --features=${{ matrix.version }} --verbose -- --test-threads 1
2929
- name: Run tests (with address sanitizer)
30-
run: LSAN_OPTIONS=suppressions=san.supp RUSTFLAGS="-Zsanitizer=address" cargo +nightly test --features=${{ matrix.version }} --verbose
30+
run: LSAN_OPTIONS=suppressions=san.supp RUSTFLAGS="-Zsanitizer=address" cargo +nightly test --target x86_64-unknown-linux-gnu --features=${{ matrix.version }} --verbose

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,5 @@ Cargo.lock
66

77
# MacOS directory conf
88
.DS_Store
9+
10+
*.cblite2/

‎Cargo.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,14 @@ enum_primitive = "0.1.1"
1717
[dev-dependencies]
1818
lazy_static = "1.5.0"
1919
regex = "1.11.1"
20+
serde_json = "1"
2021
tempdir = "0.3.7"
2122

23+
[dev-dependencies.reqwest]
24+
version = "0.12.15"
25+
default-features = false
26+
features = ["blocking", "json"]
27+
2228
[dev-dependencies.cargo-husky]
2329
version = "1"
2430
default-features = false # Disable features which are enabled by default

‎examples/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Running examples with Couchbase Sync Gateway & Server
2+
3+
Couchbase Lite is often used with replication to a central server, so it can be useful to test the full stack.
4+
The examples in this directory aim at covering these use cases.
5+
6+
## Setup the Couchbase Sync Gateway & Server
7+
8+
This process is handled through docker images, with as an entry point the file `docker-conf/docker-compose.yml`.
9+
10+
The configuration files that might interest you are:
11+
- `docker-conf/couchbase-server-dev/configure-server.sh` -> sets up the cluster, bucket and SG user
12+
- `docker-conf/db-config.json` -> contains the database configuration
13+
- `docker-conf/sync-function.js` -> contains the sync function used by the Sync Gateway
14+
15+
To start both the Sync Gateway and Couchbase Server, move to `docker-conf` through a terminal and use:
16+
17+
```shell
18+
$ docker-compose up
19+
```
20+
21+
It's very long the first time...
22+
23+
You can then access the Couchbase Server web ui through [http://localhost:8091](http://localhost:8091) (Chrome might not work, Firefox has better support).
24+
Make sure to not have another instance running.
25+
26+
## Update the config after startup
27+
28+
You can change a few things through the `curl` command.
29+
30+
#### Sync function
31+
32+
Update the file `docker-conf/sync-function.js` and run
33+
```shell
34+
$ curl -XPUT -v "http://localhost:4985/my-db/_config/sync" -H 'Content-Type: application/javascript' --data-binary @docker-conf/sync-function.js
35+
```
36+
37+
#### Database config
38+
39+
Update the file `docker-conf/db-config.json` and run
40+
41+
```shell
42+
$ curl -XPUT -v "http://localhost:4985/my-db/" -H 'Content-Type: application/json' --data-binary @docker-conf/db-config.json
43+
```
44+
45+
## Running an example
46+
47+
As of now, there is only one example: `ticket_70596`.
48+
49+
It can be run with the following command:
50+
```shell
51+
$ cargo run --features=enterprise --example ticket_70596
52+
```
53+
54+
There are utility functions available to interact with the Sync Gateway or Couchbase Server, feel free to add more if needed.

‎examples/docker-conf/couchbase-server-dev/Dockerfile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# hadolint ignore=DL3007
2+
FROM couchbase/server:enterprise
3+
COPY configure-server.sh /configure-server.sh
4+
5+
ENTRYPOINT ["/configure-server.sh"]

‎examples/docker-conf/couchbase-server-dev/configure-server.sh‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/usr/bin/env bash
2+
3+
export COUCHBASE_ADMINISTRATOR_USERNAME="cb_admin"
4+
export COUCHBASE_ADMINISTRATOR_PASSWORD="cb_admin_pwd"
5+
6+
export COUCHBASE_BUCKET="my-bucket"
7+
8+
export COUCHBASE_SG_USERNAME="syncgw"
9+
export COUCHBASE_SG_PASSWORD="syncgw-pwd"
10+
export COUCHBASE_SG_NAME="sg-service-user"
11+
12+
function retry() {
13+
for i in $(seq 1 10); do
14+
$1
15+
if [[ $? == 0 ]]; then
16+
return 0
17+
fi
18+
sleep 1
19+
done
20+
return 1
21+
}
22+
23+
function clusterInit() {
24+
couchbase-cli cluster-init \
25+
-c 127.0.0.1:8091 \
26+
--cluster-username $COUCHBASE_ADMINISTRATOR_USERNAME \
27+
--cluster-password $COUCHBASE_ADMINISTRATOR_PASSWORD \
28+
--services data,index,query \
29+
--cluster-ramsize 256 \
30+
--cluster-index-ramsize 256 \
31+
--index-storage-setting default
32+
if [[ $? != 0 ]]; then
33+
return 1
34+
fi
35+
}
36+
37+
function bucketCreate() {
38+
couchbase-cli bucket-create \
39+
-c 127.0.0.1:8091 \
40+
--username $COUCHBASE_ADMINISTRATOR_USERNAME \
41+
--password $COUCHBASE_ADMINISTRATOR_PASSWORD \
42+
--bucket-type=couchbase \
43+
--bucket-ramsize=100 \
44+
--bucket-replica=0 \
45+
--bucket $COUCHBASE_BUCKET \
46+
--wait
47+
if [[ $? != 0 ]]; then
48+
return 1
49+
fi
50+
}
51+
52+
function userSgCreate() {
53+
couchbase-cli user-manage \
54+
-c 127.0.0.1:8091 \
55+
--username $COUCHBASE_ADMINISTRATOR_USERNAME \
56+
--password $COUCHBASE_ADMINISTRATOR_PASSWORD \
57+
--set \
58+
--rbac-username $COUCHBASE_SG_USERNAME \
59+
--rbac-password $COUCHBASE_SG_PASSWORD \
60+
--rbac-name $COUCHBASE_SG_NAME \
61+
--roles bucket_full_access[*],bucket_admin[*] \
62+
--auth-domain local
63+
if [[ $? != 0 ]]; then
64+
return 1
65+
fi
66+
}
67+
68+
function main() {
69+
/entrypoint.sh couchbase-server &
70+
if [[ $? != 0 ]]; then
71+
echo "Couchbase startup failed. Exiting." >&2
72+
exit 1
73+
fi
74+
75+
# wait for service to come up
76+
until $(curl --output /dev/null --silent --head --fail http://localhost:8091); do
77+
sleep 5
78+
done
79+
80+
if couchbase-cli server-list -c 127.0.0.1:8091 --username $COUCHBASE_ADMINISTRATOR_USERNAME --password $COUCHBASE_ADMINISTRATOR_PASSWORD ; then
81+
echo "Couchbase already initialized, skipping initialization"
82+
else
83+
echo "Couchbase is not configured."
84+
echo
85+
86+
echo "Initializing the cluster...."
87+
retry clusterInit
88+
if [[ $? != 0 ]]; then
89+
echo "Cluster init failed. Exiting." >&2
90+
exit 1
91+
fi
92+
echo "Initializing the cluster [OK]"
93+
echo
94+
95+
echo "Creating the bucket...."
96+
retry bucketCreate
97+
if [[ $? != 0 ]]; then
98+
echo "Bucket create failed. Exiting." >&2
99+
exit 1
100+
fi
101+
echo "Creating the bucket [OK]"
102+
echo
103+
104+
echo "Creating Sync Gateway user...."
105+
retry userSgCreate
106+
if [[ $? != 0 ]]; then
107+
echo "User create failed. Exiting." >&2
108+
exit 1
109+
fi
110+
echo "Creating Sync Gateway user [OK]"
111+
echo
112+
113+
sleep 10
114+
115+
fi
116+
117+
wait
118+
}
119+
120+
main
121+

‎examples/docker-conf/db-config.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"import_docs": true,
3+
"enable_shared_bucket_access": true,
4+
"bucket": "my-bucket",
5+
"num_index_replicas": 0,
6+
"revs_limit": 20,
7+
"allow_conflicts": false
8+
}

‎examples/docker-conf/docker-compose.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
services:
2+
cblr-couchbase-server:
3+
ports:
4+
- "8091:8091" # REST(admin), Web console
5+
- "8093:8093" # Query service REST/HTTP traffic
6+
- "11207:11207" # memcached port (TLS)
7+
- "11210:11210" # memcached port
8+
build:
9+
context: ${PWD}/couchbase-server-dev
10+
deploy:
11+
resources:
12+
limits:
13+
memory: 2048M
14+
restart: on-failure
15+
cblr-sync-gateway:
16+
image: couchbase/sync-gateway:enterprise
17+
ports:
18+
- "4984:4984"
19+
- "4985:4985"
20+
deploy:
21+
resources:
22+
limits:
23+
memory: 512M
24+
healthcheck:
25+
test: ["CMD", "curl", "-f", "http://localhost:4985"]
26+
interval: 30s
27+
timeout: 10s
28+
retries: 5
29+
volumes:
30+
- ${PWD}/syncgw-config.json:/etc/sync_gateway/config.json:ro
31+
- ${PWD}/wait-for-couchbase-server.sh:/wait-for-couchbase-server.sh
32+
depends_on:
33+
- cblr-couchbase-server
34+
entrypoint: ["/wait-for-couchbase-server.sh"]
35+
restart: on-failure
36+
cblr-sync-gateway-setup:
37+
image: alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
38+
depends_on:
39+
cblr-sync-gateway:
40+
condition: service_healthy
41+
volumes:
42+
- ${PWD}/sync-function.js:/sync-function.js
43+
- ${PWD}/db-config.json:/db-config.json
44+
- ${PWD}/update.sh:/update.sh
45+
links:
46+
- "cblr-sync-gateway:sg"
47+
entrypoint: /update.sh

‎examples/docker-conf/sync-function.js‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
function sync(doc, oldDoc, meta) {
2+
console.log("=== New document revision ===");
3+
console.log("New doc:");
4+
console.log(doc);
5+
console.log("Old doc:");
6+
console.log(oldDoc);
7+
console.log("Metadata:");
8+
console.log(meta);
9+
10+
if(doc.channels) {
11+
channel(doc.channels);
12+
}
13+
if(doc.expiry) {
14+
// Format: "2022-06-23T05:00:00+01:00"
15+
expiry(doc.expiry);
16+
}
17+
18+
console.log("=== Document processed ===");
19+
}

‎examples/docker-conf/syncgw-config.json‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"bootstrap": {
3+
"server": "couchbase://cblr-couchbase-server",
4+
"username": "syncgw",
5+
"password": "syncgw-pwd"
6+
},
7+
"api": {
8+
"public_interface": ":4984",
9+
"admin_interface": ":4985",
10+
"admin_interface_authentication": false,
11+
"https": {}
12+
},
13+
"logging": {
14+
"console": {
15+
"rotation": {},
16+
"log_level": "debug",
17+
"log_keys": [
18+
"*"
19+
]
20+
},
21+
"error": {
22+
"rotation": {}
23+
},
24+
"warn": {
25+
"rotation": {}
26+
},
27+
"info": {
28+
"rotation": {}
29+
},
30+
"debug": {
31+
"rotation": {}
32+
},
33+
"trace": {
34+
"rotation": {}
35+
},
36+
"stats": {
37+
"rotation": {}
38+
}
39+
},
40+
"auth": {},
41+
"replicator": {},
42+
"unsupported": {}
43+
}

0 commit comments

Comments
 (0)