couchbase-lite-tests

This is the Couchbase Lite System Test Harness. It runs the system tests (tests requiring more than one device, e.g., a Couchbase Lite instance and a Sync Gateway) needed to verify a Couchbase Lite release.

The system consists of 4 components:

• An environment, built with docker compose, that contains one Couchbase Server and one Sync Gateway.
• The "client", a test harness written in python, that can configure the environment, run tests, and reports their results
• The tests, also written in python. Test are run within the client. They use it to configure the environment and then instruct servers to run tests in that environment.
• Per-platform "servers". The tests use the client instruct the server to run a specific test

What's here

• README.md - this file
• client - A python framework that configures the environment, runs tests and reports their results
• dataset - A collection datasets used in tests. Each dataset appears in two formats, as a cblite2 db, and as raw JSON
• environment - Scripts and such for setting up and tearing down a backend environment
• jenkins - The jenkins pipelines and the docker compose for running the Jenkins server.
• servers - A test server for each CBL platform. Servers run on their platform and accept instructions from the client to run tests
• spec - Documentation: The specification for this system.
• tests - The python codelets that are run in the client to configure an environment and then instruct connected servers to run a test in that environment.

Running tests

Requirements

1. Python 3.10+
2. OpenSSL 1.1 for CBS Python SDK
3. Git LFS

Environment Configuration

The tests use a configuration JSON file to get information about the environment in which they are running. As you might notice, there is a JSON Schema for it, so that will tell you more about the structure. Here is an example.

{
  "$schema": "https://packages.couchbase.com/couchbase-lite/testserver.schema.json",
  "api-version": 1,
  "test-servers": [
    {
      "url": "http://<url1>:8080"
    },
    {
      "url": "http://<url2>:8080"
    }
  ],
  "couchbase-servers": [
    {
      "hostname": "<url3>"
    }
  ],
  "sync-gateways": [
    {
      "hostname": "<url4>"
    }
  ]
}

This particular example indicates that there are two test servers running, along with one Sync Gateway and a Couchbase Server at the URLs provided. Normally you don't write this file yourself, but rather generate it using the orchestrator.

Steps for Running Test Diagnostically

1. Complete the prerequisites in the orchestrator.

2. Create a topology file and set up your backend environment (refer to the same README as 1). At this phase you have a decision to make. Either leave "download": true out of your test server entries in the topology file, or if you need to use an older version of the test server (to build with an older version of CBL), then make use of the prebuild script and the upload flag. You will need to set the environment variable LATESTBUILDS_PASSWORD appropriately in order to be able to write the resulting artifact. TIP: You don't need to do this unless you are curious, instead just make an ad-hoc run of the Jenkins job which does the same thing.

3. Using pytest, run the test you are interested in running (repeat as many times as you'd like)

4. Tip: You can access SGW logs by sending http requests to port 20000 on that machine (http://:20000/sg_debug.log for example)

5. Tip: If you have LogSlurp enabled, session.log will appear after a normal session finish containing the logs of all test servers and the TDK client interlaced.

Contributing

Notice that this repo has a .pre-commit-config.yaml file, which means it is ready to use with the pre-commit python tool. Essentially, after you clone this repo you should run

pip install pre-commit
pre-commit install

After that git pre-commit validation will check various things for you to ensure adherence to best practices and standards.