Skip to content

artifacts_test.md

Latest commit

 

History

History
277 lines (192 loc) · 9.65 KB

artifacts_test.md

File metadata and controls

277 lines (192 loc) · 9.65 KB

What are artifacts tests?

Artifacts tests are kind of smoke tests which verify that packaging artifacts like .rpm/.deb files, cloud or container images produced by ScyllaDB builds can be deployed successfully.

Private repo test is also kind of artifacts test.

Jenkins pipelines

Artifacts tests triggered by another projects which build .rpm's and .deb's. E.g., tests for CentOS triggered by centos-rpm project.

There are pipeline files for Jenkins in the repository:

vars
`-- artifactsPipeline.groovy
jenkins-pipelines
|-- artifacts-ami.jenkinsfile
|-- artifacts-centos7.jenkinsfile
|-- artifacts-debian10.jenkinsfile
|-- artifacts-debian11.jenkinsfile
|-- artifacts-docker.jenkinsfile
|-- artifacts-oel76.jenkinsfile
|-- artifacts-ubuntu2004.jenkinsfile
`-- private-repo.jenkinsfile

vars/artifactsPipeline.groovy is a pipeline call definition which used to run same pipeline with different parameters.

You should use different parameters for .rpm/.deb tests and for AMI test:

  • The only required parameter for .rpm/.deb jobs is scylla_repo, a path to a ScyllaDB repo (e.g., https://s3.amazonaws.com/downloads.scylladb.com/rpm/unstable/centos/master/latest/scylla.repo)
  • For AMI job you need two parameters: scylla_ami_id and region_name
  • For Docker job you need two parameters: scylla_docker_image (e.g., scylladb/scylla or scylladb/scylla-nightly) and scylla_version (it'll be used as Docker image tag)

To verify Scylla Manager package you need to provide scylla_repo and scylla_mgmt_address parameters. In this case Scylla Manager will be installed during artifacts test as well as ScyllaDB.

Alternatively, and even recommended, one can use the manager_version parameter to just choose the desired version.

Optionally, you can override default cloud instance type by providing instance_type parameter. This parameter can be a space-separated list of instance types.

artifacts-*.jenkinsfile files can be used to create Jenkins projects.

private-repo.jenkinsfile can be used to create a Jenkins project which verifies private repo correctness. You need to provide scylla_repo parameter which points to a private repo.

How to run artifacts tests using hydra

If you want to run it locally you should use ip_ssh_connections: 'public':

export SCT_IP_SSH_CONNECTIONS=public

and run one of the following commands:

CentOS 7

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/centos7.yaml

CentOS 8

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/centos8.yaml

Debian 10 (buster)

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/debian10.yaml

Debian 11 (bullseye)

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/debian11.yaml

RHEL 7

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/rhel7.yaml

RHEL 8

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/rhel8.yaml

Oracle Enterprise Linux 7.6

hydra run-test artifacts_test --backend aws --config test-cases/artifacts/oel76.yaml

Ubuntu 22.04 LTS (jammy)

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/ubuntu2204.yaml

Ubuntu 24.04 LTS (noble)

hydra run-test artifacts_test --backend gce --config test-cases/artifacts/ubuntu2404.yaml

AMI

You can use hydra to find Scylla AMI in the desired region (us-east-1 in example below):

hydra list-ami-versions -r us-east-1

and run the test:

SCT_AMI_ID_DB_SCYLLA="<ami you've found>" SCT_REGION_NAME=us-east-1 hydra run-test artifacts_test --backend aws --config test-cases/artifacts/ami.yaml

Docker

If you want to run against some official ScyllaDB Docker image, you should go to tag list and choose some tag name (i.e., 3.3.rc1)

SCT_SCYLLA_VERSION="<tag you've chose>" hydra run-test artifacts_test --backend docker --config test-cases/artifacts/docker.yaml

Scylla Doctor Version Management

Overview

Artifact tests use Scylla Doctor to validate the health of deployed Scylla instances. To ensure stability and avoid P0 issues from new Scylla Doctor releases, the version is hardcoded in the configuration rather than always using the latest version.

The configured version is used for both:

  • Offline installations (downloading from S3)
  • Package manager installations (apt/yum/dnf)

Current Configuration

The Scylla Doctor version is configured in defaults/test_default.yaml:

scylla_doctor_version: "1.10"

This version is used for all artifact tests by default.

Updating Scylla Doctor Version

When to Update

  • When a new Scylla Doctor version has been released with important fixes or features
  • After thorough validation that the new version works correctly with all test scenarios
  • When blocking issues in the current version require an upgrade

Update Process

  1. Change the version in configuration:

    Edit defaults/test_default.yaml and update the version:

    scylla_doctor_version: "1.10"  # Update to new version

  2. Run validation tests:

    You must run comprehensive artifact tests to verify the new version works correctly:

    # Set Scylla version for testing
export SCT_SCYLLA_VERSION=master:latest

# Test on different operating systems
hydra run-test artifacts_test --backend aws --config test-cases/artifacts/ami.yaml
hydra run-test artifacts_test --backend gce --config test-cases/artifacts/centos9.yaml
hydra run-test artifacts_test --backend gce --config test-cases/artifacts/ubuntu2204.yaml
hydra run-test artifacts_test --backend gce --config test-cases/artifacts/debian12.yaml
hydra run-test artifacts_test --backend docker --config test-cases/artifacts/docker.yaml

# Test offline installations
hydra run-test artifacts_test --backend gce --config test-cases/artifacts/rocky9.yaml

  3. Verify the tests pass:

    All artifact tests must complete successfully with the new Scylla Doctor version. Pay special attention to:

    • Scylla Doctor installation succeeds
    • All collectors run without errors
    • Vitals JSON file is generated correctly
    • No new false-positive alerts are reported

  4. Create a PR with the version update:

    Include in the PR description:

    • The new Scylla Doctor version
    • Test results showing successful validation
    • Any known issues or workarounds for the new version
    • Link to Scylla Doctor release notes if available

Backporting Version Updates

If a version update needs to be backported to a stable branch:

  1. Tag the original PR for backporting:

    Add the appropriate backport label(s) to the original PR (e.g., backport/branch-5.4, backport/branch-2024.2). The backport PR will be created automatically by the CI system.

  2. Manual testing is required:

    Once the backport PR is created automatically, you must manually run validation tests on the stable branch to ensure the version works correctly with the Scylla versions tested on that branch.

    # Set appropriate Scylla version for the target branch
export SCT_SCYLLA_VERSION=<branch-version>:latest

# Run the same validation tests
hydra run-test artifacts_test --backend aws --config test-cases/artifacts/ami.yaml
hydra run-test artifacts_test --backend gce --config test-cases/artifacts/centos9.yaml
# ... (other test commands)

  3. Verify and merge the backport PR:

    After successful manual testing, review and merge the automatically created backport PR.

Testing Checklist

Before merging any Scylla Doctor version update, ensure:

  • Version updated in defaults/test_default.yaml
  • Artifact tests pass on at least 3 different operating systems (CentOS/Rocky, Ubuntu, Debian)
  • Docker artifact test passes
  • AMI artifact test passes on AWS
  • GCE image artifact test passes
  • Offline installation tests pass
  • No new false-positive alerts in test results
  • PR includes test results and validation summary

Troubleshooting

Package Not Found

If you get an error like "Unable to find scylla-doctor package for version X.Y":

  1. Verify the version exists in S3 bucket: downloads.scylladb.com/downloads/scylla-doctor/tar/
  2. Check the exact version string format (e.g., "1.9" vs "1.9.0")
  3. Ensure you have AWS credentials configured for S3 access

Version Mismatch

If the wrong version is being used:

  1. Check that scylla_doctor_version is set in your test configuration
  2. Verify the configuration is not overridden by environment variable SCT_SCYLLA_DOCTOR_VERSION
  3. Check the test logs for the actual version being downloaded

Collector Failures

If new collectors fail in the updated version:

  1. Review the Scylla Doctor release notes for breaking changes
  2. Report JIRA issues for any collectors that are not working properly
    • Create issues in the scylla-doctor project describing the failure
    • Include test logs and error messages
    • Tag with appropriate priority and affected version
  3. Do not disable collectors - wait for the JIRA issues to be resolved before updating the scylla-doctor version
  4. Only proceed with the version update after all collector issues are fixed

Override for Testing

To test a specific Scylla Doctor version without changing defaults:

export SCT_SCYLLA_DOCTOR_VERSION="1.10"
hydra run-test artifacts_test --backend docker --config test-cases/artifacts/docker.yaml

To use the latest available version (not recommended for production tests):

export SCT_SCYLLA_DOCTOR_VERSION=""
hydra run-test artifacts_test --backend docker --config test-cases/artifacts/docker.yaml