Cassandra Benchmark Wrapper

Description

This wrapper facilitates the automated execution of the Apache Cassandra stress benchmark. Apache Cassandra is an open-source, distributed NoSQL database designed for handling large volumes of data across many commodity servers. The cassandra-stress tool is Cassandra's built-in benchmarking utility for testing write and read throughput under configurable workloads.

The wrapper provides:

• Automated Cassandra download, installation, and execution.
• Configurable mixed read/write workload ratios.
• Customizable thread counts with automatic CPU-aware defaults.
• Multi-iteration testing with averaged results.
• Result collection, processing, and verification.
• CSV and JSON output formats.
• System configuration metadata capture.
• Integration with test_tools framework.
• Optional Performance Co-Pilot (PCP) integration.

Command-Line Options

Cassandra Options:
  --run_time <minutes>: How long to run each test for. Default is 4 minutes.
  --threads <t1,t2...>: Comma separated list of the number of threads to run.
      Default is 1 to ncpus, intervals of 8, unless less than 8 cpus, then each cpu.
  --tests <write=a,read=b:write=c,read=d>...: Test definitions to run.
      Default is: write=1,read=0:write=1,read=1:write=1,read=3

General test_tools options:
  --home_parent <value>: Parent home directory. If not set, defaults to current working directory.
  --host_config <value>: Host configuration name, defaults to current hostname.
  --iterations <value>: Number of times to run the test, defaults to 1.
  --run_user: User that is actually running the test on the test system. Defaults to current user.
  --sys_type: Type of system working with (aws, azure, hostname). Defaults to hostname.
  --sysname: Name of the system running, used in determining config files. Defaults to hostname.
  --tuned_setting: Used in naming the results directory. For RHEL, defaults to current active tuned profile.
      For non-RHEL systems, defaults to 'none'.
  --use_pcp: Enable Performance Co-Pilot monitoring during test execution.
  --tools_git <value>: Git repo to retrieve the required tools from.
      Default: https://github.com/redhat-performance/test_tools-wrappers
  --usage: Display this usage message.

What the Script Does

The cassandra_run script performs the following workflow:

1. Environment Setup:

   • Clones the test_tools-wrappers repository if not present (default: ~/test_tools).
   • Sources error codes and general setup utilities.
   • Gathers system hardware information.

2. Package Installation:

   • Installs required dependencies via package_tool.
   • Dependencies are defined in cassandra.json for different OS variants (RHEL).

3. Cassandra Download & Installation:

   • Downloads Apache Cassandra 5.0.8 binary tarball from https://downloads.apache.org/cassandra/.
   • Extracts the archive and sets up executables (cassandra, cassandra-stress).
   • Creates the required logs directory.
   • Skips download if the Cassandra directory already exists.

4. Service Startup:

   • Starts the Cassandra service via systemctl.
   • Waits until Cassandra is fully active before proceeding.

5. Thread Configuration:

   • If no thread list is specified, automatically generates thread counts.
   • For systems with fewer than 8 CPUs, tests each CPU count (1 to ncpus).
   • For systems with 8+ CPUs, generates 8 evenly spaced intervals up to ncpus.

6. Test Execution:

   • Runs cassandra-stress mixed for each combination of test definition, thread count, and iteration.
   • Each test runs for the configured duration (default: 4 minutes).
   • Test definitions control the read/write ratio (e.g., write=1,read=3 means 75% reads, 25% writes).
   • Records timestamps for each test run.
   • Optionally records PCP performance data during execution.

7. Data Collection:

   • Captures raw output from each run in cassandra_iter_<iter>_threads_<threads>_<test>.out files.
   • Extracts the "Op rate" (operations per second) metric from each run.

8. Result Processing:

   • Averages Op rate across all iterations for each test/thread combination.
   • Generates CSV files with configuration metadata and performance data.
   • Creates JSON output for verification.
   • Validates results against Pydantic schema (result_schema.py).

9. Verification:

   • Validates results against Pydantic schema ensuring:
     • Thread count is a positive integer.
     • Operations per second is a positive integer.
     • Timestamps are valid datetime objects.
   • Uses csv_to_json and verify_results from test_tools.

10. Output:

    • Saves all raw output files, processed CSV/JSON, and system metadata.
    • Optionally saves PCP performance data.
    • Archives results to configured storage location.

Dependencies

Location of underlying workload: https://downloads.apache.org/cassandra/

General packages required: chkconfig (RHEL)

Runtime requirements: Apache Cassandra requires Java (JDK 11 or 17) to be pre-installed.

To run:

git clone https://github.com/redhat-performance/cassandra-wrapper
cd cassandra-wrapper/cassandra
./cassandra_run

The script will automatically detect your CPU configuration, download Cassandra, and run all default workloads.

The Cassandra Stress Benchmark

The cassandra-stress tool measures Cassandra's throughput under configurable mixed read/write workloads. It connects to a local Cassandra instance and drives operations at the specified thread concurrency.

Default Test Definitions

The wrapper runs three workload profiles by default:

1. write=1,read=0: Pure write workload (100% writes). Measures maximum write throughput.

2. write=1,read=1: Balanced read/write workload (50% writes, 50% reads). Measures mixed operation throughput.

3. write=1,read=3: Read-heavy workload (25% writes, 75% reads). Measures read-dominated throughput.

Performance Metrics

Each test run reports:

1. Op rate: Operations per second — the primary throughput metric. Higher values indicate better performance.

2. Latency: Response time statistics (mean, median, 95th/99th/99.9th percentile, max) for each operation type.

3. Total partitions: Number of partitions written/read during the test.

4. Total errors: Number of failed operations (should be 0 for a healthy system).

The wrapper extracts and averages the Op rate across iterations for the final results.

Output Files

The results directory contains:

• results_cassandra.csv: CSV file with all test performance metrics (test, threads, ops_sec, start/end timestamps)
• results_cassandra.json: JSON file with validated results data
• cassandra_iter_*threads*_*.out: Raw cassandra-stress output files from each test run
• meta_data*.yml: System metadata (CPU info, memory, kernel version)
• PCP data (if --use_pcp option used): Performance Co-Pilot monitoring data

Examples

Basic run with defaults

./cassandra_run

This runs with:

• Apache Cassandra 5.0.8
• 3 workload profiles (write-only, balanced, read-heavy)
• 4-minute duration per test
• Auto-detected thread counts based on CPU count
• 1 iteration

Run with custom thread counts

./cassandra_run --threads 1,4,8,16

Runs all workloads with 1, 4, 8, and 16 threads specifically.

Run for longer duration

./cassandra_run --run_time 10

Runs each test for 10 minutes instead of the default 4 minutes, producing more stable results.

Run multiple iterations

./cassandra_run --iterations 3

Runs the complete test suite 3 times and averages the results for consistency validation.

Run custom workload ratios

./cassandra_run --tests "write=1,read=0:write=1,read=9"

Runs a pure write test and a 90% read / 10% write test.

Run with PCP monitoring

./cassandra_run --use_pcp

Collects Performance Co-Pilot data during the run for detailed performance analysis.

Combination example

./cassandra_run --run_time 10 --threads 1,2,4,8 --iterations 3 --use_pcp

Runs 3 iterations of all workloads with specified thread counts, 10-minute duration, and PCP monitoring.

How Thread Auto-Detection Works

When no --threads option is provided:

• Systems with < 8 CPUs: Tests every thread count from 1 to ncpus (e.g., on a 4-core system: 1, 2, 3, 4).
• Systems with 8+ CPUs: Generates 8 evenly spaced intervals from 1 to ncpus (e.g., on a 64-core system: 1, 8, 16, 24, 32, 40, 48, 56, 64).

This approach balances test coverage with reasonable execution time.

How Result Averaging Works

When running multiple iterations (--iterations > 1):

1. Each iteration produces a complete set of results for all test/thread combinations.
2. Raw results are saved in separate cassandra_iter_* files.
3. The wrapper extracts the Op rate from each iteration.
4. Final results are the arithmetic mean across all iterations.

This approach:

• Reduces impact of transient system effects
• Provides more reliable performance measurements
• Helps identify result variance across runs

Return Codes

The script uses standardized error codes from test_tools error_codes:

• 0: Success
• 101: Git clone failure
• E_GENERAL: General execution errors (Cassandra start failure, CSV conversion failure)
• E_VALIDATION_FAIL: Results schema validation failure
• E_USAGE: Invalid usage/arguments

Exit codes indicate specific failure points for automated testing workflows.

Notes

Supported Platforms

• Linux: x86_64 and aarch64 architectures
• OS Support: RHEL
• Cassandra Version: 5.0.8 (hardcoded in wrapper)

Performance Considerations

• Cassandra stress tests are I/O and CPU intensive
• Write-only workloads (write=1,read=0) typically show the highest Op rates
• Read-heavy workloads (write=1,read=3) stress the storage subsystem more
• Thread scaling shows how well Cassandra utilizes available CPU cores
• Run multiple iterations (--iterations 3 or higher) for reliable results
• Ensure system is idle during testing for best consistency

Read/Write Ratio Interpretation

The test definition write=W,read=R controls the operation mix:

• write=1,read=0: 100% writes
• write=1,read=1: 50% writes, 50% reads
• write=1,read=3: 25% writes, 75% reads
• The ratio is W:(W+R) writes and R:(W+R) reads

Performance Tips

• Run multiple test iterations (--iterations 3+) to verify consistency
• Ensure system is idle (no other workloads) for best results
• Consider the active tuned profile on RHEL systems
• Use longer run times (--run_time 10+) for more stable throughput numbers
• PCP monitoring (--use_pcp) adds minimal overhead but provides detailed metrics
• Allow Cassandra to warm up — the first iteration may show lower throughput

Troubleshooting

• If Cassandra fails to start, verify Java (JDK 11 or 17) is installed
• If cassandra-stress fails, check Cassandra logs in apache-cassandra-<version>/logs/
• If results seem inconsistent, run more iterations and check system load during testing
• Use --use_pcp to collect detailed performance counters for analysis
• Check cassandra_iter_* files for detailed error messages and latency breakdowns
• For upstream Cassandra issues, see: https://cassandra.apache.org/

References

• Apache Cassandra: https://cassandra.apache.org/
• Cassandra Downloads: https://downloads.apache.org/cassandra/
• Cassandra Stress Documentation: https://cassandra.apache.org/doc/latest/cassandra/tools/cassandra_stress.html
• test_tools Framework: https://github.com/redhat-performance/test_tools-wrappers