Skip to content

README.md

Latest commit

 

History

History
163 lines (126 loc) · 4.95 KB

README.md

File metadata and controls

163 lines (126 loc) · 4.95 KB

scanner-audit

Validates vulnerability scanners against Chainguard's official test cases for Chainguard Images.

See the main repository README for official docs.

Screenshot of HTML report produced by scanner-audit: scanner-audit screenshot

How to Run

Basic usage:

go run . --env <scanner-name>

Example:

go run . --env trivy

Filter by distro or test case:

go run . --env grype -d chainguard
go run . --env grype -t fixed-vulnerabilities
go run . --env grype -d chainguard -t fixed-vulnerabilities  # Can combine filters

The audit generates report.html by default. Use -o to specify a different output path.

Requirements:

  • Docker must be running
  • Scanner-specific environment variables must be set (if required)

HTML Report

The report includes:

  • Summary: Overall pass/fail status with test counts
  • Scanner Info: Environment name, wrapper image, and configuration details
  • Test Results: Expandable sections for each distro showing:
    • Individual test case results
    • False positives/negatives
    • Expected vs actual CVEs
    • Scanner output logs

Click on distro headers or test cases to expand details.

Audit Envs

Each scanner environment requires two files in the env/ directory:

  1. <scanner>-audit-env.yaml: Melange configuration that defines the package
  2. <scanner>-wrapper.sh: Shell script that runs the scanner and outputs CVE IDs

Example structure:

env/
├── grype-audit-env.yaml
├── grype-wrapper.sh
├── trivy-audit-env.yaml
└── trivy-wrapper.sh

The tool uses Melange and Apko to build a Docker image containing your scanner, then runs it against test images.

Adding a New Scanner

  1. Create env/<scanner>-audit-env.yaml with your scanner's package definition
  2. Create env/<scanner>-wrapper.sh that:
    • Accepts an image URL as the first argument
    • Runs the scanner against that image
    • Outputs CVE IDs to stdout (one per line)
    • Sends other output to stderr

Cloud-Based Scanners and Environment Variables

Required environment variables are automatically detected from your audit-env.yaml file. In the environment.environment section of your melange YAML, list any environment variables that need to be set:

environment:
  environment:
    API_TOKEN: placeholder
    API_SECRET: placeholder
    ENDPOINT_URL: placeholder

When running scanner-audit:

  1. These variables are automatically detected from the YAML
  2. The tool validates that each variable is set in your environment
  3. Variables cannot be empty or set to "placeholder"
  4. All required variables are automatically passed to your wrapper script

Example:

# Set required environment variables
export API_TOKEN="your-token"
export API_SECRET="your-secret"

# Run the audit
go run . --env cloud-scanner

Special Environment Variable: REQUIRES_DOCKER

The REQUIRES_DOCKER environment variable is a special case. When this variable is defined in your audit-env.yaml file (with any value), scanner-audit will automatically mount the Docker socket into the container:

environment:
  environment:
    REQUIRES_DOCKER: "true"  # Any value will trigger Docker socket mounting
    # ... other variables ...

This enables scanners that need to interact with Docker (e.g., to pull images or inspect containers) to function properly. The Docker socket will be mounted as /var/run/docker.sock:/var/run/docker.sock.

Note: Unlike other environment variables, REQUIRES_DOCKER:

  • Is not validated for empty or placeholder values
  • Is not passed to the container as an environment variable
  • Only serves to trigger the Docker socket mount

Custom Scanner Installation

You can use Melange pipelines in your audit-env.yaml to install custom tools:

package:
  name: custom-scanner-audit-env
  version: 1.0.0

environment:
  contents:
    packages:
      - busybox
      - ca-certificates-bundle
      - curl

pipeline:
  - uses: fetch
    with:
      uri: https://github.com/your-org/scanner/releases/download/v1.0.0/scanner-linux-amd64.tar.gz
      expected-sha256: abc123...
  - uses: strip-prefix
    with:
      prefix: scanner-v1.0.0/
  - runs: |
      mkdir -p ${{targets.destdir}}/usr/local/bin
      mv scanner ${{targets.destdir}}/usr/local/bin/
      chmod +x ${{targets.destdir}}/usr/local/bin/scanner

CLI Flags

Flag Short Description
--env Scanner environment name (required)
--distro -d Filter to specific distro
--test-case -t Filter to specific test case
--output -o Output path for HTML report (default: report.html)
--force-build Rebuild wrapper image even if it exists
--no-remove Keep container after tests complete
--no-cache Disable XDG cache directory mounting
--answers-chainguard Custom answers file for Chainguard (path or URL)
--answers-wolfi Custom answers file (path or URL)