Pulumi Validator

A Python tool that validates Pulumi project/stack setup against best practices using the Pulumi REST API.

Features

Validates 7 key criteria:

1. API Token - Verifies project owner has configured a valid token (checks for successful deployments)
2. Project Segmentation - Checks for multiple stacks (dev, prod, etc.)
3. Deployability - Validates successful deployments
4. Local Configuration - Ensures stacks have configuration
5. ESC Configuration - Checks for Pulumi ESC environment usage
6. Drift Detection - Validates refresh operations or scheduled drift detection
7. Policy - Checks for Pulumi Policy as Code

Each criterion receives a grade: PASS, WARN (partial credit), or FAIL.

Use Case

This tool is designed for grading panels to validate Pulumi project submissions. The grader uses their own Pulumi access token to validate projects owned by different users, as long as the grader has read access to the organization/project being validated.

Installation

This project uses UV for dependency management.

# Install UV if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install the project
cd pulumi-validator
uv sync

Usage

Set up your Pulumi access token

The grader should use their own token with read access to the projects being validated:

export PULUMI_ACCESS_TOKEN=pul-xxxxxxxxxxxxx

Or pass it via --token flag.

Run validation

# Console output (default)
uv run python -m src.main --org myorg --project myproject

# JSON output
uv run python -m src.main --org myorg --project myproject --format json

# Markdown output to file
uv run python -m src.main --org myorg --project myproject --format markdown --output report.md

# Verbose mode
uv run python -m src.main --org myorg --project myproject --verbose

# Or use the wrapper script
uv run python pulumi_validator.py --org myorg --project myproject

Command-line options

• --org (required) - Pulumi organization name
• --project (required) - Pulumi project name
• --token (optional) - Override PULUMI_ACCESS_TOKEN environment variable
• --format (optional) - Output format: console, json, or markdown (default: console)
• --output (optional) - Output file path (default: stdout)
• --verbose (optional) - Enable verbose logging

Grading Criteria

API Token (PASS/FAIL)

• PASS: Project has successful deployments (proves owner has valid token)
• FAIL: No stacks or no successful deployments found

Project Segmentation (PASS/WARN/FAIL)

• PASS: 2+ stacks exist
• WARN: 1 stack exists
• FAIL: No stacks

Deployability (PASS/WARN/FAIL)

• PASS: All stacks have successful deployments
• WARN: Some stacks have successful deployments
• FAIL: No successful deployments

Local Configuration (PASS/WARN/FAIL)

• PASS: All stacks have configuration
• WARN: Some stacks have configuration
• FAIL: No configuration

ESC Configuration (PASS/WARN/FAIL)

• PASS: All stacks use ESC environments
• WARN: Some stacks use ESC
• FAIL: No ESC usage

Drift Detection (PASS/WARN/FAIL)

• PASS: All stacks have drift detection configured (refresh operations or scheduled drift detection) with no drift
• WARN: Drift detected or partial drift detection coverage
• FAIL: No drift detection configured

Policy (PASS/FAIL)

• PASS: Policy packs have been applied to the project (detected via Policy Results API)
• FAIL: No policy enforcement detected

Development

# Run tests (when implemented)
uv run pytest

# Format code
uv run ruff format

# Lint code
uv run ruff check

API Reference

This tool uses the Pulumi REST API:

• Base URL: https://api.pulumi.com/api
• Authentication: Bearer token
• Key endpoints used:
  • GET /user/stacks - List stacks with filters
  • GET /stacks/{org}/{project}/{stack} - Get stack details
  • GET /stacks/{org}/{project}/{stack}/updates - Get update history
  • GET /stacks/{org}/{project}/{stack}/deployments/settings - Get deployment settings
  • GET /stacks/{org}/{project}/{stack}/deployments/schedules - Get deployment schedules
  • GET /esc/environments/{org} - List ESC environments
  • POST /orgs/{org}/policyresults/issues - Query policy issues for a project

Exit Codes

• 0 - All validations passed or warned
• 1 - One or more validations failed or error occurred

License

MIT