Automated Testing for Agent OS Scripts

[fgrehm]

Problem Statement

Recent bugs (#240, #244) revealed critical issues that could have been caught with automated testing:

1. exclude_inherited_files ignored - Skills created for excluded standards
2. dry-run deleting files - --dry-run flag not respected in cleanup functions

Both bugs involved subtle shell scripting issues (subshell variable scope, missing conditional checks) that manual testing easily misses.

Proposed Solution

Implement automated testing using:

• bats-core - Bash Automated Testing System
• GitHub Actions - Run tests on every PR and push
• Test fixtures - Sample profiles and projects for testing

Initial Set of tests

1. Dry-run mode safety

   • Verify --dry-run never modifies filesystem
   • Test with both project-install.sh and project-update.sh
   • Check that preview output matches what would be created

2. Profile exclusions

   • Test exclude_inherited_files in profile-config.yml
   • Verify excluded standards not copied to agent-os/standards/
   • Verify excluded skills not created in .claude/skills/
   • Test both inherited and current profile exclusions

3. Profile inheritance

   • Test child profile overrides parent files
   • Test multi-level inheritance (grandparent → parent → child)
   • Verify inheritance resolution order

4. Installation modes

   • Traditional mode (standards as file references)
   • Skills mode (standards as Claude Code Skills)
   • Claude Code enabled/disabled
   • Agent OS commands enabled/disabled

5. Script safety

   • No unintended file deletions
   • Proper error handling and exit codes
   • No unbound variables (set -u compliance)

Folder Structure

Click to expand

tests/
├── fixtures/
│   ├── profiles/
│   │   ├── test-parent/
│   │   │   ├── profile-config.yml
│   │   │   └── standards/
│   │   │       └── global/
│   │   │           └── test-standard.md
│   │   └── test-child/
│   │       ├── profile-config.yml (inherits from test-parent)
│   │       └── standards/
│   │           └── global/
│   │               └── child-standard.md
│   └── projects/
│       └── sample-project/
│           └── package.json
├── unit/
│   ├── common-functions.bats      # Test utility functions
│   ├── profile-resolution.bats    # Test inheritance logic
│   └── exclusion-patterns.bats    # Test pattern matching
├── integration/
│   ├── project-install.bats       # Test full installation
│   ├── project-update.bats        # Test updates
│   └── dry-run-safety.bats        # Test dry-run mode
└── helpers/
    └── test-helpers.bash          # Shared test utilities

Example Tests

Dry-run Safety Test

#!/usr/bin/env bats

load '../helpers/test-helpers'

setup() {
  # Create temporary project with existing files
  TEST_PROJECT="$(mktemp -d)"
  mkdir -p "$TEST_PROJECT/agent-os/standards"
  echo "test content" > "$TEST_PROJECT/agent-os/standards/test.md"
}

teardown() {
  rm -rf "$TEST_PROJECT"
}

@test "project-update.sh --dry-run does not delete files" {
  # Record files before dry-run
  local files_before="$(find "$TEST_PROJECT" -type f | sort)"

  # Run with dry-run
  run "$AGENT_OS_DIR/scripts/project-update.sh" \
    --profile default \
    --base-dir "$TEST_PROJECT" \
    --dry-run

  # Verify no files were deleted
  local files_after="$(find "$TEST_PROJECT" -type f | sort)"
  [ "$files_before" = "$files_after" ]
}

@test "project-update.sh --dry-run shows preview output" {
  run "$AGENT_OS_DIR/scripts/project-update.sh" \
    --profile default \
    --base-dir "$TEST_PROJECT" \
    --dry-run

  # Verify dry-run messaging
  [[ "$output" =~ "DRY RUN" ]]
  [[ "$output" =~ "would be" ]]
}

Exclusion Pattern Test

#!/usr/bin/env bats

load '../helpers/test-helpers'

setup() {
  # Create test profile with exclusions
  TEST_PROFILE_DIR="$(mktemp -d)"

  cat > "$TEST_PROFILE_DIR/profile-config.yml" <<EOF
inherits_from: default
exclude_inherited_files:
  - standards/backend/*
  - standards/frontend/*
EOF
}

teardown() {
  rm -rf "$TEST_PROFILE_DIR"
}

@test "exclude_inherited_files excludes standards from installation" {
  TEST_PROJECT="$(mktemp -d)"

  run "$AGENT_OS_DIR/scripts/project-install.sh" \
    --profile "$TEST_PROFILE_DIR" \
    --base-dir "$TEST_PROJECT"

  # Verify backend/frontend standards not installed
  [ ! -d "$TEST_PROJECT/agent-os/standards/backend" ]
  [ ! -d "$TEST_PROJECT/agent-os/standards/frontend" ]

  rm -rf "$TEST_PROJECT"
}

@test "exclude_inherited_files excludes skills generation" {
  TEST_PROJECT="$(mktemp -d)"

  run "$AGENT_OS_DIR/scripts/project-install.sh" \
    --profile "$TEST_PROFILE_DIR" \
    --base-dir "$TEST_PROJECT" \
    --standards-as-claude-code-skills true

  # Verify no backend/frontend skills created
  local skills="$(find "$TEST_PROJECT/.claude/skills" -name "SKILL.md" 2>/dev/null || true)"

  [[ ! "$skills" =~ "backend" ]]
  [[ ! "$skills" =~ "frontend" ]]

  rm -rf "$TEST_PROJECT"
}

Benefits

1. Catch regressions early - Tests run automatically on every change
2. Confidence in refactoring - Safe to improve code knowing tests will catch breaks
3. Documentation - Tests serve as executable examples
4. Cross-platform validation - Test on both Linux and macOS
5. Faster review - Reviewers see test results immediately

Resources

• bats-core: https://github.com/bats-core/bats-core
• bats-assert: https://github.com/bats-core/bats-assert (assertions library)
• bats-support: https://github.com/bats-core/bats-support (helper functions)
• Testing Bash with Bats: https://opensource.com/article/19/2/testing-bash-bats

[smaugho]

This is definitely a must!... TBH, I'm not a fan of the current bash scripts (they're making running the project in Windows quite difficult)... if I could select a tech, I would build all the scripts in Python or JS, and using i.e. Node to run them which would make them quite OS-independent... but also, the tools for testing the code would be much more powerful (and known to most devs).

BTW, I would go with JS, as I'm quite bad in Python, but this said, I know that Python is (probably) a better option.

Now, this doesn't apply only to the scripts, for instance the Prompts themselves should be tested somehow, i.e. some changes just let prompts mentioning non existing agents (see #252), which kind of broke the specs creation workflow (it works, but it is and feel buggy). To tests these, some sort of evaluation tool could be used (i.e. promptfoo), or create a template verification tool: integrity of prompts could then be verified by some scripts (i.e. verify all the agents exists - assuming we use a single notation for them, or verify all the references / paths exist).

[Y0lan]

Stop using windows to code