Stateless File Sync Orchestrator for Repository Management
CI / CD Β Β
|
|
Β Β Β Β Quality Β Β
|
|
Security Β Β
|
|
Β Β Β Β Community Β Β
|
|
πΒ Quick Start
|
πΒ Documentation
|
β‘Β Performance
|
β¨Β Key Features
|
π§ͺΒ ExamplesΒ &Β Tests
|
π₯Β Maintainers
|
πΒ HowΒ ItΒ Works
|
π οΈΒ Code Standards
|
π€Β Contributing
|
π‘Β UsageΒ Examples
|
π€Β AIΒ Usage
|
βοΈΒ License
|
Get up and running with go-broadcast in under 5 minutes!
- Go 1.24+ (supported release) and GitHub CLI installed
- GitHub authentication:
gh auth login - MAGE-X (optional, for building from source)
Option 1: Pre-built Binary (Recommended)
Download the latest release for your platform from GitHub Releases, or use the GitHub CLI:
# macOS (Apple Silicon)
gh release download --repo mrz1836/go-broadcast --pattern '*darwin*arm64*' -D /tmp && \
tar -xzf /tmp/go-broadcast_*_darwin_arm64.tar.gz -C /tmp && \
sudo mv /tmp/go-broadcast /usr/local/bin/
# macOS (Intel)
gh release download --repo mrz1836/go-broadcast --pattern '*darwin*amd64*' -D /tmp && \
tar -xzf /tmp/go-broadcast_*_darwin_amd64.tar.gz -C /tmp && \
sudo mv /tmp/go-broadcast /usr/local/bin/
# Linux (x86_64)
gh release download --repo mrz1836/go-broadcast --pattern '*linux*amd64*' -D /tmp && \
tar -xzf /tmp/go-broadcast_*_linux_amd64.tar.gz -C /tmp && \
sudo mv /tmp/go-broadcast /usr/local/bin/Option 2: Build from Source
git clone https://github.com/mrz1836/go-broadcast.git
cd go-broadcast && go install ./cmd/go-broadcastOption 3: Upgrade Existing Installation
go-broadcast upgrade --forceCreate a sync.yaml file:
version: 1
groups:
- name: "Default Sync"
id: "default"
description: "Basic file and directory synchronization"
priority: 1
enabled: true
source:
repo: "mrz1836/template-repo"
branch: "master"
targets:
- repo: "mrz1836/target-repo"
files:
- src: ".github/workflows/ci.yml"
dest: ".github/workflows/ci.yml"
directories:
- src: ".github/actions"
dest: ".github/actions"
exclude: ["*.out", "*.test"]
transform:
repo_name: true# Validate configuration
go-broadcast validate --config sync.yaml
# Preview changes (dry run)
go-broadcast sync --dry-run --config sync.yaml
# Execute sync
go-broadcast sync --config sync.yamlWhen using --dry-run, go-broadcast provides clean, readable output showing exactly what would happen:
π π COMMIT PREVIEW
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Message: sync: update 2 files from source repository
β Files: 2 changed
β README.md, .mage.yaml
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π FILE CHANGES:
β¨ README.md (added) (+1249 bytes)
π .mage.yaml (modified) (+45 bytes)
π DRY-RUN: Pull Request Preview
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Repository: company/service-name
β Branch: chore/sync-files-20250130-143052-abc123f
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Title: [Sync] Update project files from source repository (abc123f)
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ## What Changed
β * Updated project files to synchronize with the latest ch...
β
β ## Changed Files
β
β - `README.md` (added)
β - `.mage.yaml` (modified)
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
DRY-RUN SUMMARY: Repository sync preview completed successfully
π Repository: company/service-name
πΏ Branch: chore/sync-files-20250130-143052-abc123f
π Files: 2 would be changed
π Commit: dry-run-commit-sha
π‘ Run without --dry-run to execute these changes
That's it! π go-broadcast automatically:
- Executes each group in priority order
- Clones your template repository
- Applies configured transformations
- Creates a branch in each target repository
- Commits synchronized files
- Opens a pull request for review
π‘ Pro tip: go-broadcast includes a built-in coverage system, enterprise performance, and comprehensive logging & debugging - explore the features below!
Install MAGE-X build tool
Want to contribute to go-broadcast? Use MAGE-X for building, testing, linting, and more.
# Install MAGE-X for development and building
go install github.com/mrz1836/mage-x/cmd/magex@latest
magex update:installgo-broadcast is a production-grade repository synchronization platform with enterprise performance:
- Stateless architecture - All state derived from GitHub (branches, PRs, commits)
- File & directory sync - Individual files or entire directories with intelligent filtering
- Mixed sync operations - Combine files and directories in single configurations
- Smart diff detection - Only syncs files that actually changed (content-based)
- Zero-downtime operations - Works at any scale without state corruption
- Full audit trail - Every operation tracked in Git history with metadata
- 587M+ ops/sec - Binary detection with zero memory allocations
- 239M+ ops/sec - Content comparison for identical files
- 32ms/1000 files - Directory processing with concurrent workers
- 90%+ API reduction - GitHub Tree API optimization
- Worker pools - Concurrent task execution with panic recovery
- TTL caching - High-performance cache with 13.5M+ ops/sec
- Group-based organization - Logical grouping with names, IDs, and descriptions
- Priority execution - Groups execute in order (lower number = higher priority)
- Dependency management - Groups can depend on successful completion of others
- Enable/disable control - Toggle groups without removing configuration
- Reusable lists - Define file/directory lists once, use everywhere
- Module-aware sync - Version management for Go modules with semantic versioning
- SQLite storage - Structured configuration in queryable database (pure Go, no CGO)
- YAML import/export - Seamless migration and backwards compatibility
- Universal --from-db flag - Works with all commands (sync, status, cancel, validate, modules)
- Write-time validation - Catch errors before they affect repositories
- Queryable config - "Which repos sync this file?" answered instantly
- CLI management - Full CRUD operations without editing YAML
- Transaction safety - All-or-nothing imports with automatic rollback
- Repository analytics - Track stars, forks, issues, PRs, and security alerts across orgs
- Batched GraphQL collection - Collect metadata for 25 repos per API call (96% reduction)
- Security alert tracking - Dependabot, code scanning, and secret scanning alerts with severity
- Point-in-time snapshots - Historical trend data for repository metrics
- Full documentation β
- Variable substitution - Template variables ({{VAR}} and ${VAR} syntax)
- Go module updates - Automatic repository name transformation
- Pattern-based transforms - Apply to all files in directories
- Context-aware - Different transforms per target repository
- Automatic PR creation - Creates pull requests with rich metadata & AI-generated descriptions
- PR management - Auto-assign reviewers, assignees, and labels
- Automerge labels - Add configurable automerge labels to PRs with
--automergeflag - Global settings - Organization-wide PR assignments
- Branch naming - Encoded metadata for state tracking
- Cancel operations - Abort active syncs with cleanup
- Self-updating - Built-in upgrade command with version management
- Rich dry-run mode - Beautiful previews with exact change details
- Component debugging - Targeted debugging (--debug-git, --debug-api)
- Verbose logging - Multi-level verbosity (-v, -vv, -vvv)
- Progress tracking - Real-time sync progress with statistics
- Diagnostic tool - Comprehensive system information collection
- Status command - Repository synchronization status overview
- 85%+ test coverage - Comprehensive unit and integration tests
- Fuzz testing - Security-critical components fuzz tested
- 60+ linters - Zero tolerance for code issues via golangci-lint
- Vulnerability scanning - govulncheck, nancy, CodeQL, OSSAR
- OpenSSF Scorecard - Supply chain security assessment
- Secret detection - gitleaks integration prevents leaks
- Automatic exclusions - Filters out *.out, *.test, *.exe, .DS_Store, tmp/
- Custom patterns - Add your own exclusion patterns per directory
- Hidden file control - Include or exclude dotfiles as needed
- Binary detection - Intelligent binary file detection prevents corruption
- Preserve structure - Maintains nested directory hierarchies
go-broadcast uses a stateless architecture that tracks synchronization state through GitHub itself - no databases or state files needed!
Every sync operation creates a branch with encoded metadata:
[chore/sync-files]-[group1]-[20250123-143052]-[abc123f]
β β β β
β β β ββββ Source branch commit SHA (7 chars)
β β βββββββββββββββββ Timestamp (YYYYMMDD-HHMMSS)
β ββββββββββββββββββββββββββββββββ Group ID (from config)
ββββββββββββββββββββββββββββββββββββββββββββββ Configurable prefix
-
State Discovery - Queries GitHub to find:
- Latest commit in source repository
- All sync branches in target repositories
- Open sync pull requests
-
Smart Comparison - For each target:
Source commit: abc123f (latest) Target's last sync: def456g (from branch name) Status: Behind β Needs sync β -
Content-Based Sync - Only syncs files that actually changed:
- Fetches current files from target (individual files or directory contents)
- Applies transformations to source files
- Compares content byte-by-byte with smart exclusion filtering
- Skips unchanged files and processes directories concurrently
Each PR includes structured metadata for complete traceability:
<!-- go-broadcast metadata
group:
id: default
name: Default Sync
source:
repo: company/template-repo
branch: master
commit: abc123f7890
files:
- src: .github/workflows/ci.yml
dest: .github/workflows/ci.yml
directories:
- src: .github/actions
dest: .github/actions
excluded: ["*.out", "*.test"]
files_synced: 87
processing_time_ms: 4
performance:
total_files: 88
api_calls_saved: 79
timestamp: 2025-01-23T14:30:52Z
-->
- No State Files - Everything lives in GitHub
- Atomic Operations - Each sync is self-contained
- Full Audit Trail - Branch and PR history shows all syncs
- Disaster Recovery - State can be reconstructed from GitHub
- Works at Scale - No state corruption with concurrent syncs
Sync CI/CD workflows across microservices
version: 1
groups:
- name: "CI/CD Templates"
id: "ci-cd"
priority: 1
enabled: true
source:
repo: "company/ci-templates"
branch: "main"
targets:
- repo: "company/user-service"
blob_size_limit: "0" # Disable filter for this target
files:
- src: "workflows/ci.yml"
dest: ".github/workflows/ci.yml"
transform:
variables:
SERVICE_NAME: "user-service"Sync entire directories with smart exclusions
version: 1
groups:
- name: "GitHub Configuration"
id: "github-config"
priority: 1
enabled: true
source:
repo: "company/ci-templates"
branch: "main"
targets:
- repo: "company/microservice-a"
directories:
- src: ".github/workflows"
dest: ".github/workflows"
exclude: ["*-local.yml", "*.disabled"]
- src: ".github/actions"
dest: ".github/actions"
# Smart defaults automatically exclude: *.out, *.test, *.exe, **/.DS_Store
transform:
repo_name: trueMixed file and directory synchronization
version: 1
groups:
- name: "Mixed Content Sync"
id: "mixed-sync"
priority: 1
enabled: true
source:
repo: "company/template-repo"
branch: "main"
targets:
- repo: "company/service"
files:
- src: "Makefile"
dest: "Makefile"
directories:
- src: "configs"
dest: "configs"
exclude: ["*.local", "*.secret"]
transform:
variables:
SERVICE_NAME: "user-service"Automated PR management with assignees, reviewers, and labels
version: 1
groups:
- name: "Security Policies"
id: "security-sync"
priority: 1
enabled: true
source:
repo: "company/security-templates"
branch: "main"
defaults:
pr_labels: ["automated-sync", "chore"]
pr_assignees: ["tech-lead", "platform-team"]
pr_reviewers: ["senior-dev1", "senior-dev2"]
pr_team_reviewers: ["architecture-team"]
targets:
- repo: "company/critical-service"
files:
- src: "security/policies.yml"
dest: "security/policies.yml"
# Critical service needs security team review
pr_labels: ["security-update", "high-priority"]
pr_assignees: ["security-lead"]
pr_reviewers: ["security-engineer"]
pr_team_reviewers: ["security-team"]File and directory cleanup with deletions
version: 1
groups:
- name: "Repository Cleanup"
id: "cleanup-sync"
description: "Clean up obsolete files while syncing new ones"
priority: 1
enabled: true
source:
repo: "company/template-repo"
branch: "main"
targets:
- repo: "company/service-a"
# Regular file syncing
files:
- src: ".github/workflows/new-ci.yml"
dest: ".github/workflows/ci.yml"
# File deletions - remove obsolete files
# Note: src can be empty or omitted when delete: true
files:
- dest: ".github/workflows/old-ci.yml"
delete: true
- dest: "deprecated-config.json"
delete: true
# Directory operations
directories:
- src: ".github/actions"
dest: ".github/actions"
# Directory deletions - remove entire directories
directories:
- dest: "old-docs"
delete: true
- dest: ".github/legacy-workflows"
delete: true
transform:
repo_name: trueTarget different branches for development workflows
version: 1
groups:
- name: "Development Sync"
id: "dev-sync"
priority: 1
enabled: true
source:
repo: "company/template-repo"
branch: "main"
targets:
# Sync to main branch (production)
- repo: "company/service-a"
branch: "main"
files:
- src: ".github/workflows/production.yml"
dest: ".github/workflows/ci.yml"
# Sync to develop branch (staging)
- repo: "company/service-a"
branch: "develop"
files:
- src: ".github/workflows/staging.yml"
dest: ".github/workflows/ci.yml"
transform:
variables:
ENVIRONMENT: "staging"
# Sync to feature branch
- repo: "company/service-b"
branch: "feature/new-deployment"
files:
- src: ".github/workflows/feature.yml"
dest: ".github/workflows/ci.yml"
transform:
variables:
ENVIRONMENT: "development"Review and merge pull requests
# Review and merge a single PR
go-broadcast review-pr https://github.com/owner/repo/pull/123
# Review and merge multiple PRs in batch
go-broadcast review-pr \
https://github.com/owner/repo/pull/123 \
https://github.com/owner/repo/pull/124 \
https://github.com/owner/repo/pull/125
# Use short format
go-broadcast review-pr owner/repo#123
# Review all PRs assigned to you (excludes drafts)
go-broadcast review-pr --all-assigned-prs
# Customize the review approval message
go-broadcast review-pr --message "Approved after testing" \
https://github.com/owner/repo/pull/123
# Preview all assigned PRs without executing (dry run)
go-broadcast review-pr --all-assigned-prs --dry-run
# Review all assigned PRs with custom message
go-broadcast review-pr --all-assigned-prs --message "LGTM"
# Bypass branch protection with admin privileges (requires admin access)
go-broadcast review-pr --bypass https://github.com/owner/repo/pull/123
# Bypass and ignore status checks (dangerous - use with caution)
go-broadcast review-pr --bypass --ignore-checks https://github.com/owner/repo/pull/123The review-pr command will:
- Parse the PR URL(s) to extract owner, repo, and PR number (or fetch all assigned PRs with
--all-assigned-prs) - Check if the PR is already merged or closed
- Submit an approving review with your message (default: "LGTM")
- Detect the repository's preferred merge method (squash, merge, or rebase)
- Intelligently merge the PR using a try-and-fallback approach:
- Tries to merge immediately first (optimistic)
- If branch protection blocks merge, automatically enables auto-merge
- Handles merge conflicts, pending checks, and required reviews gracefully
All Assigned PRs Mode (--all-assigned-prs):
- Automatically fetches all open pull requests assigned to you
- Excludes draft PRs from processing (only ready-for-review PRs are included)
- Cannot be used together with explicit PR URLs (mutually exclusive)
- Perfect for bulk reviewing and merging your assigned PRs
- Works with
--dry-runto preview what would be processed - Processes each PR sequentially with the same smart merge behavior
Automerge Label Gating:
- When
GO_BROADCAST_AUTOMERGE_LABELSis configured, merge operations require the PR to have one of the specified labels - Label present: Review + merge operations proceed as normal
- Label missing: Review/comment only, NO merge attempt (safe mode)
- No labels configured: Backwards compatible - merge proceeds without label check
- This prevents accidental merges and ensures only explicitly approved PRs are merged
- Configure via environment variable:
GO_BROADCAST_AUTOMERGE_LABELS=automerge,ready-to-merge
Smart Merge Behavior (Try-and-Fallback):
- The command uses an intelligent try-first approach for maximum efficiency
- Automerge label check: If labels are configured, PRs without the label are review-only
- Merge conflicts detected: If the PR has merge conflicts, it enables auto-merge immediately
- Warning: "
β οΈ PR has merge conflicts - enabling auto-merge for when conflicts are resolved"
- Warning: "
- Optimistic merge attempt: For PRs with the automerge label, it tries to merge immediately first
- Success: If merge succeeds, the PR is merged right away β
- Branch protection detected: If merge fails due to branch protection policies:
- Automatically falls back to enabling auto-merge
- Warning: "
β οΈ Branch protection blocking merge - enabling auto-merge" - Success: "β Auto-merge enabled - will merge when requirements are met"
- Handles: pending status checks, required reviews, or other protection rules
- Real errors: Other errors (permissions, PR not found, etc.) fail as expected
- This ensures PRs get merged without manual intervention once all requirements are satisfied
Admin Bypass Mode (--bypass):
- Use
--bypassto merge using admin privileges, bypassing branch protection rules - Requires admin access to the repository
- By default, still waits for status checks to pass before merging
- Use
--ignore-checkswith--bypassto skip waiting for status checks (dangerous) - Perfect for repos with "automerge" labels or other exceptions to review requirements
- Example:
go-broadcast review-pr --bypass https://github.com/owner/repo/pull/123
Merge Method Detection:
- The command automatically queries the repository settings via GitHub API
- Uses the first allowed method in this priority: squash β merge β rebase
- Handles all three merge strategies supported by GitHub
Batch Operations:
- Process multiple PRs sequentially
- Continue on error (doesn't fail fast)
- Display summary showing:
- How many PRs were merged immediately
- How many have auto-merge enabled
- How many were review-only (no automerge label)
- How many failed
- All PRs are processed even if some fail
- Non-zero exit code if any PR fails
Error Handling:
- Skips PRs that are already merged (with warning)
- Reports PRs that are closed but not merged
- Provides clear error messages for failures
- Dry-run mode shows what would happen without making changes
# Validate and preview
go-broadcast validate --config sync.yaml
go-broadcast validate --skip-remote-checks # Offline validation (no network checks)
go-broadcast validate --source-only # Only validate source repo access
go-broadcast sync --dry-run --config sync.yaml
# Execute sync
go-broadcast sync --config sync.yaml
go-broadcast sync org/specific-repo --config sync.yaml
go-broadcast sync --clear-cache --config sync.yaml # Clear module version cache before sync
# Database-backed configuration (alternative to YAML)
go-broadcast db init # Initialize database
go-broadcast db init --force # Force recreation (destroys existing data)
go-broadcast db import sync.yaml # Import existing YAML config
go-broadcast db import --force # Force import (replace existing config)
go-broadcast sync --from-db # Sync using database config
go-broadcast status --from-db # Check status from database
go-broadcast validate --from-db # Validate database config
go-broadcast cancel --from-db --groups "core" # Cancel using database config
# Database status and inspection
go-broadcast db status # Show database status (version, table counts)
go-broadcast db status --json # Status in JSON format
go-broadcast db diff # Compare DB config vs YAML file
go-broadcast db diff --yaml my-config.yaml # Compare against specific YAML file
go-broadcast db diff --detail # Show detailed field-level differences
go-broadcast db validate # Validate database consistency
go-broadcast db validate --json # Validation results in JSON
# Database export
go-broadcast db export --output backup.yaml # Export entire config to YAML
go-broadcast db export --group core --output core-group.yaml # Export specific group
go-broadcast db export --stdout # Export to stdout
# Granular CLI management (full reference: docs/database.md)
go-broadcast db group list # List all groups
go-broadcast db group enable my-group # Enable a group
go-broadcast db group disable my-group # Disable a group
go-broadcast db group create # Create a new group
go-broadcast db group update my-group # Update a group
go-broadcast db group delete my-group # Delete a group
go-broadcast db target list # List all target repositories
go-broadcast db target add owner/repo # Add a target repository
go-broadcast db target remove target-id # Remove a target repository
go-broadcast db file-list list # List all file lists
go-broadcast db file-list create # Create a new file list
go-broadcast db file-list delete list-id # Delete a file list
go-broadcast db dir-list list # List all directory lists
go-broadcast db dir-list create # Create a new directory list
go-broadcast db ref add-file-list target-id list-id # Add file list ref to target
go-broadcast db ref add-dir-list target-id list-id # Add dir list ref to target
go-broadcast db bulk add-file-list list-id t1 t2 # Add file list to multiple targets
go-broadcast db bulk add-dir-list list-id t1 t2 # Add dir list to multiple targets
# See docs/database.md for complete database documentation
# Repository analytics
go-broadcast analytics sync # Collect analytics for all orgs
go-broadcast analytics sync --org mrz1836 # Sync specific organization
go-broadcast analytics sync --security-only # Security alerts only
go-broadcast analytics status # Show analytics status
go-broadcast analytics status mrz1836/go-broadcast # Status for specific repo
# Group-based sync operations
go-broadcast sync --groups "Default Sync" --config sync.yaml # Sync only one group by name
go-broadcast sync --groups "default" --config sync.yaml # Sync by group ID
go-broadcast sync --groups "core,security" --config sync.yaml # Sync multiple groups
go-broadcast sync --skip-groups "experimental" --config sync.yaml # Skip specific groups
go-broadcast sync --groups "core" org/repo1 --config sync.yaml # Combine with target filtering
# Automerge configuration
go-broadcast sync --automerge --config sync.yaml # Add automerge labels to created PRs
go-broadcast sync --automerge --groups "core" --config sync.yaml # Automerge with group filtering (adds labels)
# Monitor status of repositories
go-broadcast status --groups "core"
go-broadcast status --groups "core,security"
# Show all groups except specific ones
go-broadcast status --skip-groups "experimental"
# Troubleshooting and diagnostics
go-broadcast diagnose # Collect system diagnostic information
go-broadcast diagnose > diagnostics.json # Save diagnostics to file
# Cancel active syncs
go-broadcast cancel # Cancel all active sync operations
go-broadcast cancel org/repo1 # Cancel syncs for specific repository
go-broadcast cancel --groups "core" # Cancel syncs for specific group (much faster!)
go-broadcast cancel --groups "core,security" # Cancel syncs for multiple groups
go-broadcast cancel --groups "core" org/repo1 # Cancel specific repo in a group
go-broadcast cancel --skip-groups "experimental" # Cancel all except experimental group
go-broadcast cancel --dry-run # Preview what would be cancelled
# Query sync metrics and history
go-broadcast metrics # Summary statistics across all sync runs
go-broadcast metrics --last 7d # Runs from last 7 days
go-broadcast metrics --last 24h # Runs from last 24 hours
go-broadcast metrics --repo owner/repo-name # Sync history for specific repository
go-broadcast metrics --run SR-20260215-abc123 # Details for a specific run ID
go-broadcast metrics --json # Output as JSON
go-broadcast metrics --last 24h --json # Recent runs in JSON format
# Review and merge pull requests
go-broadcast review-pr <pr-url> # Review and merge single PR
go-broadcast review-pr <url1> <url2> <url3> # Batch review and merge multiple PRs
go-broadcast review-pr --message "Looks good!" <pr-url> # Custom review message
go-broadcast review-pr --dry-run <pr-url> # Preview without executing
go-broadcast review-pr owner/repo#123 # Use short format
go-broadcast review-pr https://github.com/owner/repo/pull/123 # Use full URL
go-broadcast review-pr --all-assigned-prs # Review all PRs assigned to you
go-broadcast review-pr --all-assigned-prs --dry-run # Preview all assigned PRs
go-broadcast review-pr --all-assigned-prs --message "LGTM" # Custom message for all assigned PRs
go-broadcast review-pr --bypass <pr-url> # Bypass branch protection (admin)
go-broadcast review-pr --bypass --ignore-checks <pr-url> # Bypass and skip status checks
# Module management
go-broadcast modules list # List all modules in config
go-broadcast modules show github.com/example/module # Show module details
go-broadcast modules versions github.com/example/module # Show available versions for a module
go-broadcast modules validate # Validate all module configurations
go-broadcast modules list --from-db # List modules from database config
# Upgrade go-broadcast
go-broadcast upgrade # Upgrade to latest version
go-broadcast upgrade --check # Check for updates without upgrading
go-broadcast upgrade --force # Force upgrade even if already on latest
go-broadcast upgrade --verbose # Show release notes after upgrade
go-broadcast upgrade --use-binary # Install pre-built binary instead of go installπ File Transformations
transform:
repo_name: true # Updates Go module paths
variables:
SERVICE_NAME: "my-service" # {{SERVICE_NAME}} β my-service
ENVIRONMENT: "production" # ${ENVIRONMENT} β productionπ File & Directory Mapping Options
File Mapping:
files:
- src: "Makefile" # Copy to same location
dest: "Makefile"
- src: "template.md" # Rename during sync
dest: "README.md"
- src: "config/app.yml" # Move to different directory
dest: "configs/app.yml"
- dest: "old-config.json" # Delete file (src can be omitted)
delete: true
- dest: "deprecated.yml" # Delete file with explicit empty src
src: ""
delete: trueDirectory Mapping:
directories:
- src: ".github/workflows" # Basic directory sync
dest: ".github/workflows"
- src: ".github/actions" # Directory with exclusions
dest: ".github/actions"
exclude: ["*.out", "*.test", "go-coverage"]
- src: "docs" # Advanced directory options
dest: "documentation"
exclude: ["*.tmp", "**/draft/*"]
preserve_structure: true # Keep nested structure (default: true)
include_hidden: true # Include hidden files (default: true)
transform: # Apply transforms to all files
variables:
VERSION: "v2.0"
- dest: "legacy-docs" # Delete entire directory
delete: true
- dest: "old-scripts" # Delete directory (src can be omitted)
src: ""
delete: trueSmart Default Exclusions:
Automatically applied to all directories: *.out, *.test, *.exe, **/.DS_Store, **/tmp/*, **/.git
File and Directory Deletions:
- Set
delete: trueto remove files or directories from target repositories - When deleting, the
srcfield can be omitted or set to an empty string - Deletions are processed alongside regular sync operations
- Perfect for cleaning up deprecated files, old CI workflows, or restructuring projects
βοΈ Advanced Configuration
version: 1
groups:
- name: "Platform Configuration"
id: "platform-config"
priority: 1
enabled: true
source:
repo: "org/template-repo"
branch: "master"
# Global PR settings applied to ALL targets (merged with target-specific settings)
global:
pr_labels: ["automated-sync", "chore"]
pr_assignees: ["platform-team"]
pr_reviewers: ["platform-lead"]
pr_team_reviewers: ["infrastructure-team"]
# Default settings (fallback when no global or target settings)
defaults:
branch_prefix: "chore/sync-files"
pr_labels: ["maintenance"]
pr_assignees: ["maintainer1", "maintainer2"]
pr_reviewers: ["reviewer1", "reviewer2"]
pr_team_reviewers: ["platform-team"]
targets:
- repo: "org/target-repo"
branch: "main" # Target branch for PRs (defaults to repo's default branch)
files:
- src: ".github/workflows/ci.yml"
dest: ".github/workflows/ci.yml"
directories:
- src: ".github/actions"
dest: ".github/actions"
exclude: ["*.out", "go-coverage"]
transform:
repo_name: true
variables:
ENVIRONMENT: "production"
# Additional PR settings merged with global settings
# Final labels: ["automated-sync", "chore", "service-specific"]
pr_labels: ["service-specific"]
# Final assignees: ["platform-team", "service-owner"]
pr_assignees: ["service-owner"]
# Final reviewers: ["platform-lead", "service-reviewer"]
pr_reviewers: ["service-reviewer"]π Reusable File & Directory Lists
Define reusable file and directory lists to reduce configuration repetition when syncing the same files to multiple repositories.
version: 1
# Define reusable file lists
file_lists:
- id: "common-github-files"
name: "Common GitHub Files"
description: "Standard GitHub configuration files"
files:
- src: ".github/CODE_OF_CONDUCT.md"
dest: ".github/CODE_OF_CONDUCT.md"
- src: ".github/SECURITY.md"
dest: ".github/SECURITY.md"
- src: ".github/SUPPORT.md"
dest: ".github/SUPPORT.md"
- id: "editor-config"
name: "Editor Configuration"
description: "Editor and code formatting files"
files:
- src: ".editorconfig"
dest: ".editorconfig"
- src: ".gitattributes"
dest: ".gitattributes"
# Define reusable directory lists
directory_lists:
- id: "github-workflows"
name: "GitHub Actions Workflows"
description: "Standard CI/CD workflows"
directories:
- src: ".github/workflows"
dest: ".github/workflows"
exclude: ["*.tmp", "*.local"]
groups:
- name: "standard-sync"
id: "standard-sync"
source:
repo: "org/template-repo"
branch: "master"
targets:
# Use lists for multiple repos
- repo: "org/service-a"
file_list_refs: ["common-github-files", "editor-config"]
directory_list_refs: ["github-workflows"]
# Can still add inline files
files:
- src: "LICENSE"
dest: "LICENSE"
- repo: "org/service-b"
file_list_refs: ["common-github-files"]
directory_list_refs: ["github-workflows"]Benefits:
- Define file/directory lists once, use many times
- Easy updates - change lists in one place
- Mix references with inline files/directories
- Clear organization and reduced YAML duplication
β Cancel Sync Operations
When issues arise, you can cancel active sync operations to prevent unwanted changes. Group filtering makes cancellation much faster for multi-group configurations.
Cancel sync operations when issues arise:
# Cancel all active syncs (closes PRs and deletes branches)
go-broadcast cancel --config sync.yaml
# Cancel syncs for specific repositories only
go-broadcast cancel company/service1 company/service2
# Cancel syncs for specific groups (much faster for multi-group configs!)
go-broadcast cancel --groups "core"
go-broadcast cancel --groups "core,security"
# Cancel specific repository in a specific group
go-broadcast cancel --groups "third-party-libraries" company/repo1
# Cancel all except experimental group
go-broadcast cancel --skip-groups "experimental"
# Preview what would be cancelled without making changes
go-broadcast cancel --dry-run --config sync.yaml
# Close PRs but keep sync branches for later cleanup
go-broadcast cancel --keep-branches --config sync.yaml
# Add custom comment when closing PRs
go-broadcast cancel --comment "Cancelling due to template update" --config sync.yamlPerformance Benefits:
- Without group filtering: Scans all groups in your config (100+ API calls for 5 groups)
- With
--groups "core": Scans only the core group (20 API calls) - Result: 5x faster cancellation for multi-group configurations
π Global PR Assignment Configuration
The global section allows you to define PR assignments (labels, assignees, reviewers, team reviewers) that are merged with target-specific assignments rather than overridden. This provides powerful control over PR workflows across all repositories.
Merge Priority: global + target β defaults (fallback)
- Global settings apply to ALL target repositories
- Target settings are merged with global settings (duplicates removed)
- Default settings are used only when neither global nor target settings exist
version: 1
groups:
- name: "Workflow Distribution"
id: "workflow-dist"
priority: 1
enabled: true
source:
repo: "org/template-repo"
branch: "master"
# Applied to ALL PRs across all targets
global:
pr_labels: ["automated-sync", "chore"]
pr_assignees: ["platform-team"]
pr_reviewers: ["platform-lead"]
pr_team_reviewers: ["infrastructure-team"]
# Fallback settings (used only if no global/target assignments)
defaults:
branch_prefix: "chore/sync-files"
pr_labels: ["maintenance"]
targets:
# This repo gets ONLY global settings
- repo: "org/service-a"
files:
- src: ".github/workflows/ci.yml"
dest: ".github/workflows/ci.yml"
# Effective PR settings:
# Labels: ["automated-sync", "chore"]
# Assignees: ["platform-team"]
# Reviewers: ["platform-lead"]
# Team reviewers: ["infrastructure-team"]
# This repo gets global + target merged
- repo: "org/service-b"
files:
- src: ".github/workflows/ci.yml"
dest: ".github/workflows/ci.yml"
pr_labels: ["critical", "service-b"]
pr_assignees: ["service-b-owner"]
# Effective PR settings (merged):
# Labels: ["automated-sync", "chore", "critical", "service-b"]
# Assignees: ["platform-team", "service-b-owner"]
# Reviewers: ["platform-lead"] (from global)
# Team reviewers: ["infrastructure-team"] (from global)- Organization-wide standards: Apply consistent labels and assignees across all repositories
- Platform team oversight: Ensure platform team is always assigned to infrastructure changes
- Security requirements: Add security team reviewers to all template updates
- Compliance labeling: Automatically tag all PRs with audit/compliance labels
ποΈ Database-Backed Configuration
Use SQLite database instead of YAML files for dynamic configuration management, queryable config, and structured storage.
Quick Start:
# Initialize database
go-broadcast db init
# Import existing YAML configuration
go-broadcast db import sync.yaml
# Use database for all operations
go-broadcast sync --from-db
go-broadcast status --from-db --json
go-broadcast validate --from-db
go-broadcast cancel --from-db --groups "bitcoin-schema"Query your configuration:
# Which repos sync this file?
go-broadcast db query --file .github/workflows/ci.yml
# What files sync to this repo?
go-broadcast db query --repo company/service-name
# Which targets use this file list?
go-broadcast db query --file-list ai-filesExport back to YAML:
# Export entire configuration
go-broadcast db export --output backup.yaml
# Export specific group
go-broadcast db export --group core --output core-group.yamlBenefits:
- Queryable: Answer "which repos sync this file?" instantly
- Structured: SQLite schema with full validation
- Backwards compatible: Import/export YAML anytime
- Works everywhere:
--from-dbflag available on all commands - Transaction safety: All-or-nothing imports with rollback
See the complete Database Documentation for schema reference, CLI commands, and migration guide.
- Quick Start β Get up and running in 5 minutes with the Quick Start guide
- Configuration Guide β Complete guide to group-based configuration at docs/configuration-guide.md
- Database Backend β Structured configuration storage with SQLite at docs/database.md
- Repository Analytics β Track repo metrics and security alerts across organizations at docs/database.md
- Module-Aware Sync β Smart module versioning and synchronization at docs/module-sync.md
- Group Examples β Practical configuration patterns at docs/group-examples.md
- Usage Examples β Real-world scenarios in the Usage Examples section
- Directory Sync Guide β Complete guide to directory synchronization at docs/directory-sync.md
- Configuration Examples β Browse practical patterns in the examples directory
- Troubleshooting β Solve common issues with the troubleshooting guide
- API Reference β Dive into the godocs at pkg.go.dev/github.com/mrz1836/go-broadcast
- Integration Tests β End-to-end testing examples in test/integration
- Internal Utilities β Shared testing and validation utilities in internal packages
- Performance β Check the latest numbers in the Performance section
- Repository Features β Full list of built-in CI/CD, security, and DX features at .github/docs/repository-features.md
- GitHub Workflows β Workflow descriptions and the control center at .github/docs/workflows.md
π€ AI-Powered Text Generation (Optional)
go-broadcast can optionally use AI to generate intelligent PR descriptions and commit messages based on diff analysis.
Quick Setup:
export GO_BROADCAST_AI_ENABLED=true
export ANTHROPIC_API_KEY=sk-ant-...What Gets Generated:
| Feature | AI Output | Fallback |
|---|---|---|
| PR Body | Context-aware description | Static template |
| Commit Message | Descriptive conventional commit | sync: update N files |
Supported Providers:
| Provider | Env Var | Default Model |
|---|---|---|
| Anthropic | ANTHROPIC_API_KEY |
claude-sonnet-4-5-20250929 |
| OpenAI | OPENAI_API_KEY |
gpt-5.2 |
GEMINI_API_KEY |
gemini-3-pro-preview |
Performance Optimizations:
- Caching: Identical diffs across repos reuse cached AI responses (SHA256-keyed)
- Smart Truncation: Large diffs are intelligently truncated to stay within token limits
- Retry Logic: Transient failures are automatically retried with exponential backoff
Granular Control:
GO_BROADCAST_AI_PR_ENABLED=true # AI for PRs only
GO_BROADCAST_AI_COMMIT_ENABLED=true # AI for commits onlyImportant: AI failures never block sync operations - they fall back to static templates silently.
See the modular environment configuration for all AI configuration options.
Repository Features
This repository includes 25+ built-in features covering CI/CD, security, code quality, developer experience, and community tooling.
Library Deployment
This project uses goreleaser for streamlined binary and library deployment to GitHub. To get started, install it via:
brew install goreleaserThe release process is defined in the .goreleaser.yml configuration file.
Then create and push a new Git tag using:
magex version:bump push=true bump=patch branch=masterThis process ensures consistent, repeatable releases with properly versioned artifacts and citation metadata.
Build Commands
View all build commands
magex helpGitHub Workflows
All workflows are driven by modular configuration in .github/env/ β no YAML editing required.
Updating Dependencies
To update all dependencies in go.mod files, run:
# Update all dependencies
magex deps:update
# Update all modules
magex deps:update all-modulesThis command ensures all dependencies are brought up to date in a single step, including Go modules and any tools managed by MAGE-X. It is the recommended way to keep your development environment and CI in sync with the latest versions.
To update the environment files with the latest tool versions, run:
# Dry run to preview changes
magex updateToolVersions
# Apply minor/patch updates (major version upgrades are skipped by default)
UPDATE_VERSIONS=true magex updateToolVersions
# Apply all updates including major version upgrades (e.g., v1.x.x to v2.x.x)
UPDATE_VERSIONS=true ALLOW_MAJOR_UPGRADES=true magex updateToolVersionsPre-commit Hooks
Set up the Go-Pre-commit System to run the same formatting, linting, and tests defined in AGENTS.md before every commit:
# Install and set up pre-commit hooks
go install github.com/mrz1836/go-pre-commit/cmd/go-pre-commit@latest
go-pre-commit install
# Run pre-commit hooks manually
go-pre-commit run --all-filesThe system is configured via the modular environment configuration and provides 17x faster execution than traditional Python-based pre-commit hooks. See the complete documentation for details.
Logging and Debugging
go-broadcast provides comprehensive logging capabilities designed for debugging, monitoring, and troubleshooting. The logging system features intuitive verbose flags, component-specific debug modes, and automatic sensitive data redaction.
# Basic logging levels
go-broadcast sync --log-level debug # Debug level logging
go-broadcast sync --log-level info # Info level logging (default)
go-broadcast sync --log-level warn # Warning level logging
go-broadcast sync --log-level error # Error level logging
# Collect comprehensive diagnostic information
go-broadcast diagnose # Display diagnostic info to stdout
go-broadcast diagnose > diagnostics.json # Save diagnostics to file for supportNote: Advanced verbose flags (-v, -vv, -vvv) and component-specific debug flags (--debug-git, --debug-api, etc.) are planned features not yet implemented. The current implementation supports --log-level for basic debugging.
- ERROR: Critical failures that prevent operation
- WARN: Important issues that don't stop execution
- INFO: High-level operation progress (default)
- DEBUG: Detailed operation information (
--log-level debug)
All operations are timed automatically. Look for duration_ms in logs:
# Find slow operations
go-broadcast sync --log-format json 2>&1 | \
jq -r 'select(.duration_ms > 5000) | "\(.operation): \(.duration_ms)ms"'- All tokens and secrets are automatically redacted
- Audit trail for configuration changes and repository access
- No sensitive data is ever logged
Git Authentication Issues
# Enable debug logging to see git operations
go-broadcast sync --log-level debug
# Common indicators:
# - "Authentication failed" in git output
# - "Permission denied" errors
# - Check GH_TOKEN or GITHUB_TOKEN environment variablesAPI Rate Limiting
# Monitor operations with debug logging
go-broadcast sync --log-level debug 2>&1 | grep -i "rate"File Transformation Issues
# Use debug logging to see operation details
go-broadcast sync --log-level debug
# Debug output shows:
# - File operations
# - Configuration processing
# - Error detailsState Discovery Problems
# Enable debug logging for repository state information
go-broadcast sync --log-level debug
# Debug output includes:
# - Repository access attempts
# - Branch discovery
# - File discovery process# Save debug session
go-broadcast sync --log-level debug 2> debug-$(date +%Y%m%d-%H%M%S).log
# Review debug logs
go-broadcast sync --log-level debug 2>&1 | tee sync-debug.logThe diagnose command collects comprehensive system information for troubleshooting:
# Collect all diagnostic information (JSON format)
go-broadcast diagnose
# Information collected includes:
# - System details (OS, architecture, CPU count, hostname)
# - go-broadcast --version and build information
# - Git and GitHub CLI versions
# - Environment variables (sensitive data automatically redacted)
# - Configuration file status and validation results
# - Timestamp and runtime informationNote: JSON log format (--log-format json) is a planned feature. The diagnose command provides JSON output for system information.
| Variable | Description | Example |
|---|---|---|
NO_COLOR |
Disable colored output | 1 |
Note: Environment variables for log level and format are planned features not yet implemented.
For more detailed information, see the comprehensive logging guide and enhanced troubleshooting guide.
All unit tests and examples run via GitHub Actions and use Go version 1.24.x. View the configuration file.
Run all tests (fast):
magex testRun all tests with race detector (slower):
magex test:raceEnterprise-grade performance - Designed for high-scale repository management with zero-allocation critical paths.
| Operation | Performance | Memory |
|---|---|---|
| Binary Detection | 587M+ ops/sec | Zero allocations |
| Content Comparison | 239M+ ops/sec | Zero allocations |
| Cache Operations | 13.5M+ ops/sec | Minimal memory |
| Batch Processing | 23.8M+ ops/sec | Concurrent safe |
| Directory Sync | 32ms/1000 files | Linear scaling |
| Exclusion Engine | 107ns/op | Zero allocations |
Commands for Benchmarking
# Run quick benchmarks (CI default, <5 minutes)
magex bench
# Run heavy benchmarks manually (10-30 minutes)
# Includes worker pools, large datasets, real-world scenarios
mage benchHeavy
# Run all benchmarks (30-60 minutes)
mage benchAll
# Benchmark specific components
go test -bench=. -benchmem ./internal/algorithms
go test -bench=. -benchmem ./internal/cache
# Run heavy benchmarks with custom settings
go test -bench=. -benchmem -tags=bench_heavy -benchtime=1s ./...
# Try the profiling demo
go run ./cmd/profile_demoNote: Heavy benchmarks (worker pools with 1000+ tasks, large directory syncs, memory efficiency tests) are excluded from CI to prevent timeouts. Use mage benchHeavy for comprehensive performance testing during development.
Complete Benchmark Results & Profiling Tools
- π¬ 100+ Benchmarks covering all major components
- π CPU & Memory Profiling with detailed analysis
- π Performance Reports in HTML, JSON, and Markdown
- π Goroutine Analysis for concurrency debugging
- β‘ Zero-Allocation operations in critical paths
The following benchmarks were run on Apple M1 Max (updated January 2025):
| Benchmark | Operations | ns/op | B/op | allocs/op |
|---|---|---|---|---|
| Core Algorithms | ||||
| BinaryDetection (Small Text) | 5,852,616 | 204.5 | 0 | 0 |
| BinaryDetection (Large Text) | 179,217 | 6,606 | 0 | 0 |
| BinaryDetection (Small Binary) | 335,143,730 | 3.6 | 0 | 0 |
| BinaryDetection (Large Binary) | 587,204,924 | 2.0 | 0 | 0 |
| DiffOptimized (Identical) | 239,319,295 | 5.0 | 0 | 0 |
| DiffOptimized (Different) | 4,035,818 | 297.2 | 240 | 10 |
| DiffOptimized (Large Similar) | 250,452 | 4,711 | 5,492 | 7 |
| BatchProcessor | 23,842,558 | 54.3 | 25 | 1 |
| Cache Operations | ||||
| Cache Set | 6,067,380 | 177.4 | 48 | 4 |
| Cache Get (Hit) | 11,481,175 | 103.8 | 7 | 1 |
| Cache Get (Miss) | 13,565,466 | 89.4 | 32 | 2 |
| Cache GetOrLoad | 11,330,936 | 106.2 | 16 | 1 |
| Performance Profiling | ||||
| CaptureMemStats | 58,352 | 20,476 | 0 | 0 |
| CaptureMemoryStats | 3,475 | 302,402 | 107 | 4 |
| MeasureOperation | 4,032 | 316,467 | 107 | 4 |
go-broadcast is designed for efficiency:
- Binary detection executes 587M+ operations/second with zero allocations for binary files
- Content comparison performs 239M+ operations/second for identical files with zero allocations
- Cache operations handle 13.5M+ get operations/second with minimal memory usage
- Batch processing manages 23.8M+ operations/second for concurrent tasks
- Memory profiling captures detailed statistics at 58K+ operations/second
- Performance monitoring measures operations at 3K+ captures/second with comprehensive metrics
- Zero-allocation paths optimized algorithms avoid memory allocation in critical operations
- Concurrent operations sync multiple repositories simultaneously (configurable concurrency)
- GitHub API optimization reduces API calls through intelligent state discovery
- Memory efficiency most core operations use minimal allocations
- Test coverage maintained at >85% across core packages with comprehensive error handling
Performance varies based on GitHub API rate limits, network conditions, and repository sizes.
π Complete Performance Guide:
- Performance Guide - Complete benchmarking, profiling, and optimization reference
Read more about this Go project's code standards.
Read the AI Usage & Assistant Guidelines for details on how AI is used in this project and how to interact with AI assistants.
![]() |
|---|
| MrZ |
View the contributing guidelines and please follow the code of conduct.
All kinds of contributions are welcome π! The most basic way to show your support is to star π the project, or to raise issues π¬. You can also support this project by becoming a sponsor on GitHub π or by making a bitcoin donation to ensure this journey continues indefinitely! π
