feat: add Python complexity check, CI enforcement, and fix secret set bug (#5077)

marcusquinn · alex-solovyev · web-flow · commit 7d45d8104dab · 2026-03-16T16:04:55.000Z
* feat: add Python complexity check, CI enforcement, and fix secret set bug Three improvements to close the Codacy quality gate gap: 1. Python complexity check (linters-local.sh): runs Lizard (same tool Codacy uses, CCN>8) and Pyflakes locally before push. Advisory gate since Codacy is the hard gate for Python. 2. CI enforcement (code-quality.yml): new complexity-check job runs function-complexity, nesting-depth, file-size, and Python analysis on every PR. Blocks merges that exceed regression thresholds. 3. Fix secret-helper.sh read_secret_input(): print_info messages were going to stdout, which gets captured by $() in cmd_set. The prompt text was stored as the secret value in gopass. Fixed by redirecting all user-facing messages to stderr. Also documents verified Codacy API patterns (deltaStatistics, per-file issues, search by language) in codacy.md for the daily audit pipeline. * fix: bump function complexity threshold to 420 (baseline is 404) * fix: align all complexity thresholds with actual CI baseline (404/245/33) * fix: extract shared lint file-discovery, raise complexity threshold, fix pyflakes count - Extract .agents/scripts/lint-file-discovery.sh as single source of truth for file exclusion patterns (archived/, _archive/, supervisor-archived/), used by both CI workflow and linters-local.sh (CodeRabbit review feedback) - Raise function complexity threshold from 400 to 410 to accommodate current baseline of 404 violations (CI was failing at 400) - Fix pyflakes count using grep -c . instead of wc -l which returns 1 for empty input (Gemini review feedback) Addresses PR #5077 review feedback from CodeRabbit and Gemini. --------- Co-authored-by: Alexey <1556417+alex-solovyev@users.noreply.github.com>
diff --git a/.agents/scripts/lint-file-discovery.sh b/.agents/scripts/lint-file-discovery.sh
@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+# =============================================================================
+# Lint File Discovery - Shared file collection for quality checks
+# =============================================================================
+# Centralises file-discovery and exclusion logic used by both CI
+# (.github/workflows/code-quality.yml) and local linting (linters-local.sh).
+#
+# Exclusion policy (single source of truth):
+#   _archive/            - local archive directories
+#   archived/            - archived code (versioned for reference, not maintained)
+#   supervisor-archived/ - archived supervisor modules
+#
+# Usage (CI — git-based):
+#   source .agents/scripts/lint-file-discovery.sh
+#   lint_shell_files      # populates LINT_SH_FILES (newline-separated)
+#   lint_python_files     # populates LINT_PY_FILES (newline-separated)
+#
+# Usage (local — find-based, includes setup-modules/ and setup.sh):
+#   source .agents/scripts/lint-file-discovery.sh
+#   lint_shell_files_local   # populates LINT_SH_FILES (null-separated array)
+#   lint_python_files_local  # populates LINT_PY_FILES_LOCAL (null-separated array)
+#
+# Both modes apply identical exclusion patterns.
+# =============================================================================
+
+# Include guard
+[[ -n "${_LINT_FILE_DISCOVERY_LOADED:-}" ]] && return 0
+_LINT_FILE_DISCOVERY_LOADED=1
+
+# Exclusion pattern for grep -v (pipe-separated, used with grep -E)
+# Single source of truth for all archive/excluded directories.
+readonly LINT_EXCLUDE_PATTERN='_archive/|archived/|supervisor-archived/'
+
+# -----------------------------------------------------------------------------
+# Git-based discovery (CI mode)
+# -----------------------------------------------------------------------------
+# Uses git ls-files — works in CI where the full repo is checked out.
+# Results are newline-separated file paths.
+
+LINT_SH_FILES=""
+LINT_PY_FILES=""
+
+# Populate LINT_SH_FILES with all tracked .sh files, excluding archived dirs.
+lint_shell_files() {
+	LINT_SH_FILES=$(git ls-files '*.sh' | grep -Ev "$LINT_EXCLUDE_PATTERN")
+	return 0
+}
+
+# Populate LINT_PY_FILES with all tracked .py files, excluding archived dirs.
+lint_python_files() {
+	LINT_PY_FILES=$(git ls-files '*.py' | grep -Ev "$LINT_EXCLUDE_PATTERN" || true)
+	return 0
+}
+
+# -----------------------------------------------------------------------------
+# Find-based discovery (local mode)
+# -----------------------------------------------------------------------------
+# Uses find — includes setup-modules/ and setup.sh from repo root.
+# Results populate bash arrays for safe iteration over paths with spaces.
+
+LINT_SH_FILES_LOCAL=()
+LINT_PY_FILES_LOCAL=()
+
+# Populate LINT_SH_FILES_LOCAL array with shell files from .agents/scripts/,
+# setup-modules/, and setup.sh — excluding archived directories.
+lint_shell_files_local() {
+	LINT_SH_FILES_LOCAL=()
+	while IFS= read -r -d '' f; do
+		LINT_SH_FILES_LOCAL+=("$f")
+	done < <(find .agents/scripts -name "*.sh" \
+		-not -path "*/_archive/*" \
+		-not -path "*/archived/*" \
+		-not -path "*/supervisor-archived/*" \
+		-print0 2>/dev/null | sort -z)
+
+	# Include setup-modules/ (extracted setup.sh modules) if present
+	while IFS= read -r -d '' f; do
+		LINT_SH_FILES_LOCAL+=("$f")
+	done < <(find setup-modules -name "*.sh" -print0 2>/dev/null | sort -z)
+
+	# Include setup.sh entry point itself
+	if [[ -f "setup.sh" ]]; then
+		LINT_SH_FILES_LOCAL+=("setup.sh")
+	fi
+	return 0
+}
+
+# Populate LINT_PY_FILES_LOCAL array with Python files from .agents/scripts/,
+# excluding archived directories.
+lint_python_files_local() {
+	LINT_PY_FILES_LOCAL=()
+	while IFS= read -r -d '' f; do
+		LINT_PY_FILES_LOCAL+=("$f")
+	done < <(find .agents/scripts -name "*.py" \
+		-not -path "*/_archive/*" \
+		-not -path "*/archived/*" \
+		-not -path "*/supervisor-archived/*" \
+		-print0 2>/dev/null | sort -z)
+	return 0
+}
diff --git a/.agents/scripts/linters-local.sh b/.agents/scripts/linters-local.sh
@@ -19,6 +19,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit
 source "${SCRIPT_DIR}/shared-constants.sh"
+source "${SCRIPT_DIR}/lint-file-discovery.sh"
 
 set -euo pipefail
 
@@ -41,50 +42,35 @@ readonly MAX_STRING_LITERAL_ISSUES=2300
 # Thresholds are set above the current baseline to catch regressions, not existing debt.
 # Existing debt is tracked by the code-simplifier (priority 8, human-gated).
 #
-# Baseline (2026-03-16): 373 functions >100 lines, 222 files >8 nesting, 26 files >1500 lines
+# Baseline (2026-03-16): 404 functions >100 lines, 245 files >8 nesting, 33 files >1500 lines
 # These thresholds allow the current baseline but block significant new additions.
 # Reduce thresholds as existing debt is paid down.
 #
-# - Function length: warn >50, block >100. Threshold allows current 373 + small margin.
-# - Nesting depth: warn >5, block >8. Threshold allows current 222 + small margin.
-# - File size: warn >800, block >1500. Threshold allows current 26 + small margin.
+# - Function length: warn >50, block >100. Threshold allows current 404 + small margin.
+# - Nesting depth: warn >5, block >8. Threshold allows current 245 + small margin.
+# - File size: warn >800, block >1500. Threshold allows current 33 + small margin.
 readonly MAX_FUNCTION_LENGTH_WARN=50
 readonly MAX_FUNCTION_LENGTH_BLOCK=100
-readonly MAX_FUNCTION_LENGTH_VIOLATIONS=400
+readonly MAX_FUNCTION_LENGTH_VIOLATIONS=420
 readonly MAX_NESTING_DEPTH_WARN=5
 readonly MAX_NESTING_DEPTH_BLOCK=8
-readonly MAX_NESTING_VIOLATIONS=230
+readonly MAX_NESTING_VIOLATIONS=260
 readonly MAX_FILE_LINES_WARN=800
 readonly MAX_FILE_LINES_BLOCK=1500
-readonly MAX_FILE_SIZE_VIOLATIONS=30
+readonly MAX_FILE_SIZE_VIOLATIONS=40
 
 print_header() {
 	echo -e "${BLUE}Local Linters - Fast Offline Quality Checks${NC}"
 	echo -e "${BLUE}================================================================${NC}"
 	return 0
 }
 
-# Collect all shell scripts to lint, including modularised subdirectories
-# (e.g. memory/, supervisor-modules/, setup/) but excluding archived code.
-# Excludes: _archive/, archived/, supervisor-archived/ — these are versioned
-# for reference but not actively maintained (reduces lint noise).
-# Also includes setup-modules/ and setup.sh from the repo root.
-# Populates the ALL_SH_FILES array for use by check functions.
+# Collect all shell scripts to lint via shared file-discovery helper.
+# Exclusion policy is centralised in lint-file-discovery.sh (single source of
+# truth shared with CI). Populates ALL_SH_FILES array for check functions.
 collect_shell_files() {
-	ALL_SH_FILES=()
-	while IFS= read -r -d '' f; do
-		ALL_SH_FILES+=("$f")
-	done < <(find .agents/scripts -name "*.sh" -not -path "*/_archive/*" -not -path "*/archived/*" -not -path "*/supervisor-archived/*" -print0 2>/dev/null | sort -z)
-
-	# Include setup-modules/ (extracted setup.sh modules) if present
-	while IFS= read -r -d '' f; do
-		ALL_SH_FILES+=("$f")
-	done < <(find setup-modules -name "*.sh" -print0 2>/dev/null | sort -z)
-
-	# Include setup.sh entry point itself
-	if [[ -f "setup.sh" ]]; then
-		ALL_SH_FILES+=("setup.sh")
-	fi
+	lint_shell_files_local
+	ALL_SH_FILES=("${LINT_SH_FILES_LOCAL[@]}")
 	return 0
 }
 
@@ -927,13 +913,9 @@ check_file_size() {
 		append_file_size_result "$file" "$tmp_file" "$MAX_FILE_LINES_WARN" "$MAX_FILE_LINES_BLOCK"
 	done
 
-	# Also check Python files in the scripts directory
-	local py_files=()
-	while IFS= read -r -d '' f; do
-		py_files+=("$f")
-	done < <(find .agents/scripts -name "*.py" -not -path "*/_archive/*" -not -path "*/archived/*" -print0 2>/dev/null | sort -z)
-
-	for file in "${py_files[@]}"; do
+	# Also check Python files in the scripts directory (shared discovery)
+	lint_python_files_local
+	for file in "${LINT_PY_FILES_LOCAL[@]}"; do
 		append_file_size_result "$file" "$tmp_file" "$MAX_FILE_LINES_WARN" "$MAX_FILE_LINES_BLOCK"
 	done
 
@@ -966,6 +948,86 @@ check_file_size() {
 	return 1
 }
 
+# =============================================================================
+# Python Complexity Check (Codacy alignment — GH#4939)
+# =============================================================================
+# Codacy uses Lizard for cyclomatic complexity analysis on Python files.
+# This local check runs the same tool with the same threshold (CCN > 8)
+# to catch complexity issues before they reach Codacy.
+# Also checks for unused imports (pyflakes) and security patterns (semgrep-lite).
+
+check_python_complexity() {
+	echo -e "${BLUE}Checking Python Complexity (Codacy alignment)...${NC}"
+
+	# Collect Python files (shared discovery)
+	lint_python_files_local
+	local py_files=("${LINT_PY_FILES_LOCAL[@]}")
+
+	if [[ ${#py_files[@]} -eq 0 ]]; then
+		print_info "No Python files found in .agents/scripts/"
+		return 0
+	fi
+
+	local violations=0
+	local warnings=0
+
+	# Check 1: Lizard cyclomatic complexity (same tool Codacy uses)
+	if command -v lizard &>/dev/null; then
+		local lizard_out
+		lizard_out=$(lizard --CCN 8 --warnings_only "${py_files[@]}" 2>/dev/null || true)
+		if [[ -n "$lizard_out" ]]; then
+			local lizard_count
+			lizard_count=$(echo "$lizard_out" | grep -c "warning:" 2>/dev/null || echo "0")
+			lizard_count=${lizard_count//[^0-9]/}
+			lizard_count=${lizard_count:-0}
+			violations=$((violations + lizard_count))
+
+			if [[ "$lizard_count" -gt 0 ]]; then
+				print_warning "Lizard: $lizard_count functions exceed cyclomatic complexity 8"
+				echo "$lizard_out" | grep "warning:" | head -10
+				if [[ "$lizard_count" -gt 10 ]]; then
+					echo "  ... and $((lizard_count - 10)) more"
+				fi
+			fi
+		fi
+	else
+		print_info "Lizard not installed (pipx install lizard) — skipping cyclomatic complexity"
+	fi
+
+	# Check 2: Pyflakes for unused imports (Codacy uses Prospector/pyflakes)
+	if command -v pyflakes &>/dev/null; then
+		local pyflakes_out
+		pyflakes_out=$(pyflakes "${py_files[@]}" 2>/dev/null || true)
+		if [[ -n "$pyflakes_out" ]]; then
+			local pyflakes_count
+			pyflakes_count=$(echo "$pyflakes_out" | grep -c . 2>/dev/null || echo "0")
+			pyflakes_count=${pyflakes_count//[^0-9]/}
+			pyflakes_count=${pyflakes_count:-0}
+			warnings=$((warnings + pyflakes_count))
+
+			if [[ "$pyflakes_count" -gt 0 ]]; then
+				print_warning "Pyflakes: $pyflakes_count issues (unused imports, undefined names)"
+				echo "$pyflakes_out" | head -10
+				if [[ "$pyflakes_count" -gt 10 ]]; then
+					echo "  ... and $((pyflakes_count - 10)) more"
+				fi
+			fi
+		fi
+	else
+		print_info "Pyflakes not installed (pipx install pyflakes) — skipping import checks"
+	fi
+
+	local total=$((violations + warnings))
+	# Python complexity is advisory for now — Codacy is the hard gate.
+	# This gives early feedback without blocking local development.
+	if [[ "$total" -eq 0 ]]; then
+		print_success "Python complexity: ${#py_files[@]} files checked, no issues"
+	else
+		print_warning "Python complexity: $total issues ($violations complexity, $warnings pyflakes)"
+	fi
+	return 0
+}
+
 check_remote_cli_status() {
 	print_info "Remote Audit CLIs Status (use /code-audit-remote for full analysis)..."
 
@@ -1368,6 +1430,11 @@ main() {
 		echo ""
 	fi
 
+	if ! should_skip_gate "python-complexity"; then
+		check_python_complexity || exit_code=1
+		echo ""
+	fi
+
 	check_remote_cli_status
 	echo ""
 
diff --git a/.agents/scripts/secret-helper.sh b/.agents/scripts/secret-helper.sh
@@ -236,26 +236,31 @@ read_secret_input() {
 	local name="$1"
 	local value=""
 
+	# CRITICAL: All user-facing messages MUST go to stderr (>&2), not stdout.
+	# This function's stdout is captured by $() in cmd_set — any print_info
+	# or echo to stdout gets stored as part of the secret value. This was the
+	# root cause of GH#4939: prompt text was stored in gopass instead of the
+	# actual secret.
 	if [[ -t 0 ]]; then
-		print_info "Enter secret value for $name (input hidden):"
-		print_info "Paste only the secret value, then press Enter"
+		print_info "Enter secret value for $name (input hidden):" >&2
+		print_info "Paste only the secret value, then press Enter" >&2
 		IFS= read -rs value || true
-		echo ""
+		echo "" >&2
 	else
 		IFS= read -r value || true
 	fi
 
 	value="${value%$'\r'}"
 
 	if [[ -z "$value" ]]; then
-		print_error "No secret value received for $name"
-		print_info "Run again and paste only the secret value"
+		print_error "No secret value received for $name" >&2
+		print_info "Run again and paste only the secret value" >&2
 		return 1
 	fi
 
 	if [[ "$value" =~ ^[[:space:]]*aidevops[[:space:]]+secret[[:space:]]+set([[:space:]]|$) ]]; then
-		print_error "Input for $name looks like a command, not a secret value"
-		print_info "Paste the secret value itself, not 'aidevops secret set ...'"
+		print_error "Input for $name looks like a command, not a secret value" >&2
+		print_info "Paste the secret value itself, not 'aidevops secret set ...'" >&2
 		return 1
 	fi
 
diff --git a/.agents/tools/code-review/codacy.md b/.agents/tools/code-review/codacy.md
@@ -52,14 +52,41 @@ catching the same issues locally before code reaches Codacy:
 | `nesting-depth` | Cyclomatic complexity | >5 levels | >8 levels | `nesting-depth` |
 | `file-size` | File length | >800 lines | >1500 lines | `file-size` |
 
+Additionally, `python-complexity` runs Lizard (same tool Codacy uses) and Pyflakes locally:
+
+| Check | Codacy tool | Threshold | Gate |
+|-------|-------------|-----------|------|
+| `python-complexity` | Lizard CCN | >8 (advisory) | `python-complexity` |
+
 These gates are set above the current baseline to catch regressions. As existing debt
 is paid down (via code-simplifier issues), reduce the thresholds. The gates also cover
 Python files in `.agents/scripts/` for file-size checks.
 
-Skip via bundle config: add `"function-complexity"`, `"nesting-depth"`, or `"file-size"`
-to `skip_gates` in the project bundle.
+CI enforcement: `.github/workflows/code-quality.yml` runs the same checks on every PR
+via the `complexity-check` job, blocking merges that exceed thresholds.
+
+Skip via bundle config: add `"function-complexity"`, `"nesting-depth"`, `"file-size"`,
+or `"python-complexity"` to `skip_gates` in the project bundle.
+
+## Codacy API Patterns (verified working)
+
+```bash
+# Commit delta statistics (new issues count + complexity delta)
+curl -s -H "api-token: $CODACY_API_TOKEN" \
+  "https://app.codacy.com/api/v3/analysis/organizations/gh/marcusquinn/repositories/aidevops/commits/<SHA>/deltaStatistics"
+
+# Per-file new issues (paginate with cursor)
+curl -s -H "api-token: $CODACY_API_TOKEN" \
+  "https://app.codacy.com/api/v3/analysis/organizations/gh/marcusquinn/repositories/aidevops/commits/<SHA>/files?limit=100"
+# Filter: .data[] | select(.quality.deltaNewIssues > 0)
+
+# Search all issues (POST, filter by language)
+curl -s -H "api-token: $CODACY_API_TOKEN" -H "Content-Type: application/json" \
+  -X POST "https://app.codacy.com/api/v3/analysis/organizations/gh/marcusquinn/repositories/aidevops/issues/search?limit=50" \
+  -d '{"languages": ["Python"]}'
+```
 
-**Updating via API:**
+**Updating quality gate via API:**
 
 ```bash
 # Update PR gate
diff --git a/.github/workflows/code-quality.yml b/.github/workflows/code-quality.yml
