feat: add comprehensive skill validation CI by CybotTM · Pull Request #21 · netresearch/skill-repo-skill

CybotTM · 2026-02-25T14:08:22Z

Summary

Enhanced validate-skill.sh with 16+ checks covering all issues from the skill standardization session
Added reusable validate.yml workflow that all 26 skill repos can call
Added pre-commit hook for local validation before each commit
Fixed skill-repo-skill's own composer.json path and plugin.json author.url

Checks Added

Category	Check	Type
SKILL.md	Only `name` + `description` in frontmatter	Error
SKILL.md	Word count <= 500	Error
SKILL.md	Description starts with "Use when"	Error
SKILL.md	Name pattern (lowercase-hyphens, max 64)	Error
composer.json	Exists	Error
composer.json	Type is `ai-agent-skill`	Error
composer.json	Name matches `netresearch/agent-*`	Error
composer.json	`extra.ai-agent-skill` path exists	Error
plugin.json	Exists at `.claude-plugin/plugin.json`	Error
plugin.json	Name matches SKILL.md	Error
plugin.json	`skills` is array	Error
plugin.json	Skill paths exist	Error
plugin.json	`author.url` is company URL	Error
Files	README.md, LICENSE, .gitignore, release.yml	Error
Files	No composer.lock	Error

Usage by Other Repos

Each repo adds one job to their lint.yml:

  validate:
    name: Skill Validation
    uses: netresearch/skill-repo-skill/.github/workflows/validate.yml@main

Test plan

Validated against skill-repo-skill itself (all green)
Tested against cli-tools-skill and security-audit-skill (correctly detects issues)
CI passes on this PR
Deploy to all 25 other repos after merge

Add checks for: frontmatter-only fields, word count, description prefix, composer.json path validation and name pattern, plugin.json structure, cross-file consistency, .gitignore and release.yml presence. Also fix skill-repo-skill's own composer.json path and plugin.json author.url to pass the new validation. Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de> Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>

Add reusable workflow that other skill repos can call with: uses: netresearch/skill-repo-skill/.github/workflows/validate.yml@main Add validate job to skill-repo-skill's own lint.yml. Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de> Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>

Hook auto-discovers validate-skill.sh in the repo and runs it before each commit. Template provided for other repos to copy. Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de> Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>

Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de> Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>

gemini-code-assist · 2026-02-25T14:08:26Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copilot

Pull request overview

Adds a comprehensive, reusable validation pipeline for Netresearch skill repositories, enforcing standardized structure and metadata via a shared CI workflow and local git hooks.

Changes:

Expanded skills/skill-repo/scripts/validate-skill.sh to validate SKILL.md, composer.json, plugin.json, and required repo files with cross-file consistency checks.
Added a reusable GitHub Actions workflow (.github/workflows/validate.yml) and wired validation into this repo’s CI (lint.yml).
Added pre-commit hook support (hook file + template) and aligned repo metadata (composer.json path + plugin.json author URL).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
skills/skill-repo/templates/pre-commit.template	Provides a distributable pre-commit hook template to run validation locally.
skills/skill-repo/scripts/validate-skill.sh	Implements the new, stricter validation rules and cross-file checks.
docs/plans/2026-02-25-skill-validation-ci.md	Documents the rollout plan and validation checklist for the CI/hook approach.
composer.json	Updates `extra.ai-agent-skill` to point at the actual SKILL.md path in this repo layout.
Build/hooks/pre-commit	Adds an installable pre-commit hook to run validation before commits.
.github/workflows/validate.yml	Introduces the reusable workflow that other skill repos can call.
.github/workflows/lint.yml	Adds a validation job to run the script in CI for this repo.
.claude-plugin/plugin.json	Updates `author.url` to the required Netresearch company URL.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-25T14:16:47Z

skills/skill-repo/scripts/validate-skill.sh

+    # Name pattern
+    COMP_NAME=$(python3 -c "import json; print(json.load(open('$REPO_DIR/composer.json')).get('name',''))" 2>/dev/null || echo "")
+    if [[ "$COMP_NAME" == netresearch/agent-* ]]; then
+        success "composer.json name: $COMP_NAME"
    else
-        error "composer.json type should be 'ai-agent-skill'"
+        error "composer.json name must match netresearch/agent-*: $COMP_NAME"
    fi


This script relies on python3 for JSON parsing but doesn’t validate that python3 is available. On systems without Python 3, these commands will fall back to empty strings and produce misleading validation errors. Consider adding an early command -v python3 check that errors with a clear message when Python 3 isn’t installed.

Copilot · 2026-02-25T14:16:48Z

.github/workflows/validate.yml

+      - name: Checkout validation script
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          repository: netresearch/skill-repo-skill


The workflow checks out netresearch/skill-repo-skill without pinning a ref. This makes validation non-deterministic (the script version can change independently of the reusable workflow ref that callers use) and can lead to unexpected failures. Consider checking out the validation script at the same ref as the reusable workflow (or documenting that it intentionally always tracks the default branch).

Suggested change

repository: netresearch/skill-repo-skill

repository: netresearch/skill-repo-skill

ref: main

Copilot · 2026-02-25T14:16:48Z

docs/plans/2026-02-25-skill-validation-ci.md

@@ -0,0 +1,517 @@
+# Skill Validation CI Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.


This plan document includes tool-specific instructions addressed to an AI agent ("For Claude" / "REQUIRED SUB-SKILL"). If this repo’s docs are intended for humans, consider removing or moving these AI-invocation notes to avoid confusion and to keep implementation plans tool-agnostic.

Suggested change

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

> Implementation note: This plan is intended to be executed step by step as written.

Copilot · 2026-02-25T14:16:48Z

docs/plans/2026-02-25-skill-validation-ci.md

+### Task 1: Write Enhanced validate-skill.sh
+
+**Files:**
+- Modify: `/home/cybot/projects/skill-repo-skill/main/skills/skill-repo/scripts/validate-skill.sh`


The plan references absolute local filesystem paths (e.g. /home/cybot/projects/...). These won’t apply to other contributors and can make the plan harder to follow. Consider using repo-relative paths instead.

Suggested change

- Modify: `/home/cybot/projects/skill-repo-skill/main/skills/skill-repo/scripts/validate-skill.sh`

- Modify: `main/skills/skill-repo/scripts/validate-skill.sh`

Copilot · 2026-02-25T14:16:49Z

skills/skill-repo/scripts/validate-skill.sh

+    # Frontmatter delimiter
+    if head -1 "$SKILL_FILE" | grep -q "^---$"; then
+        success "SKILL.md has frontmatter"

-    # Check frontmatter
-    if head -1 "$SKILL_DIR/SKILL.md" | grep -q "^---$"; then
-        success "SKILL.md has frontmatter delimiter"
+        # Extract frontmatter fields (between first two --- lines)
+        FRONTMATTER=$(sed -n '2,/^---$/{ /^---$/d; p; }' "$SKILL_FILE")
+


The SKILL.md frontmatter check only verifies the opening --- delimiter, but does not verify that a closing --- delimiter exists. If the closing delimiter is missing, FRONTMATTER will include the rest of the file and the script may incorrectly treat the document as having valid frontmatter. Consider explicitly checking for a second --- line (e.g., within the first N lines) and erroring if it’s missing before extracting fields.

Copilot · 2026-02-25T14:16:49Z

skills/skill-repo/scripts/validate-skill.sh

+    if [[ -n "$AUTHOR_URL" ]]; then
+        if [[ "$AUTHOR_URL" == "https://www.netresearch.de" ]]; then
+            success "plugin.json author.url is correct"
+        else
+            error "plugin.json author.url must be https://www.netresearch.de (got: $AUTHOR_URL)"
+        fi


The plugin.json author.url check only runs when author.url is non-empty. Per the documented rules, missing author.url should be treated as an error as well; otherwise repos can omit it and still pass validation.

Suggested change

if [[ -n "$AUTHOR_URL" ]]; then

if [[ "$AUTHOR_URL" == "https://www.netresearch.de" ]]; then

success "plugin.json author.url is correct"

else

error "plugin.json author.url must be https://www.netresearch.de (got: $AUTHOR_URL)"

fi

if [[ -z "$AUTHOR_URL" ]]; then

error "plugin.json author.url is missing or empty; it must be https://www.netresearch.de"

elif [[ "$AUTHOR_URL" == "https://www.netresearch.de" ]]; then

success "plugin.json author.url is correct"

else

error "plugin.json author.url must be https://www.netresearch.de (got: $AUTHOR_URL)"

Copilot · 2026-02-25T14:16:50Z

skills/skill-repo/scripts/validate-skill.sh

+    COMP_NAME=$(python3 -c "import json; print(json.load(open('$REPO_DIR/composer.json')).get('name',''))" 2>/dev/null || echo "")
+    if [[ "$COMP_NAME" == netresearch/agent-* ]]; then
+        success "composer.json name: $COMP_NAME"
    else
-        error "composer.json type should be 'ai-agent-skill'"
+        error "composer.json name must match netresearch/agent-*: $COMP_NAME"
    fi

-    # Check for skill plugin dependency
-    if grep -q "composer-agent-skill-plugin" "$SKILL_DIR/composer.json"; then
+    # Plugin dependency
+    if grep -q "composer-agent-skill-plugin" "$REPO_DIR/composer.json"; then
        success "composer.json requires skill plugin"
    else
        warning "composer.json should require netresearch/composer-agent-skill-plugin"
    fi

-    # Check for ai-agent-skill extra
-    if grep -q '"ai-agent-skill"' "$SKILL_DIR/composer.json"; then
-        success "composer.json has ai-agent-skill extra"
+    # ai-agent-skill extra path exists
+    SKILL_PATH=$(python3 -c "import json; print(json.load(open('$REPO_DIR/composer.json')).get('extra',{}).get('ai-agent-skill',''))" 2>/dev/null || echo "")


The python3 -c invocations here interpolate $REPO_DIR directly into a single-quoted Python string (e.g. open('$REPO_DIR/composer.json')), which allows an attacker who can control the REPO_DIR argument to inject arbitrary Python code by including a quote and payload in the path. This would result in arbitrary code execution when the validation script is run against a maliciously named directory or when REPO_DIR is otherwise attacker-controlled. To harden this, avoid embedding shell variables inside the Python source string and instead pass the JSON path as a command-line argument or via environment/sys.argv and open it safely inside Python.

Copilot · 2026-02-25T14:16:50Z

skills/skill-repo/scripts/validate-skill.sh

+    PLUGIN_NAME=$(python3 -c "import json; print(json.load(open('$PLUGIN_FILE')).get('name',''))" 2>/dev/null || echo "")
+    if [[ -n "$NAME" ]] && [[ "$PLUGIN_NAME" == "$NAME" ]]; then
+        success "plugin.json name matches SKILL.md: $PLUGIN_NAME"
+    elif [[ -n "$NAME" ]]; then
+        error "plugin.json name '$PLUGIN_NAME' does not match SKILL.md name '$NAME'"
    fi
-done

-# Summary
+    # Skills is array
+    SKILLS_TYPE=$(python3 -c "import json; s=json.load(open('$PLUGIN_FILE')).get('skills'); print('array' if isinstance(s, list) else type(s).__name__)" 2>/dev/null || echo "unknown")
+    if [[ "$SKILLS_TYPE" == "array" ]]; then
+        success "plugin.json skills is array"
+
+        # Check each skill path exists as directory
+        MISSING_PATHS=$(python3 -c "
+import json, os
+data = json.load(open('$PLUGIN_FILE'))
+for path in data.get('skills', []):
+    full = os.path.join('$REPO_DIR', path)
+    if not os.path.isdir(full):
+        print(path)
+" 2>/dev/null || true)
+        if [[ -z "$MISSING_PATHS" ]]; then
+            success "All plugin.json skill paths exist"
+        else
+            while IFS= read -r p; do
+                error "plugin.json skill path missing: $p"
+            done <<< "$MISSING_PATHS"
+        fi
+    else
+        error "plugin.json skills must be an array (got: $SKILLS_TYPE)"
+    fi
+
+    # Author URL
+    AUTHOR_URL=$(python3 -c "import json; print(json.load(open('$PLUGIN_FILE')).get('author',{}).get('url',''))" 2>/dev/null || echo "")


All python3 -c calls in this block interpolate $PLUGIN_FILE and $REPO_DIR directly into single-quoted Python strings (e.g. open('$PLUGIN_FILE'), os.path.join('$REPO_DIR', path)), so a crafted value containing quotes or control characters can break out of the string and inject arbitrary Python code. If an attacker can influence the repository path or plugin file path the script is pointed at, running validate-skill.sh would then execute their Python payload with the developer’s permissions. To mitigate this, keep the inline Python code static and pass file paths as arguments (or via sys.argv/environment) rather than concatenating shell variables into the Python source string.

CybotTM added 4 commits February 25, 2026 15:07

docs: add skill validation CI implementation plan

6a76eda

Signed-off-by: Sebastian Mendel <sebastian.mendel@netresearch.de> Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>

CybotTM requested a review from Copilot February 25, 2026 14:08

Copilot started reviewing on behalf of CybotTM February 25, 2026 14:09 View session

CybotTM merged commit 2f1cbbe into main Feb 25, 2026
8 checks passed

CybotTM deleted the feat/skill-validation-ci branch February 25, 2026 14:14

Copilot AI reviewed Feb 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add comprehensive skill validation CI#21

feat: add comprehensive skill validation CI#21
CybotTM merged 4 commits intomainfrom
feat/skill-validation-ci

CybotTM commented Feb 25, 2026

Uh oh!

gemini-code-assist bot commented Feb 25, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	repository: netresearch/skill-repo-skill
	repository: netresearch/skill-repo-skill
	ref: main

		@@ -0,0 +1,517 @@
		# Skill Validation CI Implementation Plan

		> For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

	> For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
	> Implementation note: This plan is intended to be executed step by step as written.

	- Modify: `/home/cybot/projects/skill-repo-skill/main/skills/skill-repo/scripts/validate-skill.sh`
	- Modify: `main/skills/skill-repo/scripts/validate-skill.sh`

Conversation

CybotTM commented Feb 25, 2026

Summary

Checks Added

Usage by Other Repos

Test plan

Uh oh!

gemini-code-assist bot commented Feb 25, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants