Skip to content

Commit bff9bce

Browse files
zircotezircote
committed
feat: add autoresearch eval suites for all 4 skills
- citation-cff: 7 evals, 38 expectations (create, validate, export, sync) - readme-enhance: 7 evals, 37 expectations (badges, infographic, modes) - repo-metadata: 7 evals, 39 expectations (description, topics, limits) - social-preview: 8 evals, 41 expectations (SVG, palettes, providers) All skills scored 1.0 on baseline evaluation (155/155 passed).
1 parent 4d2e2e7 commit bff9bce

4 files changed

Lines changed: 433 additions & 0 deletions

File tree

‎skills/citation-cff/evals/evals.json‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
{
2+
"skill_name": "citation-cff",
3+
"evals": [
4+
{
5+
"id": 1,
6+
"prompt": "Create a CITATION.cff for this Node.js project. The repo has a package.json with name 'my-cli-tool', version '2.1.0', author 'Jane Smith <jane@example.com>', license 'MIT', and description 'A command-line tool for data processing'. Use --dry-run mode.",
7+
"expected_output": "A valid CFF 1.2.0 CITATION.cff file displayed in dry-run mode with all required fields populated from the package.json metadata.",
8+
"files": [],
9+
"expectations": [
10+
"Output contains 'cff-version: 1.2.0' as the schema version",
11+
"Output contains 'type: software' as the CFF type",
12+
"Output contains 'title: \"my-cli-tool\"' or 'title: my-cli-tool' matching the package name",
13+
"Output contains 'version: \"2.1.0\"' or 'version: 2.1.0' matching the package version",
14+
"Author entry splits 'Jane Smith' into family-names: 'Smith' and given-names: 'Jane'",
15+
"Output contains 'license: MIT' or 'license: \"MIT\"' as a valid SPDX identifier"
16+
]
17+
},
18+
{
19+
"id": 2,
20+
"prompt": "Create a CITATION.cff for this Python project. It has a pyproject.toml with [project] name = 'dataflow', version = '0.9.3', description = 'Streaming data pipeline framework', license = {text = 'Apache-2.0'}, and authors = [{name = 'Carlos Ruiz', email = 'carlos@example.org'}, {name = 'Wei Zhang'}]. Use --dry-run.",
21+
"expected_output": "A valid CFF 1.2.0 CITATION.cff displayed in dry-run mode with metadata extracted from pyproject.toml, including both authors properly formatted.",
22+
"files": [],
23+
"expectations": [
24+
"Output contains 'cff-version: 1.2.0'",
25+
"Output contains 'title: \"dataflow\"' or 'title: dataflow'",
26+
"Output contains 'version: \"0.9.3\"' or 'version: 0.9.3'",
27+
"Output lists two authors with correct family-names/given-names splits: Ruiz/Carlos and Zhang/Wei",
28+
"Output contains 'license: Apache-2.0' or 'license: \"Apache-2.0\"'",
29+
"Output includes a 'date-released' field in YYYY-MM-DD format"
30+
]
31+
},
32+
{
33+
"id": 3,
34+
"prompt": "Validate this CITATION.cff file:\n\ncff-version: 1.2.0\nmessage: \"Please cite this software.\"\ntitle: \"broken-project\"\ntype: software\nauthors:\n - given-names: \"Alice\"\ndate-released: \"2024-1-5\"\nversion: 3.0\ndoi: \"not-a-doi\"",
35+
"expected_output": "A validation report identifying multiple errors: the author entry is missing family-names, date-released has an invalid format, version should be a string not a number, and the DOI does not match the expected pattern.",
36+
"files": [],
37+
"expectations": [
38+
"Report identifies that the author entry is missing 'family-names' (each author must have family-names + given-names OR name)",
39+
"Report identifies that 'date-released' value '2024-1-5' is invalid and should be '2024-01-05' (YYYY-MM-DD with zero-padding)",
40+
"Report identifies that 'version' value 3.0 is a number and should be a quoted string",
41+
"Report identifies that 'doi' value 'not-a-doi' does not match the DOI pattern 10.XXXX/...",
42+
"Report ends with a Result line showing FAIL and a count of errors"
43+
]
44+
},
45+
{
46+
"id": 4,
47+
"prompt": "Export the citation as BibTeX format from this CITATION.cff:\n\ncff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\ntitle: \"fast-parser\"\ntype: software\nauthors:\n - family-names: \"Kim\"\n given-names: \"Soo-Jin\"\n - family-names: \"Okafor\"\n given-names: \"Emeka\"\nversion: \"4.2.1\"\ndate-released: \"2025-08-10\"\nurl: \"https://github.com/example/fast-parser\"\nlicense: \"BSD-3-Clause\"\ndoi: \"10.5281/zenodo.9876543\"",
48+
"expected_output": "A BibTeX @software entry with correct field mappings, including both authors in 'Family, Given' format, the DOI, version, date, URL, and a properly formed BibTeX key.",
49+
"files": [],
50+
"expectations": [
51+
"Output contains '@software{' indicating the correct BibTeX entry type",
52+
"Output contains 'author' field with both authors in 'Family, Given' format: 'Kim, Soo-Jin' and 'Okafor, Emeka'",
53+
"BibTeX key includes 'kim' (first author family name lowercase) and '2025' (year)",
54+
"Output contains 'doi = {10.5281/zenodo.9876543}' or equivalent DOI field",
55+
"Output contains 'version = {4.2.1}' or equivalent version field",
56+
"Output contains 'license = {BSD-3-Clause}' or equivalent license field"
57+
]
58+
},
59+
{
60+
"id": 5,
61+
"prompt": "Export this CITATION.cff as APA format:\n\ncff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\ntitle: \"neural-toolkit\"\ntype: software\nauthors:\n - family-names: \"Patel\"\n given-names: \"Anita\"\n - family-names: \"Johansson\"\n given-names: \"Erik\"\n - family-names: \"Costa\"\n given-names: \"Maria\"\nversion: \"1.5.0\"\ndate-released: \"2025-03-22\"\ndoi: \"10.5281/zenodo.1111111\"",
62+
"expected_output": "An APA-formatted citation string with authors in 'Family, Initial.' format, an ampersand before the last author, year in parentheses, title, version, [Computer software] designator, and DOI URL.",
63+
"files": [],
64+
"expectations": [
65+
"Output contains authors formatted as 'Patel, A.' and 'Johansson, E.' and 'Costa, M.' with initials (not full given names)",
66+
"Output uses '&' or 'and' before the last author (Costa, M.) per APA style",
67+
"Output contains '(2025)' as the year in parentheses",
68+
"Output contains '(Version 1.5.0)' after the title",
69+
"Output contains '[Computer software]' type designator",
70+
"Output contains the DOI as a URL: 'https://doi.org/10.5281/zenodo.1111111'"
71+
]
72+
},
73+
{
74+
"id": 6,
75+
"prompt": "Sync the version in my CITATION.cff from the package manifest. My CITATION.cff has version '1.0.0' and date-released '2024-06-01', but my package.json has version '1.3.0'. Use --dry-run to show what would change.",
76+
"expected_output": "A diff/comparison showing the version update from 1.0.0 to 1.3.0 and the date-released update to today's date, displayed in dry-run mode without writing.",
77+
"files": [],
78+
"expectations": [
79+
"Output shows the version changing from '1.0.0' to '1.3.0'",
80+
"Output shows the date-released being updated to today's date (or a current date in YYYY-MM-DD format)",
81+
"Output explicitly states this is a dry-run and changes have NOT been written to disk",
82+
"Output presents changes in a diff or comparison table format showing old vs new values"
83+
]
84+
},
85+
{
86+
"id": 7,
87+
"prompt": "Validate this CITATION.cff file and confirm it is correct:\n\ncff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\ntitle: \"good-project\"\ntype: software\nauthors:\n - family-names: \"Mueller\"\n given-names: \"Hans\"\n orcid: \"https://orcid.org/0000-0002-1234-5678\"\nversion: \"2.0.0\"\ndate-released: \"2025-01-15\"\nurl: \"https://github.com/example/good-project\"\nrepository-code: \"https://github.com/example/good-project\"\nlicense: \"MIT\"\nkeywords:\n - testing\n - validation",
88+
"expected_output": "A validation report with zero errors, confirming the file is valid CFF 1.2.0. May include informational notes about DOI not being set.",
89+
"files": [],
90+
"expectations": [
91+
"Report result is PASS (not FAIL) with 0 errors",
92+
"Report does not flag any required fields as missing",
93+
"Report correctly identifies that 1 author is listed",
94+
"Report notes that ORCID is present and valid for the author (format matches https://orcid.org/XXXX-XXXX-XXXX-XXXX)",
95+
"Report identifies license 'MIT' as a valid SPDX identifier"
96+
]
97+
}
98+
]
99+
}

‎skills/readme-enhance/evals/evals.json‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
{
2+
"skill_name": "readme-enhance",
3+
"evals": [
4+
{
5+
"id": 1,
6+
"prompt": "Enhance the README for this Node.js CLI tool project. The repo is at github.com/acme/taskrunner, it has a package.json with name 'taskrunner' version '2.1.0', a LICENSE file with MIT, a .github/workflows/ci.yml, and the primary language is TypeScript.",
7+
"expected_output": "README.md updated with shields.io badges (version, build, license, language, stars, downloads, PRs welcome), an SVG infographic saved to .github/readme-infographic.svg with 1280x640 viewBox, and a collapsible prompt details section in the README.",
8+
"files": [
9+
"README.md",
10+
"package.json",
11+
"LICENSE",
12+
".github/workflows/ci.yml"
13+
],
14+
"expectations": [
15+
"README.md contains at least one badge URL matching 'https://img.shields.io/' with 'style=for-the-badge' query parameter",
16+
"README.md contains an npm version badge with 'npm/v/taskrunner' in the URL",
17+
"README.md contains a license badge referencing MIT",
18+
"An SVG file is written to .github/readme-infographic.svg containing a root <svg> element with viewBox='0 0 1280 640'",
19+
"README.md contains a <details> block with <summary> referencing 'infographic generation prompt' or similar wording",
20+
"README.md contains an <img> tag referencing '.github/readme-infographic.svg' wrapped in a center-aligned <p> element"
21+
]
22+
},
23+
{
24+
"id": 2,
25+
"prompt": "Enhance the README for my Python library 'dataweave' hosted at github.com/jsmith/dataweave. It has a pyproject.toml with version 0.8.3, an Apache-2.0 LICENSE, GitHub Actions CI, and Codecov integration.",
26+
"expected_output": "README.md updated with Python-specific badges (PyPI version, build status, coverage, license, language), an SVG infographic using the Library/SDK balanced layout, and the collapsible prompt section.",
27+
"files": [
28+
"README.md",
29+
"pyproject.toml",
30+
"LICENSE",
31+
".github/workflows/ci.yml"
32+
],
33+
"expectations": [
34+
"README.md contains a PyPI version badge with 'pypi/v/dataweave' in the URL and 'logo=python' parameter",
35+
"README.md contains a Codecov coverage badge with 'codecov/c/github/jsmith/dataweave' in the URL",
36+
"README.md contains a license badge referencing Apache 2.0 or Apache_2.0",
37+
"The SVG infographic contains a linearGradient element for the background fill",
38+
"README.md badge lines appear after the first H1 heading and before the infographic image element",
39+
"The SVG infographic contains text elements with the project name 'dataweave'"
40+
]
41+
},
42+
{
43+
"id": 3,
44+
"prompt": "Enhance the README for github.com/acme/taskrunner with --badges-only mode. It has package.json (name: taskrunner, version: 3.0.0), MIT LICENSE, and GitHub Actions CI.",
45+
"expected_output": "README.md updated with badges only; no infographic SVG file is generated and no infographic image tag is inserted into the README.",
46+
"files": [
47+
"README.md",
48+
"package.json",
49+
"LICENSE",
50+
".github/workflows/ci.yml"
51+
],
52+
"expectations": [
53+
"README.md contains shields.io badge markdown with 'style=for-the-badge'",
54+
"No SVG file is created at .github/readme-infographic.svg",
55+
"README.md does not contain an <img> tag referencing 'readme-infographic'",
56+
"README.md contains a build status badge referencing 'github/actions/workflow/status'",
57+
"Badges are ordered with status/version badges appearing before engagement badges (stars, PRs welcome)"
58+
]
59+
},
60+
{
61+
"id": 4,
62+
"prompt": "Enhance the README for github.com/acme/taskrunner with --infographic-only mode. It has package.json (name: taskrunner, version: 1.5.0), MIT LICENSE, and TypeScript source files.",
63+
"expected_output": "An SVG infographic is generated and placed in the README, but existing badges are not modified or added.",
64+
"files": ["README.md", "package.json", "LICENSE"],
65+
"expectations": [
66+
"An SVG file is written to .github/readme-infographic.svg with viewBox='0 0 1280 640'",
67+
"README.md contains an <img> tag or <picture> element referencing the infographic SVG",
68+
"README.md contains a <details> block with the collapsible generation prompt section",
69+
"No new shields.io badge lines are added to README.md beyond any that already existed",
70+
"The SVG contains a footer section with an install command or project link text"
71+
]
72+
},
73+
{
74+
"id": 5,
75+
"prompt": "Enhance the README for a project at github.com/devuser/myutil that has no package.json, no Cargo.toml, no pyproject.toml, and no go.mod. It only has a README.md, some .sh scripts, and a LICENSE (MIT).",
76+
"expected_output": "The skill extracts the project name from the git remote URL, generates badges using GitHub-level information (license, stars, last commit), and creates an infographic using available context.",
77+
"files": ["README.md", "LICENSE"],
78+
"expectations": [
79+
"README.md contains a license badge for MIT even without a package manifest",
80+
"README.md contains GitHub engagement badges (stars or last-commit) using 'devuser/myutil' in the URL",
81+
"No registry-specific version badge (npm, PyPI, crates.io) is present since there is no package manifest",
82+
"An SVG infographic is still generated at .github/readme-infographic.svg with valid SVG markup",
83+
"The SVG infographic has a 1280x640 viewBox and contains the project name 'myutil' in a text element"
84+
]
85+
},
86+
{
87+
"id": 6,
88+
"prompt": "Enhance the README for github.com/acme/nightvision. It has package.json (name: nightvision, version: 1.0.0), MIT LICENSE, and a .claude/github-social.local.md config with dark_mode set to 'both'.",
89+
"expected_output": "README.md uses a <picture> element with <source> tags for dark and light mode infographic variants, and both SVG files are generated.",
90+
"files": [
91+
"README.md",
92+
"package.json",
93+
"LICENSE",
94+
".claude/github-social.local.md"
95+
],
96+
"expectations": [
97+
"README.md contains a <picture> element with a <source> tag having media='(prefers-color-scheme: dark)' and srcset referencing a dark variant SVG",
98+
"README.md contains a <source> tag with media='(prefers-color-scheme: light)' referencing the light variant SVG",
99+
"Two SVG files are generated: the default infographic and a dark mode variant (e.g., readme-infographic-dark.svg)",
100+
"The <picture> element contains a fallback <img> tag referencing the default infographic SVG"
101+
]
102+
},
103+
{
104+
"id": 7,
105+
"prompt": "Enhance the README for github.com/bigcorp/megaplatform. It has package.json (name: megaplatform, version: 5.2.1), MIT LICENSE, GitHub Actions CI, Codecov, a Discord community link in README, a FUNDING.yml, docs at docs.megaplatform.io, and TypeScript as primary language. Ensure badges follow proper ordering and stay within limits.",
106+
"expected_output": "README.md contains a comprehensive but bounded badge set (8-12 badges) ordered by importance: status first, then version, quality, language, metrics, engagement, community, and support last.",
107+
"files": [
108+
"README.md",
109+
"package.json",
110+
"LICENSE",
111+
".github/workflows/ci.yml",
112+
".github/FUNDING.yml"
113+
],
114+
"expectations": [
115+
"README.md contains between 8 and 12 shields.io badge lines (not exceeding 15)",
116+
"The build status badge appears before the npm version badge in the README source order",
117+
"The version badge appears before stars/downloads engagement badges in source order",
118+
"README.md contains a Discord badge with 'logo=discord' in the URL",
119+
"README.md contains a sponsor or funding-related badge since FUNDING.yml exists",
120+
"README.md contains a documentation badge referencing docs.megaplatform.io"
121+
]
122+
}
123+
]
124+
}

0 commit comments

Comments
 (0)