Skip to content

Commit afb6dad

Browse files
zealoushackerzealoushacker
authored
Merge pull request anthropics#191 from anthropics/zh/add-notebook-ci-validation
ci: add notebook CI/CD with validation and testing
2 parents 3e74f6c + 1653c3b commit afb6dad

21 files changed

+3403
-1
lines changed

‎.claude/commands/link-review.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
allowed-tools: Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)
3+
description: Review links in changed files for quality and security issues
4+
---
5+
6+
Review the links in the changed files and check for potential issues:
7+
8+
## Link Quality Checks
9+
1. **Broken Links**: Identify any links that might be broken or malformed
10+
2. **Outdated Links**: Check for links to deprecated resources or old documentation
11+
3. **Security**: Ensure no links to suspicious or potentially harmful sites
12+
4. **Best Practices**:
13+
- Links should use HTTPS where possible
14+
- Internal links should use relative paths
15+
- External links should be to stable, reputable sources
16+
17+
## Specific Checks for Anthropic Content
18+
- Links to Claude documentation should point to the latest versions
19+
- API documentation links should be current
20+
- Model documentation should reference current models, not deprecated ones
21+
- GitHub links should use the correct repository paths
22+
23+
## Report Format
24+
Provide a clear summary with:
25+
- ✅ Valid and well-formed links
26+
- ⚠️ Links that might need attention (e.g., HTTP instead of HTTPS)
27+
- ❌ Broken or problematic links that must be fixed
28+
29+
If all links look good, provide a brief confirmation.
30+
31+
**IMPORTANT: Post your review as a comment on the pull request using the command: `gh pr comment ${{ github.event.pull_request.number }} --body "your review content"`**

‎.claude/commands/model-check.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
allowed-tools: Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)
3+
description: Validate Claude model usage against current public models
4+
---
5+
6+
Review the changed files for Claude model usage.
7+
8+
First, fetch the current list of allowed models from:
9+
https://docs.anthropic.com/en/docs/about-claude/models/overview.md
10+
11+
Then check:
12+
1. All model references are from the current public models list
13+
2. Flag any deprecated models (older Sonnet 3.5, Opus 3 versions)
14+
3. Flag any internal/non-public model names
15+
4. Suggest using aliases ending in -latest for better maintainability
16+
17+
Provide clear, actionable feedback on any issues found.
18+
19+
**IMPORTANT: Post your findings as a comment on the pull request using the command: `gh pr comment ${{ github.event.pull_request.number }} --body "your findings"`**

‎.claude/commands/notebook-review.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
allowed-tools: Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)
3+
description: Comprehensive review of Jupyter notebooks and Python scripts
4+
---
5+
6+
Review the changes to Jupyter notebooks and Python scripts in this PR. Please check for:
7+
8+
## Model Usage
9+
Verify all Claude model references against the current list at:
10+
https://docs.anthropic.com/en/docs/about-claude/models/overview.md
11+
- Flag any deprecated models (older Sonnet 3.5, Opus 3 versions)
12+
- Flag any internal/non-public model names
13+
- Suggest current alternatives when issues found
14+
- Recommend aliases ending in -latest for stability
15+
16+
## Code Quality
17+
- Python code follows PEP 8 conventions
18+
- Proper error handling
19+
- Clear variable names and documentation
20+
- No hardcoded API keys (use os.getenv("ANTHROPIC_API_KEY"))
21+
22+
## Notebook Structure
23+
- Clear introduction explaining what the notebook demonstrates and why it's useful
24+
- Configuration instructions (how to set up API keys, install dependencies, etc.)
25+
- Connecting explanations between cells that help users understand the flow
26+
- Clear markdown explanations between code cells
27+
- Logical flow from simple to complex
28+
- Outputs preserved for educational value
29+
- Dependencies properly imported
30+
31+
## Security
32+
- Check for any hardcoded API keys or secrets (not just Anthropic keys)
33+
- Ensure all sensitive credentials use environment variables (os.environ, getenv, etc.)
34+
- Flag any potential secret patterns (tokens, passwords, private keys)
35+
- Note: Educational examples showing "what not to do" are acceptable if clearly marked
36+
- Safe handling of user inputs
37+
- Appropriate use of environment variables
38+
39+
Provide a clear summary with:
40+
- ✅ What looks good
41+
- ⚠️ Suggestions for improvement
42+
- ❌ Critical issues that must be fixed
43+
44+
**IMPORTANT: Post your review as a comment on the pull request using the command: `gh pr comment ${{ github.event.pull_request.number }} --body "your review"`**

‎.env.example‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Anthropic API Configuration
2+
# Copy this file to .env and add your API key
3+
# Get your API key at: https://console.anthropic.com/settings/keys
4+
5+
ANTHROPIC_API_KEY=sk-ant-api03-...
6+
7+
# Optional: Default model for testing (recommended for cost savings)
8+
CLAUDE_MODEL=claude-3-5-haiku-latest
9+
10+
# Optional: Test mode settings
11+
TEST_MODE=true
12+
MAX_TOKENS=10
13+
14+
# Optional: Verbose output for debugging
15+
DEBUG=false

‎.github/workflows/claude-link-review.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Claude Link Review
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize]
6+
paths:
7+
- '**.md'
8+
- '**.mdx'
9+
- '**.ipynb'
10+
- 'README.md'
11+
12+
jobs:
13+
link-review:
14+
runs-on: ubuntu-latest
15+
permissions:
16+
contents: read
17+
pull-requests: write
18+
id-token: write
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v4
22+
with:
23+
fetch-depth: 0
24+
25+
- name: Run Claude Link Review
26+
uses: anthropics/claude-code-action@v1
27+
with:
28+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
29+
github_token: ${{ secrets.GITHUB_TOKEN }}
30+
prompt: "/link-review"

‎.github/workflows/claude-model-check.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: Claude Model Check
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize]
6+
paths:
7+
- 'skills/**/*.ipynb'
8+
- '**.py'
9+
- '**.md'
10+
11+
permissions:
12+
contents: read
13+
pull-requests: write
14+
15+
jobs:
16+
model-check:
17+
runs-on: ubuntu-latest
18+
19+
steps:
20+
- uses: actions/checkout@v4
21+
with:
22+
fetch-depth: 0
23+
24+
- name: Claude Model Validation
25+
uses: anthropics/claude-code-action@v1
26+
with:
27+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
28+
github_token: ${{ secrets.GITHUB_TOKEN }}
29+
prompt: "/model-check"

‎.github/workflows/claude-notebook-review.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Claude Notebook Review
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize]
6+
paths:
7+
- 'skills/**/*.ipynb'
8+
- 'pyproject.toml'
9+
- 'uv.lock'
10+
- 'scripts/**/*.py'
11+
12+
jobs:
13+
notebook-review:
14+
runs-on: ubuntu-latest
15+
permissions:
16+
contents: read
17+
pull-requests: write
18+
id-token: write
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v4
22+
with:
23+
fetch-depth: 0
24+
25+
- name: Run Claude Notebook Review
26+
uses: anthropics/claude-code-action@v1
27+
with:
28+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
29+
github_token: ${{ secrets.GITHUB_TOKEN }}
30+
prompt: "/notebook-review"

‎.github/workflows/links.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: Link Check
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize]
6+
schedule:
7+
- cron: "0 0 * * SUN"
8+
workflow_dispatch:
9+
10+
permissions:
11+
contents: read
12+
pull-requests: write
13+
14+
jobs:
15+
check-links:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Setup Python
21+
uses: actions/setup-python@v5
22+
with:
23+
python-version: '3.11'
24+
25+
- name: Convert notebooks for link extraction
26+
run: |
27+
pip install jupyter nbconvert
28+
mkdir -p temp_md
29+
30+
for nb in skills/**/*.ipynb; do
31+
echo "Converting: $nb"
32+
jupyter nbconvert --to markdown "$nb" \
33+
--output-dir=temp_md \
34+
--ExtractOutputPreprocessor.enabled=False
35+
done
36+
37+
- name: Check Links with Lychee
38+
id: lychee
39+
uses: lycheeverse/lychee-action@v2
40+
with:
41+
args: |
42+
--config lychee.toml
43+
--format markdown
44+
--no-progress
45+
skills/**/*.md
46+
temp_md/*.md
47+
README.md
48+
output: lychee-report.md
49+
fail: false
50+
51+
- name: Comment PR with results
52+
if: github.event_name == 'pull_request' && steps.lychee.outputs.exit_code != 0
53+
uses: marocchino/sticky-pull-request-comment@v2
54+
with:
55+
header: link-check
56+
path: lychee-report.md
57+
58+
- name: Upload link check results
59+
if: always()
60+
uses: actions/upload-artifact@v4
61+
with:
62+
name: link-check-results
63+
path: |
64+
lychee-report.md
65+
.lycheecache

‎.github/workflows/notebook-quality.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: Notebook Quality Check
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'skills/**/*.ipynb'
7+
- 'pyproject.toml'
8+
- 'uv.lock'
9+
push:
10+
branches: [main]
11+
paths:
12+
- 'skills/**/*.ipynb'
13+
14+
permissions:
15+
contents: read
16+
pull-requests: write
17+
18+
jobs:
19+
validate-notebooks:
20+
runs-on: ubuntu-latest
21+
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- name: Install uv
26+
uses: astral-sh/setup-uv@v4
27+
with:
28+
enable-cache: true
29+
cache-dependency-glob: "uv.lock"
30+
31+
- name: Set up Python 3.11
32+
run: uv python install 3.11
33+
34+
- name: Install dependencies
35+
run: |
36+
uv sync --frozen --all-extras
37+
38+
- name: Lint notebooks with Ruff
39+
run: |
40+
uv run ruff check skills/**/*.ipynb --show-fixes || true
41+
uv run ruff format skills/**/*.ipynb --check || true
42+
43+
- name: Validate notebook structure
44+
run: |
45+
uv run python scripts/validate_notebooks.py
46+
47+
# Only run API tests on main branch or for maintainers (costs money)
48+
- name: Execute notebooks (API Testing)
49+
if: |
50+
github.event_name == 'push' ||
51+
github.event.pull_request.author_association == 'MEMBER' ||
52+
github.event.pull_request.author_association == 'OWNER'
53+
env:
54+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
55+
run: |
56+
mkdir -p test_outputs
57+
for notebook in skills/*/guide.ipynb; do
58+
echo "📓 Testing: $notebook"
59+
output_name=$(basename $(dirname "$notebook"))
60+
# Use nbconvert to execute notebooks and save outputs
61+
uv run jupyter nbconvert --to notebook \
62+
--execute "$notebook" \
63+
--ExecutePreprocessor.kernel_name=python3 \
64+
--ExecutePreprocessor.timeout=120 \
65+
--output "test_outputs/${output_name}_executed.ipynb" \
66+
--output-dir=. \
67+
|| echo "⚠️ Failed: $notebook"
68+
done
69+
70+
# Mock testing for external contributors
71+
- name: Execute notebooks (Mock Testing)
72+
if: |
73+
github.event_name == 'pull_request' &&
74+
github.event.pull_request.author_association != 'MEMBER' &&
75+
github.event.pull_request.author_association != 'OWNER'
76+
run: |
77+
echo "🔒 Running in mock mode for external contributor"
78+
79+
for notebook in skills/*/guide.ipynb; do
80+
echo "📓 Validating structure: $notebook"
81+
uv run python -m nbformat.validator "$notebook"
82+
done
83+
84+
- name: Upload test outputs
85+
if: always()
86+
uses: actions/upload-artifact@v4
87+
with:
88+
name: notebook-test-outputs
89+
path: test_outputs/
90+
retention-days: 7

‎.gitignore‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,4 +137,16 @@ dmypy.json
137137
/examples/data/transactions*
138138
*.DS_Store
139139
tmp_*
140-
examples/fine-tuned_qa/local_cache/*
140+
examples/fine-tuned_qa/local_cache/*
141+
142+
# CI/CD and testing
143+
.model_cache.json
144+
test_outputs/
145+
.ruff_cache/
146+
lychee-report.md
147+
.lycheecache
148+
149+
# Notebook validation
150+
.notebook_validation_state.json
151+
.notebook_validation_checkpoint.json
152+
validation_report_*.md

0 commit comments

Comments
 (0)