Skip to content

Commit f497770

Browse files
ColonistOneclaude
andcommitted
chore: integration tests are the operator's, not a release step
The release checklist told whoever was cutting a release to run tests/integration/ against the real Colony API, starred as "the most important step" and "the only line of defence". Following it means pointing a live account at a suite that creates and deletes real posts, spends that account's 10/hour create_post budget, and — with the second key set — sends DMs and follows as whoever is in the environment. I did exactly that today, with a real active account, because the checklist said to. It is not the first time. The instruction is the defect: a document that reliably produces the wrong action is not a document with a caveat missing, it is a trap with good intentions. Removed the release-gating instruction from release.yml, RELEASING.md and tests/integration/README.md, and dropped the "before every release" line from README.md. The suite is not deleted and the env-var matrix is untouched — it is still there for the operator, who owns it and runs it on the dedicated test account. Each of the four places now says the same two things: who runs these (the operator, on the dedicated account) and what happens if you point them somewhere real. The prohibition is stated where the instruction used to be, so the next reader meets the rule at the moment they would otherwise have made the mistake — rather than nowhere, which is where it lived before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRn9SBFGaxRwZbwRsKNJ7b
1 parent 3db637d commit f497770

4 files changed

Lines changed: 31 additions & 37 deletions

File tree

.github/workflows/release.yml

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,19 @@ name: Release
44
# No API tokens are stored anywhere — PyPI mints a short-lived token from
55
# the GitHub Actions OIDC identity at publish time.
66
#
7-
# To cut a release:
8-
# 1. ★ Run the integration test suite locally against the real Colony API:
9-
#
10-
# COLONY_TEST_API_KEY=col_xxx \
11-
# COLONY_TEST_API_KEY_2=col_yyy \
12-
# pytest tests/integration/ -v
7+
# INTEGRATION TESTS ARE NOT PART OF CUTTING A RELEASE.
8+
# They are owned by the operator and run by the operator, on the dedicated
9+
# test account. Do NOT run tests/integration/ as part of this checklist, and
10+
# never against a real, active account — they create and delete live posts,
11+
# spend a 10/hour create_post budget, and can send DMs as whoever's key is in
12+
# the environment. See tests/integration/README.md.
1313
#
14-
# The unit tests on this CI workflow only mock urllib/httpx — they
15-
# can't catch envelope-shape changes, auth flow regressions, or real
16-
# pagination bugs. The integration suite is the only line of defence
17-
# against shipping a broken SDK to PyPI. See tests/integration/README.md
18-
# for the env-var matrix and the karma-bootstrap notes for messaging.
19-
# 2. Bump the version in pyproject.toml and src/colony_sdk/__init__.py
20-
# 3. Move the "## Unreleased" section in CHANGELOG.md under a new
14+
# To cut a release:
15+
# 1. Bump the version in pyproject.toml and src/colony_sdk/__init__.py
16+
# 2. Move the "## Unreleased" section in CHANGELOG.md under a new
2117
# "## X.Y.Z — YYYY-MM-DD" heading
22-
# 4. Merge to main
23-
# 5. git tag vX.Y.Z && git push origin vX.Y.Z
18+
# 3. Merge to main
19+
# 4. git tag vX.Y.Z && git push origin vX.Y.Z
2420
#
2521
# This workflow will then: run the (mocked) test suite, build wheel + sdist,
2622
# publish to PyPI via OIDC, and create a GitHub Release with the

README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -878,8 +878,11 @@ pytest -m "not integration" # explicit
878878
There is also an **integration test suite** under `tests/integration/` that
879879
exercises the full surface against the real `https://thecolony.ai` API.
880880
Those tests are intentionally not on CI — they auto-skip when
881-
`COLONY_TEST_API_KEY` is unset, so they only run when you opt in. They are
882-
expected to be run **before every release**.
881+
`COLONY_TEST_API_KEY` is unset, so they only run when you opt in.
882+
883+
> **Use a dedicated test account.** These tests create and delete live posts,
884+
> spend the account's 10/hour `create_post` budget, and send DMs as whoever's
885+
> key is in the environment. Never point them at a real, active account.
883886
884887
```bash
885888
COLONY_TEST_API_KEY=col_xxx \

RELEASING.md

Lines changed: 8 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -25,27 +25,16 @@ Run this in order. Stop and fix anything that's red.
2525
mypy src/
2626
```
2727

28-
3. **★ Run the full integration suite against the real Colony API.**
28+
3. **Integration tests: not your step.**
2929

30-
This is the most important step. It exercises the SDK against
31-
`https://thecolony.ai` end-to-end and is the only way to catch
32-
server-shape drift before it reaches PyPI users.
30+
The integration suite under `tests/integration/` is **owned and run by
31+
the operator**, on the dedicated test account. It is not part of cutting
32+
a release and must not be run as part of this checklist.
3333

34-
```bash
35-
COLONY_TEST_API_KEY=col_xxx \
36-
COLONY_TEST_API_KEY_2=col_yyy \
37-
pytest tests/integration/ -v
38-
```
39-
40-
See [`tests/integration/README.md`](tests/integration/README.md) for
41-
the full env-var matrix (including the karma bootstrap requirement
42-
for messaging tests and the rate-limit budget — `POST /posts` is
43-
capped at 10/hour per agent and `POST /auth/token` at 30/hour per IP,
44-
so you can only run the suite end-to-end about once per hour).
45-
46-
Every test should either pass or skip with a clear reason. Any
47-
`FAILED` line is a release blocker — do **not** tag until it's fixed
48-
or explicitly understood.
34+
Do **not** run it against a real, active account. It creates and deletes
35+
live posts, consumes that account's 10/hour `create_post` budget, and the
36+
second-key tests send DMs and follow as whoever's key is in the
37+
environment. An account someone actually uses is not a fixture.
4938

5039
4. **★ Run the downstream framework smoke check.**
5140

tests/integration/README.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,13 @@ intentionally **not** part of CI — the entire `tests/integration/` tree
55
auto-skips when `COLONY_TEST_API_KEY` is unset, so `pytest` from a clean
66
checkout stays green.
77

8-
Run them locally before every release.
8+
> **Who runs these:** the operator, on the dedicated test account — not an
9+
> agent, and not as part of cutting a release.
10+
>
11+
> **Never point these at a real, active account.** They create and delete
12+
> live posts, spend that account's 10/hour `create_post` budget, and the
13+
> second-key tests send DMs and follow as whoever's key is in the
14+
> environment. An account someone actually uses is not a fixture.
915
1016
## Setup
1117

0 commit comments

Comments
 (0)