Add incident response plan (aio-libs#12914)

Author: Dreamsorcerer

CHANGES/12914.contrib.rst

@@ -0,0 +1,3 @@
+Added admin documentation on incident response and on running reproducer code
+safely, covering security vulnerability handling and supply-chain, account, and
+CI/infrastructure compromise -- by :user:`Dreamsorcerer`.

docs/contributing-admins.rst

@@ -8,6 +8,19 @@ For regular contributors, return to :doc:`contributing`.
 
 .. contents::
    :local:
+   :depth: 1
+
+.. highlight:: none
+
+Running reproducer code
+-----------------------
+
+.. warning::
+
+   When evaluating a bug report or vulnerability report, treat reproducer code as
+   untrusted. If you don't understand what it does or are unfamiliar with a library
+   it imports **do not run it** (and ask the reporter to provide a simpler reproducer).
+   We also recommend that any reproducers you do run are executed in a container.
 
 Creating a new release
 ----------------------
@@ -60,3 +73,287 @@ If doing a minor release:
 #. Update both ``target-branch`` backports for Dependabot to reference the new branch name in ``.github/dependabot.yml``.
 #. Delete the older backport label (e.g. backport-3.8): https://github.com/aio-libs/aiohttp/labels
 #. Add a new backport label (e.g. backport-3.10).
+
+Incident response
+-----------------
+
+This section covers responding to a reported security vulnerability and to
+three classes of compromise -- of the supply chain, of a maintainer account,
+and of the project infrastructure. It picks up *after* a report or a problem
+is in hand.
+
+.. note::
+
+   Vulnerability reports arrive through the organization's ``SECURITY.md``
+   -- GitHub private vulnerability reporting, or email to the security coordinators.
+   Never triage a suspected vulnerability in a public issue or pull request.
+
+The security coordinator who first triages a report is its *incident lead*,
+and may explicitly hand off to another coordinator. The lead owns the GitHub
+Security Advisory (GHSA) draft, fix coordination, the release, and notification.
+
+Severity tiers
+~~~~~~~~~~~~~~
+
+aiohttp is volunteer-maintained, so there is no response-time commitment. The
+tier guides prioritization, the release decision, and how widely the fix is
+announced. All security fixes -- regardless of tier -- land on ``master`` and
+are backported to the currently supported ``x.y`` branch (and the next
+``x.y`` branch, if one is in development).
+
+High
+   A concrete remote impact an attacker can actually achieve against a
+   default deployment. Examples: a single request that stops the server from
+   handling further requests (a server-wide denial of service), reading files
+   on the server outside the project root, remote code execution,
+   authentication bypass. Cut a dedicated security release once the fix lands.
+
+Medium
+   Bounded impact, or impact that needs a non-default option, an unusual
+   configuration, or a local position. Examples: request smuggling demonstrated
+   to cause a real issue against a common proxy, or a DoS that use significant server
+   resources with a low effort sustained attack.
+   Cut a dedicated security release.
+
+Low
+   Limited or hard-to-exploit impact, or impact confined to debug or
+   non-default paths. Examples: minor version or error-message disclosure,
+   parser leniency with no demonstrated security impact, or request smuggling
+   that has not been demonstrated to cause a real issue against a common
+   proxy. May ride the next routine release rather than cutting a dedicated
+   security release.
+
+Severity is rated by demonstrated attacker impact. A denial of service that
+disables further request handling for the whole server from a single request
+leans High; one that requires sustained low-effort traffic to consume
+significant server resources leans Medium. Request smuggling and parser bugs
+are rated by what an attacker can actually achieve against a common proxy or
+in a realistic deployment, not by the shape of the bug.
+
+Responding to a reported vulnerability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. **Acknowledge and confirm.** Reply to the reporter in the GitHub private
+   report or the email thread. Reproduce the issue
+   (`but don't run untrusted code <Running reproducer code_>`_) against
+   ``master`` and the maintained ``x.y`` branches. If it is not reproducible
+   or is out of scope, record the decision and close the report.
+#. **Open the GHSA draft.** For an email report, if the reporter can't use Github
+   for some reason, then open a new advisory to track the issue.
+#. **Develop the fix privately** in the GHSA's temporary private fork ("Start a
+   temporary private fork" in the draft) -- never on a public branch or in a
+   public pull request. Include a regression test unless it would provide too
+   much information for an attacker to replicate.
+#. **Title the advisory** with the vulnerability class and affected component.
+#. **Write the description** using the GHSA description template::
+
+      ### Summary
+
+      <one or two sentences describing the vulnerability>
+
+      ### Impact
+
+      <who is affected, prerequisites, and what an attacker achieves>
+
+      ### Workaround
+
+      <if relevant, any workarounds for users unable to upgrade>
+
+      ---
+
+      Patch: <link to commit fixing the issue on the stable x.y branch>
+
+   An important detail that Github will use to decide the CVE severity is exploit
+   maturity. Try to clarify the likelihood of a vulnerability being attacked
+   (e.g. if exploit code is available, or active attacks in-the-wild already exist).
+#. **Set the affected and fixed versions**.
+#. **Assign a severity tier** (see `Severity tiers`_).
+#. **Credit the reporter** (if report came by email) in the GHSA Credits field.
+#. **Credit the developer** using "Remediation developer" if different from Reporter.
+#. **Request a CVE** by clicking the button in the advisory.
+#. **Get the fix reviewed** in the private fork by at least one other maintainer.
+#. **Consider early notify** for high severity issues
+   (see `Notifying about a disclosed vulnerability`_).
+#. **Coordinate a release.**
+
+   #. **Agree the timing.** For an embargoed high-severity incident, set the
+      release date to align with the lift of any private-list embargo.
+   #. **Merge all the private forks** and create and merge the backports for
+      each.
+   #. **Create the release.** Follow `Creating a new release`_.
+#. **Update patch link** in GHSA description.
+#. **Publish the GHSA** usually around 1 day after release.
+#. **Notify** according to the severity tier
+   (see `Notifying about a disclosed vulnerability`_).
+#. **Run the post-incident steps** (see `Post-incident`_).
+
+Notifying about a disclosed vulnerability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Notification is cumulative -- each tier adds to the ones below it. Mind the
+timing: the ``linux-distros`` pre-notification happens under embargo, *before*
+the release, while everything else happens at or after the release.
+
+Baseline (all severities)
+   The published GHSA (which feeds the CVE and the GitHub Advisory Database).
+
+Higher (Medium-High)
+   For most high severity issues, additionally post to the public ``oss-security``
+   mailing list once the advisory is public.
+
+Highest (High-Critical)
+   For the most serious issues, additionally *before* the public release, send
+   an embargoed pre-notification to the private distribution-security lists -- primarily
+   ``linux-distros`` -- and attempt to notify affected downstreams. Use the
+   `vuln_search.py dependents-enumeration script
+   <https://github.com/aio-libs/.github/blob/master/tools/vuln_search.py>`_
+   to search aiohttp's GitHub dependents; it needs a GitHub token and may need to be
+   run over a few days, so start it early.
+
+Keep the embargo as short as practical. Typically an embargo of 1 or 2 days is expected.
+
+Template for the embargoed ``linux-distros`` pre-notification::
+
+    To: linux-distros@vs.openwall.org
+    Subject: aiohttp: <one-line summary of the vulnerability>
+
+    We are coordinating disclosure of a security vulnerability in aiohttp
+    (https://github.com/aio-libs/aiohttp), the asyncio HTTP client/server
+    library.
+
+    Summary
+      <2-3 sentences: the flaw and its impact>
+
+    Affected versions
+      <e.g. aiohttp < 3.12.15 (all 3.11.x and 3.12.x)>
+
+    CVE
+      <CVE-XXXX-XXXXX, or "requested via GitHub Security Advisory, ID pending">
+
+    Fix
+      <link to the patch, or attach the diff>
+
+    Proposed public disclosure date
+      <YYYY-MM-DD> -- as short as practical. On that date we will publish the
+      GitHub Security Advisory, ship a patched PyPI release, and post to
+      oss-security.
+
+    This issue is not yet public; please observe the embargo until that date.
+
+    Contact: <incident lead name and security email>
+
+Template for the public ``oss-security`` disclosure::
+
+    To: oss-security@lists.openwall.com
+    Subject: CVE-XXXX-XXXXX: aiohttp <one-line summary>
+
+    A security vulnerability has been fixed in aiohttp, the asyncio HTTP
+    client/server library (https://github.com/aio-libs/aiohttp).
+
+    CVE: CVE-XXXX-XXXXX
+    Advisory: <GHSA URL>
+
+    Affected versions: <...>
+    Fixed versions:    <...>
+
+    Description
+      <what the flaw is>
+
+    Impact
+      <what an attacker can achieve, and any prerequisites>
+
+    Mitigation
+      <workaround, or "upgrade to <version>">
+
+Post-incident
+~~~~~~~~~~~~~
+
+#. **Update** ``THREAT_MODEL.md``. Have an AI coding agent fetch the new advisories
+   and update the document. Review the changes.
+#. **File follow-up hardening** as ordinary public issues or pull requests.
+
+Supply-chain or release compromise
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A malicious or tampered release on PyPI, or a compromised publish pipeline.
+
+#. **Pull the bad artifact.** On PyPI, *yank* the affected release -- this
+   hides it from new resolves while keeping existing pins working. *Delete* it
+   only if it is actively malicious and removal is the safer choice. This needs
+   a PyPI project owner or maintainer account.
+#. **Verify the legitimate artifacts with Sigstore.** Every genuine release is
+   signed by the release pipeline, with the ``.sigstore`` bundles attached to
+   the GitHub Release. Verify the ``sdist`` and wheels; a divergence from what is on
+   PyPI localizes the tampering.
+#. **Lock the publish path.** Publishing uses a PyPI OIDC trusted publisher, so
+   there is no long-lived token to rotate. Instead, on PyPI temporarily remove
+   the trusted-publisher binding, and on GitHub restrict or pause the ``pypi``
+   deployment environment used by the release job.
+#. **Audit the release inputs.** Review recent changes to the release workflow,
+   its tag trigger, and the third-party action versions it uses.
+#. **Audit the committed Cython sources.** The generated ``.c`` files ship in
+   the ``sdist`` and are not checked against their ``.pyx`` sources in CI (see
+   ``THREAT_MODEL.md`` section 5.19). Regenerate them with ``make cythonize``
+   and ``git diff`` against the released revision. Re-verify the
+   ``vendor/llhttp`` pin and ``package-lock.json``.
+#. **Reissue a clean release** via `Creating a new release`_ once the cause is
+   fixed. Never re-publish over a yanked version number.
+#. **Notify** per severity -- a malicious published release is High.
+
+Maintainer account compromise
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A GitHub or PyPI account takeover, or leaked publishing or signing material.
+
+#. **Suspend the account.** Another GitHub organization owner can suspend it,
+   revoke its repository access, and start the audit-log review. If the
+   compromised account is the sole organization owner, or no other owner is
+   reachable, contact GitHub Support at https://support.github.com/contact
+   for emergency account lock down and organization recovery.
+#. **Kill sessions and credentials** on the affected account. On GitHub: change
+   the password, sign out all sessions, revoke every personal access token,
+   OAuth app, and SSH or GPG key, then re-enroll two-factor authentication.
+   On PyPI: change the password, re-enroll two-factor authentication,
+   and revoke API tokens.
+#. **Audit PyPI project ownership.** OIDC publishing means there is usually no
+   long-lived PyPI token to leak, but a compromised project owner can add a
+   trusted publisher or upload directly. Review the project collaborators and
+   trusted-publisher bindings, and remove anything unrecognized.
+#. **Audit what the account could have done** from the GitHub audit log:
+   pushes, tag creation, branch-protection or settings changes, new secrets or
+   deploy keys, new collaborators, and workflow edits.
+#. **Revert and re-verify.** Force-revert any unauthorized commits or tags, and
+   re-verify recent releases against their ``Sigstore`` bundles. If a malicious
+   release shipped, escalate to `Supply-chain or release compromise`_.
+#. **Handle signing material.** ``Sigstore`` signing is key-less, so there is no
+   static signing key to rotate. Any separate GPG key used for signed tags must
+   be treated as compromised, then revoked and rotated.
+
+CI or infrastructure compromise
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A compromise of GitHub Actions, repository settings, or branch protections.
+
+#. **Freeze the release path.** Pause the ``pypi`` deployment environment and
+   stop tag-triggered deploys until the compromise is scoped.
+#. **Review branch protection and required reviews** on ``master`` and the
+   ``x.y`` branches. These are enforced on GitHub and are not visible in the
+   repository; confirm nothing was relaxed and restore the known-good
+   configuration.
+#. **Audit Actions secrets and environments** -- the repository and
+   organization secrets, the ``pypi`` environment's protection rules and
+   reviewers, and any new deploy keys. Confirm the workflow keeps empty
+   top-level permissions and that ``id-token: write`` is scoped to the deploy
+   job only.
+#. **Diff recently changed workflows** with ``git log -- .github/workflows/``,
+   especially the release workflow, the auto-merge workflow (which runs in the
+   privileged ``pull_request_target`` context), and the CodeQL workflow. Look
+   for added steps, changed action references, or new triggers.
+#. **Check the committed Cython sources** as in `Supply-chain or release
+   compromise`_ -- regenerate them and ``git diff``.
+#. **Re-pin and rebuild.** If a third-party action was compromised, pin the
+   affected actions by full commit hash, re-run CI from a clean known-good
+   revision, and re-verify with ``Sigstore`` any release made during the suspected
+   window.
+#. **Escalate** to `Supply-chain or release compromise`_ if a release shipped
+   through the compromised pipeline.

docs/spelling_wordlist.txt

@@ -36,6 +36,7 @@ backoff
 backend
 backends
 backport
+backported
 Backport
 Backporting
 backports
@@ -321,6 +322,7 @@ seekable
 sendfile
 serializable
 serializer
+severities
 shourtcuts
 skipuntil
 Skyscanner
@@ -362,6 +364,7 @@ toolbar
 toplevel
 towncrier
 tp
+triages
 tuples
 UI
 un