GitHub App: post comments on PRs (#12230)

- The FileTreeDiff dataclass was changed to a normal class, to make it
easier to interact with its contents in a dynamic way.
- Since a new FileTreeDiffFile class was introduced, the old one was
renamed to FileTreeDiffManifestFile, since it's a class only used in the
manifest dataclass.
- The comment is posted after the manifest from the PR is created, since
it depends on having the manifest ready to get the diff.
- The application creates one comment per-project linked to the remote
repository. First I tried including the information of several projects
in one single comment, but I think it's better for each project to have
their own comment (so users get the notification for when each build is
done).
- There is a new option in the project model, so users can decide if
this feature should be enabled. This option is hidden for projects not
linked to a GH app. I tried using the pre-validation form thing, but it
isn't per-field, and it blocks the whole form from saving.
- Try/catch around the indexers, so if one fails, we can still get the
build overview posted.
- For the comment, I took some inspiration from
https://docs.codecov.com/docs/pull-request-comments and from my initial
ideas around the UI for FTD. My vision is to keep the comment minimal,
but still have all the information available if needed. This means that
all metadata is either hidden or one-liner, while the list of top files
changed is a list. The list of all files changed is included in a table,
so we can add more information if needed (I'm envisioning putting here a
redirect suggestion, for example), and also should be search friendly,
so users can easily search all files that are marked as
changed/added/deleted.
- Should the comment also respect the ignore patterns from the addon
config? It's also kind of confusing that we have these settings in two
places... in addons and in the PR form.
- The list of the top 5 files changed is literally the 5 files from the
diff (sorted). I was also playing around with maybe taking one file from
each category (added/modifed/deleted), but will see how useful is just
having the top first.

You can see a live example at
stsewd/rtd-test-builds#1 (comment).

![Screenshot 2025-06-24 at 20-08-57 Update index rst by stsewd · Pull
Request #1 ·
stsewd_rtd-test-builds](https://github.com/user-attachments/assets/fe68a161-128b-4521-880d-de6b58e69365)


Closes #11780

Author: stsewd

docs/user/pull-requests.rst

@@ -24,6 +24,14 @@ Build status report
 
        GitHub build status reporting
 
+Build overview with changed files
+    We create a comment on the pull request including a link to the preview of the documentation,
+    and a list of the :doc:`files that changed </visual-diff>` between the current pull request and the latest version of the project's documentation.
+
+    .. note::
+
+       This feature is only available for projects connected to a :ref:`reference/git-integration:GitHub App`.
+
 Pull request notifications
     A pull request notifications is shown at the top of preview pages,
     which let readers know they aren't viewing an active version of the project.

readthedocs/builds/reporting.py

@@ -0,0 +1,41 @@
+from django.conf import settings
+from django.template.loader import render_to_string
+
+from readthedocs.builds.models import Build
+from readthedocs.filetreediff import get_diff
+
+
+def get_build_overview(build: Build) -> str | None:
+    """
+    Generate a build overview for the given build.
+
+    The overview includes a diff of the files changed between the current
+    build and the base version of the project (latest by default).
+
+    The returned string is rendered using a Markdown template,
+    which can be included in a comment on a pull request.
+    """
+    project = build.project
+    base_version = project.addons.options_base_version or project.get_latest_version()
+    if not base_version:
+        return None
+
+    diff = get_diff(
+        current_version=build.version,
+        base_version=base_version,
+    )
+    if not diff:
+        return None
+
+    return render_to_string(
+        "core/build-overview.md",
+        {
+            "PRODUCTION_DOMAIN": settings.PRODUCTION_DOMAIN,
+            "project": project,
+            "current_version": diff.current_version,
+            "current_version_build": diff.current_version_build,
+            "base_version": diff.base_version,
+            "base_version_build": diff.base_version_build,
+            "diff": diff,
+        },
+    )

readthedocs/builds/tasks.py

@@ -27,6 +27,7 @@
 from readthedocs.builds.constants import TAG
 from readthedocs.builds.models import Build
 from readthedocs.builds.models import Version
+from readthedocs.builds.reporting import get_build_overview
 from readthedocs.builds.utils import memcache_lock
 from readthedocs.core.utils import send_email
 from readthedocs.core.utils import trigger_build
@@ -428,6 +429,48 @@ def send_build_status(build_pk, commit, status):
     return False
 
 
+@app.task(max_retries=3, default_retry_delay=60, queue="web")
+def post_build_overview(build_pk):
+    build = Build.objects.filter(pk=build_pk).first()
+    if not build:
+        return
+
+    version = build.version
+    log.bind(
+        build_id=build.pk,
+        project_slug=build.project.slug,
+        version_slug=version.slug,
+    )
+
+    if not version.is_external:
+        log.debug("Build is not for an external version, skipping build overview.")
+        return
+
+    service_class = build.project.get_git_service_class()
+    if not service_class:
+        log.debug("Project isn't connected to a Git service, skipping build overview.")
+        return
+
+    if not service_class.supports_commenting:
+        log.debug("Git service doesn't support creating comments.")
+        return
+
+    comment = get_build_overview(build)
+    if not comment:
+        log.debug("No build overview available, skipping posting comment.")
+        return
+
+    for service in service_class.for_project(build.project):
+        service.post_comment(
+            build=build,
+            comment=comment,
+        )
+        log.debug("PR comment posted successfully.")
+        return
+
+    log.debug("No service available, no build overview posted.")
+
+
 @app.task(queue="web")
 def send_build_notifications(version_pk, build_pk, event):
     version = Version.objects.get_object_or_log(pk=version_pk)

readthedocs/builds/tests/test_tasks.py

@@ -1,22 +1,35 @@
 from datetime import datetime, timedelta
+from textwrap import dedent
 from unittest import mock
 
+from django.contrib.auth.models import User
 from django.test import TestCase, override_settings
 from django.utils import timezone
 from django_dynamic_fixture import get
 
 from readthedocs.builds.constants import (
     BRANCH,
+    BUILD_STATE_FINISHED,
     EXTERNAL,
     EXTERNAL_VERSION_STATE_CLOSED,
     EXTERNAL_VERSION_STATE_OPEN,
+    LATEST,
     TAG,
 )
 from readthedocs.builds.models import Build, BuildCommandResult, Version
 from readthedocs.builds.tasks import (
     archive_builds_task,
     delete_closed_external_versions,
+    post_build_overview,
 )
+from readthedocs.filetreediff.dataclasses import FileTreeDiff, FileTreeDiffFileStatus
+from readthedocs.oauth.constants import GITHUB_APP
+from readthedocs.oauth.models import (
+    GitHubAccountType,
+    GitHubAppInstallation,
+    RemoteRepository,
+)
+from readthedocs.oauth.services import GitHubAppService
 from readthedocs.projects.models import Project
 
 
@@ -110,3 +123,217 @@ def test_archive_builds(self, build_commands_storage):
         self.assertEqual(Build.objects.count(), 10)
         self.assertEqual(Build.objects.filter(cold_storage=True).count(), 5)
         self.assertEqual(BuildCommandResult.objects.count(), 50)
+
+
+@override_settings(
+    PRODUCTION_DOMAIN="readthedocs.org",
+    PUBLIC_DOMAIN="readthedocs.io",
+    RTD_EXTERNAL_VERSION_DOMAIN="readthedocs.build",
+)
+class TestPostBuildOverview(TestCase):
+
+    def setUp(self):
+        self.user = get(User)
+        self.github_app_installation = get(
+            GitHubAppInstallation,
+            installation_id=1111,
+            target_id=1111,
+            target_type=GitHubAccountType.USER,
+        )
+        self.remote_repository = get(
+            RemoteRepository,
+            name="repo",
+            full_name="user/repo",
+            vcs_provider=GITHUB_APP,
+            github_app_installation=self.github_app_installation,
+        )
+
+        self.project = get(
+            Project,
+            name="My project",
+            slug="my-project",
+            users=[self.user],
+            remote_repository=self.remote_repository,
+        )
+        self.base_version = self.project.versions.get(slug=LATEST)
+        self.base_version.built = True
+        self.base_version.save()
+        self.base_version_build = get(
+            Build,
+            project=self.project,
+            version=self.base_version,
+            commit="1234abcd",
+            state=BUILD_STATE_FINISHED,
+            success=True,
+        )
+        self.current_version = get(
+            Version,
+            project=self.project,
+            verbose_name="1",
+            slug="1",
+            type=EXTERNAL,
+            active=True,
+            built=True,
+        )
+        self.current_version_build = get(
+            Build,
+            project=self.project,
+            version=self.current_version,
+            commit="5678abcd",
+            state=BUILD_STATE_FINISHED,
+            success=True,
+        )
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    @mock.patch("readthedocs.builds.reporting.get_diff")
+    def test_post_build_overview(self, get_diff, post_comment):
+        get_diff.return_value = FileTreeDiff(
+            current_version=self.current_version,
+            current_version_build=self.current_version_build,
+            base_version=self.base_version,
+            base_version_build=self.base_version_build,
+            files=[
+                ("index.html", FileTreeDiffFileStatus.modified),
+                ("changes.html", FileTreeDiffFileStatus.added),
+                ("deleteme.html", FileTreeDiffFileStatus.deleted),
+            ],
+            outdated=False,
+        )
+        post_build_overview(build_pk=self.current_version_build.pk)
+        expected_comment = dedent(
+            f"""
+            ## Documentation build overview
+
+            > 📚 [My project](https://readthedocs.org/projects/my-project/) | 🛠️ build [#{self.current_version_build.id}](https://readthedocs.org/projects/my-project/builds/{self.current_version_build.id}/) (5678abcd) | 🔍 [preview](http://my-project--1.readthedocs.build/en/1/)
+
+            ### Files changed
+
+            > Comparing with [latest](http://my-project.readthedocs.io/en/latest/) (1234abcd)
+
+
+            <details>
+            <summary>Show files (3) | 1 modified | 1 added | 1 deleted</summary>
+
+            | File | Status |
+            | --- | --- |
+            | [changes.html](http://my-project--1.readthedocs.build/en/1/changes.html) | ➕ added |
+            | [deleteme.html](http://my-project--1.readthedocs.build/en/1/deleteme.html) | ❌ deleted |
+            | [index.html](http://my-project--1.readthedocs.build/en/1/index.html) | 📝 modified |
+
+
+            </details>
+
+            """
+        )
+        post_comment.assert_called_once_with(
+            build=self.current_version_build,
+            comment=expected_comment,
+        )
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    @mock.patch("readthedocs.builds.reporting.get_diff")
+    def test_post_build_overview_more_than_5_files(self, get_diff, post_comment):
+        get_diff.return_value = FileTreeDiff(
+            current_version=self.current_version,
+            current_version_build=self.current_version_build,
+            base_version=self.base_version,
+            base_version_build=self.base_version_build,
+            files=[
+                ("index.html", FileTreeDiffFileStatus.modified),
+                ("changes.html", FileTreeDiffFileStatus.added),
+                ("deleteme.html", FileTreeDiffFileStatus.deleted),
+                ("one.html", FileTreeDiffFileStatus.modified),
+                ("three.html", FileTreeDiffFileStatus.modified),
+                ("two.html", FileTreeDiffFileStatus.modified),
+            ],
+            outdated=False,
+        )
+        post_build_overview(build_pk=self.current_version_build.pk)
+        expected_comment = dedent(
+            f"""
+            ## Documentation build overview
+
+            > 📚 [My project](https://readthedocs.org/projects/my-project/) | 🛠️ build [#{self.current_version_build.id}](https://readthedocs.org/projects/my-project/builds/{self.current_version_build.id}/) (5678abcd) | 🔍 [preview](http://my-project--1.readthedocs.build/en/1/)
+
+            ### Files changed
+
+            > Comparing with [latest](http://my-project.readthedocs.io/en/latest/) (1234abcd)
+
+
+            <details>
+            <summary>Show files (6) | 4 modified | 1 added | 1 deleted</summary>
+
+            | File | Status |
+            | --- | --- |
+            | [changes.html](http://my-project--1.readthedocs.build/en/1/changes.html) | ➕ added |
+            | [deleteme.html](http://my-project--1.readthedocs.build/en/1/deleteme.html) | ❌ deleted |
+            | [index.html](http://my-project--1.readthedocs.build/en/1/index.html) | 📝 modified |
+            | [one.html](http://my-project--1.readthedocs.build/en/1/one.html) | 📝 modified |
+            | [three.html](http://my-project--1.readthedocs.build/en/1/three.html) | 📝 modified |
+            | [two.html](http://my-project--1.readthedocs.build/en/1/two.html) | 📝 modified |
+
+
+            </details>
+
+            """
+        )
+        post_comment.assert_called_once_with(
+            build=self.current_version_build,
+            comment=expected_comment,
+        )
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    @mock.patch("readthedocs.builds.reporting.get_diff")
+    def test_post_build_overview_no_files_changed(self, get_diff, post_comment):
+        get_diff.return_value = FileTreeDiff(
+            current_version=self.current_version,
+            current_version_build=self.current_version_build,
+            base_version=self.base_version,
+            base_version_build=self.base_version_build,
+            files=[],
+            outdated=False,
+        )
+        post_build_overview(build_pk=self.current_version_build.pk)
+        expected_comment = dedent(
+            f"""
+            ## Documentation build overview
+
+            > 📚 [My project](https://readthedocs.org/projects/my-project/) | 🛠️ build [#{self.current_version_build.id}](https://readthedocs.org/projects/my-project/builds/{self.current_version_build.id}/) (5678abcd) | 🔍 [preview](http://my-project--1.readthedocs.build/en/1/)
+
+            ### Files changed
+
+            > Comparing with [latest](http://my-project.readthedocs.io/en/latest/) (1234abcd)
+
+
+            No files changed.
+
+            """
+        )
+        post_comment.assert_called_once_with(
+            build=self.current_version_build,
+            comment=expected_comment,
+        )
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    def test_post_build_overview_no_external_version(self, post_comment):
+        assert not self.base_version.is_external
+        post_build_overview(build_pk=self.base_version_build.pk)
+        post_comment.assert_not_called()
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    def test_post_build_overview_no_github_app_project(self, post_comment):
+        self.project.remote_repository = None
+        self.project.save()
+
+        assert not self.project.is_github_app_project
+        assert self.current_version.is_external
+        post_build_overview(build_pk=self.current_version_build.pk)
+        post_comment.assert_not_called()
+
+    @mock.patch.object(GitHubAppService, "post_comment")
+    @mock.patch("readthedocs.builds.reporting.get_diff")
+    def test_post_build_overview_no_diff_available(self, get_diff, post_comment):
+        get_diff.return_value = None
+        assert self.current_version.is_external
+        post_build_overview(build_pk=self.current_version_build.pk)
+        post_comment.assert_not_called()