ci: use pagination to retrieve complete list of changed files [backport #6818 to 1.18] (#6829)

Backport of #6818 to 1.18

We make sure to fetch all the pages from the call to the pull request
GitHub REST API so that we can get the full list of changed files,
instead of just the first 30 entries.

## Checklist

- [x] Change(s) are motivated and described in the PR description.
- [x] Testing strategy is described if automated tests are not included
in the PR.
- [x] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed. If no release note is required, add label
`changelog/no-changelog`.
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer has explicitly acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment.
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
- [x] If this PR touches code that signs or publishes builds or
packages, or handles credentials of any kind, I've requested a review
from `@DataDog/security-design-and-guidance`.
- [x] This PR doesn't touch any of that.

Author: P403n1x87

scripts/needs_testrun.py

@@ -3,6 +3,7 @@
 from argparse import ArgumentParser
 import fnmatch
 from functools import cache
+from itertools import count
 import json
 import logging
 import os
@@ -72,13 +73,18 @@ def get_changed_files(pr_number: int, sha: t.Optional[str] = None) -> t.Set[str]
     """
     rest_check_failed = False
     if sha is None:
+        files = set()
         try:
-            url = f"https://api.github.com/repos/datadog/dd-trace-py/pulls/{pr_number}/files"
-            headers = {"Accept": "application/vnd.github+json"}
-            return {_["filename"] for _ in json.load(urlopen(Request(url, headers=headers)))}
+            for page in count(1):
+                url = f"https://api.github.com/repos/datadog/dd-trace-py/pulls/{pr_number}/files?page={page}"
+                headers = {"Accept": "application/vnd.github+json"}
+                result = {_["filename"] for _ in json.load(urlopen(Request(url, headers=headers)))}
+                if not result:
+                    return files
+                files |= result
         except Exception:
             rest_check_failed = True
-            LOGGER.warning("Failed to get changed files from GitHub API")
+            LOGGER.warning("Failed to get changed files from GitHub API", exc_info=True)
 
     if sha is not None or rest_check_failed:
         diff_base = sha or get_merge_base(pr_number)
@@ -175,7 +181,9 @@ def main() -> bool:
 
     argp.add_argument("suite", help="The suite to use", type=str)
     argp.add_argument("--pr", help="The PR number", type=int, default=_get_pr_number())
-    argp.add_argument("--sha", help="Commit hash to use as diff base (defaults to PR merge root)", type=lambda v: v or None)
+    argp.add_argument(
+        "--sha", help="Commit hash to use as diff base (defaults to PR merge root)", type=lambda v: v or None
+    )
     argp.add_argument("--verbose", "-v", action="store_true", help="Verbose output")
 
     args = argp.parse_args()