Allow scanners to run asynchronously and send their results later (#24447)

Author: willdurand

docs/topics/api/overview.rst

@@ -481,6 +481,7 @@ These are `v5` specific changes - `v4` changes apply also.
 * 2025-09-01: added ``created`` and ``updated`` query parameters to addon search api. https://github.com/mozilla/addons/issues/15814
 * 2025-09-18: added /rollback endpoint to version detail api. https://github.com/mozilla/addons/issues/15696
 * 2026-02-19: added ``listingcontentreview`` endpoint for addons. https://github.com/mozilla/addons/issues/16050
+* 2026-02-19: added the ability to patch a scanner result. https://github.com/mozilla/addons/issues/16004
 
 .. _`#11380`: https://github.com/mozilla/addons-server/issues/11380/
 .. _`#11379`: https://github.com/mozilla/addons-server/issues/11379/

docs/topics/api/scanners.rst

@@ -7,9 +7,9 @@ Scanners
     These APIs are subject to change at any time and are for internal use only.
 
 
----------------------
-Scanner Results
----------------------
+--------------------
+List scanner results
+--------------------
 
 .. _scanner-results:
 
@@ -29,3 +29,24 @@ This endpoint returns a list of labelled scanner results.
     :>json object results: The scanner (raw) results.
     :>json string created: The date the result was created, formatted with `this format <http://ecma-international.org/ecma-262/5.1/#sec-15.9.1.15>`_.
     :>json string|null model_version: The model version when applicable, ``null`` otherwise.
+
+
+----------------------
+Patch - Update results
+----------------------
+
+.. _scanner-result-patch:
+
+This endpoint allows to update scanner results.
+
+    .. note::
+        Requires JWT authentication using the service account credentials
+        associated with the scanner webhook.
+
+.. http:patch:: /api/v5/scanner/results/(int:pk)/
+
+    :query string id: The scanner result ID.
+    :<json object results: The scanner results.
+    :statuscode 204: Results successfully updated.
+    :statuscode 400: Invalid payload.
+    :statuscode 409: Scanner results already recorded.

docs/topics/development/scanner_pipeline.md

@@ -9,8 +9,10 @@ occur on AMO. These webhooks are registed in the AMO (django) admin.
 
 When a [scanner webhook event](#scanner-webhook-events) occurs, AMO will send an
 HTTP request to each webhook subscribed to this event. The payload sent to the
-webook depends on the event. The response from each webhook will lead to the
-creation of a [scanner result](#scanner-results).
+webook depends on the event. AMO creates a [scanner result](#scanner-results)
+before calling the webhook, and includes a `scanner_result_url` in the payload
+that allows the scanner to [asynchronously send its results](#asynchronous-scanning)
+back to AMO.
 
 Each service registered as a scanner webhook must be protected with a shared
 secret (api) key. Read [the scanners authentication
@@ -48,11 +50,13 @@ validation chain.
 
 The payload sent looks like this. Assuming correct permissions, the URL in
 `download_url` allows the services notified for this event to download the (raw)
-uploaded file.
+uploaded file. The `scanner_result_url` allows the scanner to send results
+asynchronously.
 
 ```json
 {
-  "download_url": "http://olympia.test/uploads/file/42"
+  "download_url": "http://olympia.test/uploads/file/42",
+  "scanner_result_url": "http://olympia.test/api/v5/scanner/results/123/"
 }
 ```
 
@@ -68,7 +72,8 @@ The payload sent looks like this:
   "version_id": 42,
   "download_source_url": "http://olympia.test/downloads/source/42",
   "license_slug": "MPL-2.0",
-  "activity_log_id": 2170
+  "activity_log_id": 2170,
+  "scanner_result_url": "http://olympia.test/api/v5/scanner/results/124/"
 }
 ```
 
@@ -112,6 +117,8 @@ These actions are defined in `src/olympia/scanners/actions.py`.
 (scanners-authentication)=
 ### Authentication
 
+#### Authenticating incoming webhook calls
+
 Scanners must verify the incoming requests using the `Authorization` header and
 not allow unauthenticated requests. For every webhook call, AMO will send this
 header using the _API key_ defined in the Django admin as follows:
@@ -124,11 +131,57 @@ Authorization: HMAC-SHA256 <hexdigest>
 the _API key_ used as the secret key. Make sure to hash the _raw_ request's
 body.
 
+#### Authenticating asynchronous result submissions
+
+When sending results asynchronously via PATCH to the `scanner_result_url`,
+scanners must authenticate using JWT credentials. Each scanner webhook has an
+automatically created service account, and the JWT keys for this account are
+displayed in the Django admin after creating the webhook.
+
+Use the JWT key and secret to generate a JWT token and include it in the
+`Authorization` header when making PATCH requests to submit results
+asynchronously.
+
 ### API response
 
-Scanners must return a JSON response that contains the following fields:
+Scanners can choose to return results synchronously or asynchronously:
+
+#### Synchronous response
+
+Scanners can return a JSON response immediately that contains the following fields:
 
 - `version`: the scanner version
+- `matchedRules`: an array of matched rule identifiers (string)
+
+(asynchronous-scanning)=
+#### Asynchronous response
+
+Scanners can also return a quick acknowledgment response (or any response) and
+send their results later using the `scanner_result_url` provided in the webhook
+payload. This is useful for long-running scans.
+
+To send results asynchronously:
+
+1. The scanner receives a webhook call with a `scanner_result_url` in the payload
+2. The scanner returns a quick response (e.g., HTTP 202 Accepted)
+3. The scanner performs its analysis
+4. The scanner sends a PATCH request to the `scanner_result_url` with the results
+
+The PATCH request must be authenticated using the [service account JWT
+credentials](#scanners-authentication) and include a JSON body with a `results`
+field:
+
+```json
+{
+  "results": {
+    "version": "1.0.0",
+    "matchedRules": []
+  }
+}
+```
+
+The `results` field should contain the same data structure as a synchronous
+response would return.
 
 ### Creating a new scanner
 
@@ -149,7 +202,13 @@ import { createExpressApp } from "addons-scanner-utils";
 const handler = (req, res) => {
   console.log({ data: req.body });
 
+  // Option 1: Synchronous response
   res.json({ version: "1.0.0" });
+
+  // Option 2: Asynchronous response (for long-running scans)
+  // res.status(202).json({ message: "Scan started" });
+  // // Perform scanning asynchronously and later send results to:
+  // // req.body.scanner_result_url
 };
 
 const app = createExpressApp({
@@ -183,7 +242,8 @@ When uploading a new file, you should see the following in the console:
 ```js
 {
   data: {
-    download_url: "http://olympia.test/uploads/file/fa7868396b7e44ef8a0711f608f534f7/?access_token=w0Tl7qmJqBMQ4gtitKbcdKozulWVQWhkU0wEA10N"
+    download_url: "http://olympia.test/uploads/file/fa7868396b7e44ef8a0711f608f534f7/?access_token=w0Tl7qmJqBMQ4gtitKbcdKozulWVQWhkU0wEA10N",
+    scanner_result_url: "http://olympia.test/api/v5/scanner/results/123/"
   }
 }
 ```

src/olympia/api/tests/utils.py

@@ -52,6 +52,11 @@ def put(self, url, data=None, **client_kwargs):
             url, data, HTTP_AUTHORIZATION=self.authorization(), **client_kwargs
         )
 
+    def patch(self, url, data=None, **client_kwargs):
+        return self.client.patch(
+            url, data, HTTP_AUTHORIZATION=self.authorization(), **client_kwargs
+        )
+
     def auth_required(self, cls):
         """
         Tests that the JWT Auth class is on the class, without having

src/olympia/constants/permissions.py

@@ -102,6 +102,9 @@
 # Can submit language packs. #11788 and #11793
 LANGPACK_SUBMIT = AclPermission('LanguagePack', 'Submit')
 
+# Scanners can use a special endpoint to update their results.
+SCANNERS_PATCH_RESULTS = AclPermission('Scanners', 'PatchResults')
+
 # Can submit add-ons signed with Mozilla internal certificate, or add-ons with
 # a guid ending with reserved suffixes like @mozilla.com
 SYSTEM_ADDON_SUBMIT = AclPermission('SystemAddon', 'Submit')

src/olympia/scanners/admin.py

@@ -1191,9 +1191,7 @@ def formatted_events_list(self, obj):
 
     def service_account(self, obj):
         try:
-            user = UserProfile.objects.get_service_account(
-                name=obj.service_account_name
-            )
+            user = obj.service_account
         except UserProfile.DoesNotExist:
             return '(will be automatically created)'
 
@@ -1213,9 +1211,7 @@ def save_model(self, request, obj, form, change):
         if not change:
             # Display the JWT keys only once on creation.
             try:
-                user = UserProfile.objects.get_service_account(
-                    name=obj.service_account_name
-                )
+                user = obj.service_account
                 api_key = APIKey.get_jwt_key(user=user)
                 messages.add_message(
                     request,

src/olympia/scanners/api_urls.py

@@ -1,11 +1,18 @@
 from django.conf import settings
 from django.urls import re_path
 
-from .views import ScannerResultView
+from .views import ScannerResultView, patch_scanner_result
 
 
-urlpatterns = (
-    [re_path(r'^results/$', ScannerResultView.as_view(), name='scanner-results')]
-    if settings.INTERNAL_ROUTES_ALLOWED
-    else []
-)
+urlpatterns = [
+    re_path(
+        r'^results/(?P<pk>\d+)/$',
+        patch_scanner_result,
+        name='scanner-result-patch',
+    ),
+]
+
+if settings.INTERNAL_ROUTES_ALLOWED:
+    urlpatterns.append(
+        re_path(r'^results/$', ScannerResultView.as_view(), name='scanner-results')
+    )

src/olympia/scanners/migrations/0070_add_patch_results_permission_to_group.py

@@ -0,0 +1,22 @@
+from django.db import migrations
+
+from olympia.constants.scanners import SCANNER_SERVICE_ACCOUNTS_GROUP
+
+
+def add_permission_to_group(apps, schema_editor):
+    Group = apps.get_model('access', 'Group')
+
+    group = Group.objects.get(name=SCANNER_SERVICE_ACCOUNTS_GROUP)
+    rules = group.rules.split(',') if group.rules else []
+    rules.append('Scanners:PatchResults')
+    group.rules = ','.join(rules)
+    group.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('scanners', '0069_add_service_accounts_to_group'),
+    ]
+
+    operations = [migrations.RunPython(add_permission_to_group)]

src/olympia/scanners/models.py

@@ -294,7 +294,7 @@ class Meta:
 
     def save(self, *args, **kwargs):
         service_account, created = UserProfile.objects.get_or_create_service_account(
-            name=self.service_account_name,
+            name=self._service_account_name,
             notes=(
                 'Service account automatically created for '
                 f'the "{self.name}" scanner webhook.'
@@ -308,9 +308,13 @@ def save(self, *args, **kwargs):
         return super().save(*args, **kwargs)
 
     @property
-    def service_account_name(self):
+    def _service_account_name(self):
         return f'webhook-{self.name}'
 
+    @property
+    def service_account(self):
+        return UserProfile.objects.get_service_account(self._service_account_name)
+
     def __str__(self):
         return self.name
 
@@ -366,6 +370,10 @@ class Meta(AbstractScannerResult.Meta):
     def rule_model(self):
         return self.matched_rules.rel.model
 
+    @property
+    def webhook(self):
+        return self.webhook_event.webhook if self.webhook_event else None
+
     def get_rules_queryset(self):
         # See: https://github.com/mozilla/addons-server/issues/13143
         return super().get_rules_queryset().filter(is_active=True)

src/olympia/scanners/serializers.py

@@ -23,3 +23,20 @@ class Meta:
 
     def get_scanner(self, obj):
         return obj.get_scanner_name()
+
+
+class PatchScannerResultSerializer(serializers.Serializer):
+    """Serializer for updating scanner result data via PATCH endpoint."""
+
+    results = serializers.JSONField()
+
+    def validate(self, data):
+        # Validate that no extra fields are present in the initial data.
+        if hasattr(self, 'initial_data'):
+            extra_fields = set(self.initial_data.keys()) - set(self.fields.keys())
+            if extra_fields:
+                raise serializers.ValidationError(
+                    {field: 'Unexpected field.' for field in extra_fields}
+                )
+
+        return data