🐛 [storage-resize-images] Content filter does not work

[artanisdesign]

[READ] Step 1: Are you in the right place?

Yes — bug specific to storage-resize-images.

[REQUIRED] Step 2: Describe your configuration

• Extension name: storage-resize-images
• Extension version: latest (master at time of filing)
• Cloud Function region: europe-west2
• Configuration values:
  • Content filter level: BLOCK_LOW_AND_ABOVE (also reproduced with BLOCK_MEDIUM_AND_ABOVE)
  • Custom content filter prompt: tested with multiple variants (see below)
  • Path to placeholder image: default
  • All other settings: defaults

[REQUIRED] Step 3: Describe the problem

The content filtering feature silently fails open on borderline NSFW images. Inappropriate images that should be blocked are processed normally with no placeholder substitution, despite the filter being configured at the strictest level. The root cause is a mismatch between the error shape the extension's catch handler expects and the actual error shape produced by Gemini 2.5 Flash when its input-side safety refuses to respond.

There are two related bugs in functions/src/content-filter.ts:

Bug 1 — Schema validation failure on safety refusal is not treated as a block.

When Gemini 2.5 Flash's input-side safety declines to respond to an image (which happens often on borderline content — exactly the content the filter should be catching), it returns an empty response with no finishReason. Genkit's structured output validator then throws INVALID_ARGUMENT: Schema validation failed because null doesn't match the required {response: string} schema.

The catch handler in performContentCheck only recognises errors with error.detail?.response?.finishReason === "blocked":

} catch (error) {
  if (error.detail?.response?.finishReason === "blocked") {
    log.contentFilterBlocked();
    return false;
  }
  throw error;
}

The schema validation error doesn't have that shape, so it gets rethrown. After 3 retries (which all fail identically because the failure is deterministic, not transient), the error propagates up to processContentFilter, which catches it, sets failed = true, and returns. Because the placeholder substitution logic only runs when filterResult === false (not when failed === true), the original image is processed without any filtering applied.

Bug 2 — The no-prompt path cannot block based on model verdict.

When no custom prompt is configured, effectivePrompt is "Is this image appropriate?" with maxOutputTokens: 1, but the verdict-checking branch is gated on prompt !== null:

if (result.output?.response === "yes" && prompt !== null) {
  log.customFilterBlocked();
  return false;
}
return true;

This means without a custom prompt, the model's response is never inspected — the function always returns true unless the catch handler fires. Combined with Bug 1, this means users who select a content filter level but don't set a custom prompt get effectively no filtering at all, even though the UI implies otherwise.

Steps to reproduce:

1. Install storage-resize-images with CONTENT_FILTER_LEVEL=BLOCK_LOW_AND_ABOVE.
2. Set CUSTOM_FILTER_PROMPT to anything moderation-shaped, e.g.:

   You are classifying images for a family-friendly platform. Look at the image and decide whether it depicts any of: people in bed together, people who appear undressed or covered only by sheets, romantic kissing, or intimate physical contact. Answer 'yes' if any of these are present, otherwise 'no'.

3. Upload a borderline NSFW image (e.g. a stock photo of people in bed together, no explicit nudity).
4. Observe Cloud Functions logs for ext-storage-resize-images-generateResizedImage.

Expected result

The image is recognised as inappropriate and replaced with the placeholder image before resizing. A log entry indicates the content filter blocked the image.

Actual result

Three retry attempts fail with identical errors:

Unexpected Error whilst evaluating content of image with Gemini (Attempt 2/3).
ValidationError [GenkitError]: INVALID_ARGUMENT: Schema validation failed. Parse Errors:

- (root): must be object

Provided data:

null

Required JSON schema:

{
  "type": "object",
  "properties": {
    "response": { "type": "string" }
  },
  "required": ["response"],
  "additionalProperties": true,
  "$schema": "http://json-schema.org/draft-07/schema#"
}
    at parseSchema (/workspace/node_modules/@genkit-ai/core/lib/schema.js:104:21)
    at GenerateResponse.assertValidSchema (/workspace/node_modules/@genkit-ai/ai/lib/generate/response.js:78:37)
    ...
  status: 'INVALID_ARGUMENT',
  code: 400,

The retries don't help because the failure is deterministic (same image + same prompt → same null response every time). After the final retry, the error propagates to processContentFilter, the image is resized normally, and no placeholder is substituted. From the user's perspective, the filter has silently allowed inappropriate content through.

Removing the custom prompt and relying only on CONTENT_FILTER_LEVEL does not help, because of Bug 2 above — the no-prompt path's verdict check is gated on prompt !== null, so the model's response is never inspected.

Suggested fix

The catch handler in performContentCheck should treat schema validation failures with null content as implicit blocks, since in practice an empty response from Gemini on an image input almost always indicates the input-side safety system declined to engage:

} catch (error) {
  if (error.detail?.response?.finishReason === "blocked") {
    log.contentFilterBlocked();
    return false;
  }
  // Gemini 2.5 Flash returns null content (rather than a clean
  // SAFETY finishReason) when input-side safety refuses on image
  // inputs. Genkit surfaces this as a schema validation error.
  // Treat it as a block.
  if (
    error.status === "INVALID_ARGUMENT" &&
    error.originalMessage?.includes("Provided data:\n\nnull")
  ) {
    log.contentFilterBlocked();
    return false;
  }
  throw error;
}

Additionally, the prompt !== null gate on the verdict check should be removed so the no-prompt path can actually use the model's response (or the no-prompt path should be removed entirely if it isn't intended to do verdict-based filtering).

Without one of these fixes, the content filtering feature does not function as documented for the most common use case (borderline NSFW imagery on a user-upload platform), and fails silently in a way that's only discoverable by reading Cloud Functions logs and the extension source code.

[cabljac]

Hi thanks for raising this, we'll look into it and create a fix asap.