rerank() can return undefined documents for invalid provider ranking indices

[canblmz1]

Description

rerank() maps every provider-returned ranking[i].index straight back onto the
input documents array with a bare documents[index] lookup, and never validates
the index. The public result contract is non-optional:

// RerankResult<VALUE>
ranking: Array<{ originalIndex: number; score: number; document: VALUE }>;
rerankedDocuments: Array<VALUE>;

@ai-sdk/provider documents RerankingModelV4Result.ranking[].index as "the index
of the document in the original list of documents before reranking".

When a reranking model returns an index that is out of range or non-integer,
rerank() resolves successfully with document: undefined in ranking and
undefined entries in rerankedDocuments — no error, no retry, and the onEnd /
telemetry end event fires as if the call succeeded. The undefined then tends to
surface later as a TypeError far from the cause.

This is the same shape as #20355 (embed() resolves with an undefined embedding)
and #20351 (empty choices[]): a structurally valid but semantically invalid
successful provider response passed through as a value that violates the SDK's own
result type.

Actual behavior

documents = ['a', 'b', 'c'], one ranking entry, default maxRetries:

provider index	outcome	ranking[0].document	rerankedDocuments	doRerank calls	onEnd fired
3 (=== length)	resolves	undefined	[undefined]	1	yes
-1	resolves	undefined	[undefined]	1	yes
5 (> length)	resolves	undefined	[undefined]	1	yes
1.5	resolves	undefined	[undefined]	1	yes

The malformed response is not retried (the provider is called once), so this is
deterministic rather than a transient failure.

Expected behavior

An invalid provider ranking index should not produce a successful RerankResult
containing undefined documents. Adjacent malformed-successful-response handling
in the same package surfaces this as InvalidResponseDataError
(embed-many.ts validateEmbeddingCount; #20358 / #20362) — but the exact error
type and whether the check belongs inside or outside the retry callback are for
maintainers to decide.

Control (unchanged, valid input)

ranking indices [2, 0]  ->  rerankedDocuments ['c', 'a']

Valid in-range rankings continue to work as before.

Source pointers (main @ c3c189c068baefe62808b4eca5432fe092dc37cb)

• packages/ai/src/rerank/rerank.ts — document: documents[ranking.index] in
  both the onEnd event payload and the DefaultRerankResult construction (both
  after retry() resolves); no Number.isInteger / 0 <= index < documents.length
  check.
• packages/ai/src/rerank/rerank-result.ts — ranking[].document: VALUE,
  rerankedDocuments: Array<VALUE>.
• packages/provider/src/reranking-model/v4/reranking-model-v4-result.ts —
  index semantics.

I searched issues, PRs, and recent commits (rerank invalid index,
rerank out of range, rerank undefined document, ranking index validation,
InvalidResponseDataError rerank) and found nothing covering rerank index
validation.

Reproduction

Minimal, using the public API and the shipped mock (ai @ main,
c3c189c068baefe62808b4eca5432fe092dc37cb — current main at the time of
reproduction):

import { rerank } from 'ai';
import { MockRerankingModelV4 } from 'ai/test';

const result = await rerank({
  model: new MockRerankingModelV4({
    // documents.length === 3, so index 3 is out of range
    doRerank: async () => ({ ranking: [{ index: 3, relevanceScore: 0.9 }] }),
  }),
  documents: ['a', 'b', 'c'],
  query: 'q',
});

console.log(result.ranking[0].document); // => undefined
console.log(result.rerankedDocuments);   // => [ undefined ]

As a failing test in packages/ai/src/rerank/:

import { expect, it } from 'vitest';
import { MockRerankingModelV4 } from '../test/mock-reranking-model-v4';
import { rerank } from './rerank';

it('does not resolve with an undefined document for an out-of-range index', async () => {
  const model = new MockRerankingModelV4({
    doRerank: async () => ({ ranking: [{ index: 3, relevanceScore: 0.9 }] }),
  });

  await expect(
    rerank({ model, documents: ['a', 'b', 'c'], query: 'q' }),
  ).rejects.toThrow();
});

On current main this test fails: rerank() resolves with
DefaultRerankResult { ranking: [{ document: undefined, originalIndex: 3, score: 0.9 }] }.

AI SDK Version

• ai: main @ c3c189c068baefe62808b4eca5432fe092dc37cb (also reproduces on published ai@7.0.77)
• Node: v22.13+

Code of Conduct

• I agree to follow this project's Code of Conduct

[ai-sdk-factory]

Identify issue type

Identified as Bug
Confidence: High

Rationale

This is a concrete failure in existing SDK behavior: malformed provider indices produce successful results that violate the documented result type instead of raising an error.

Evidence

• rerank() accepts out-of-range, negative, and non-integer provider ranking indices without validation.
• Invalid indices resolve successfully with undefined documents despite the public RerankResult contract declaring non-optional VALUE entries.
• A minimal reproduction and failing test demonstrate the behavior on current main and published ai@7.0.77.

[ai-sdk-factory]

Bug Reproduction - AI SDK 7

Bug was reproduced.

Branch: main
PR: successful reproduction pull request

Details

• On main at c3c189c068baefe62808b4eca5432fe092dc37cb, packages/ai/src/rerank/rerank.ts maps provider indices through unvalidated documents[index] lookups despite the non-optional result contract.
• Reproduction artifact examples/ai-functions/src/reproduction/rerank-invalid-provider-indices.ts showed indices 3, -1, 5, and 1.5 all resolving with undefined in ranking, rerankedDocuments, and the onEnd payload.
• Each malformed case called doRerank once and fired onEnd once as a successful operation; invalid indices were expected not to expose successful results or end events containing undefined documents.
• The valid control ranking [2, 0] continued to produce ['c', 'a'].

[ai-sdk-factory]

Bugfix - AI SDK 7

Created bugfix and opened PR: #20423

Root Cause

rerank() mapped provider indices through unchecked documents[index] lookups after the provider call. The reproduction confirmed that out-of-range, negative, and fractional indices all produced undefined documents.

Summary

Validated provider ranking indices as in-range integers and now throw InvalidResponseDataError before constructing results or firing onEnd. Added a patch changeset and removed reproduction-only artifacts.

Testing

Added regression coverage for out-of-range, negative, and fractional indices, verifying rejection, a single provider call, and no onEnd callback.

[ai-sdk-factory]

Bug Reproduction - AI SDK 5

Issue report appears invalid.

Target branch: release-v5.0

• The reported primary failure cannot occur on the target branch because no rerank() API or RerankResult can be constructed there.
• The invalid assumption is that issue rerank() can return undefined documents for invalid provider ranking indices #20421's AI SDK 7 rerank() and MockRerankingModelV4 APIs exist on release-v5.0.
• packages/ai/package.json identifies the target as ai@5.0.255; packages/ai/src/index.ts has no rerank export, and packages/ai/test/index.ts has no reranking mock export.
• packages/provider/package.json identifies @ai-sdk/provider@2.0.4, while packages/provider/src/index.ts exposes V2 model families and no reranking model contract.
• content/docs/01-announcing-ai-sdk-6-beta/index.mdx explicitly states that AI SDK 6 introduces native reranking support, confirming it is unavailable in AI SDK 5.

Relevant Documentation

• https://ai-sdk.dev/v5/docs/reference/ai-sdk-core
• https://ai-sdk.dev/docs/reference/ai-sdk-core/rerank

[ai-sdk-factory]

Bug Reproduction - AI SDK 6

Bug was reproduced.

Branch: release-v6.0
PR: successful reproduction pull request

Details

• On release-v6.0, packages/ai/src/rerank/rerank.ts maps provider indices through an unchecked documents[ranking.index] lookup despite the non-optional result contract in packages/ai/src/rerank/rerank-result.ts.
• Reproduction artifact examples/ai-functions/src/reproduction/issue-20421-rerank-invalid-provider-indices.ts showed indices 3, -1, 5, and 1.5 each resolving with one undefined ranking document and one undefined reranked document.
• Every malformed case called doRerank exactly once with default retry settings, confirming the malformed successful response was not retried.
• The valid control indices [2, 0] returned ['c', 'a'] and called the provider once.
• This branch uses RerankingModelV3, whose index semantics match the reported V4 contract; the V6 rerank() API does not expose the report's V7-specific onEnd callback.

Relevant Documentation

• https://ai-sdk.dev/docs/reference/ai-sdk-core/rerank