feat(core): Add byte size limit and oldest first truncation for gen_ai messages

[naaa760]

Description:

fixes: #17809

• Implements message truncation for gen_ai.request.messages and vercel.ai.prompt to prevent oversized payloads from exceeding ingestion limits.

Changes:

• Add messageTruncation.ts utility with byte size calculation and oldest-first truncation
• Apply truncation to all AI integrations (OpenAI, Google GenAI, Anthropic, Vercel AI)
• Set 100KB default limit for gen_ai messages
• Preserve recent message context by removing oldest messages first

[RulaKhaled]

Thank you for contributing 🎉 ! Let's address some comments before merging this

[RulaKhaled]

can you revert the comment removal to help others understand the request structure? this could also be a string

[RulaKhaled]

Can you add JSDocs for this files?

[RulaKhaled]

can you also revert back this JSDoc comment?

[RulaKhaled]

can you also revert back this JSDoc comment?

[naaa760]

@RulaKhaled
addressed the comments
let me know!

[RulaKhaled]

Thanks for resolving comments, it already looks better! I've pushed a quick commit for truncation functionality + added some JSDocs here :4d047e5 (this is a different branch) could you give this a shot? :) + can you take care of tests and the quick comments left?

[RulaKhaled]

It's ok to keep the previous getByteSize where you use TextEncoder directly, the binary search is meant to be for:

1. truncating strings messages
2. truncating array of messages

e.g:

export function truncateMessagesByBytes(messages: unknown[], maxBytes: number): unknown[] {
if (!Array.isArray(messages) || messages.length === 0) {
  return messages;
}

const fullSize = getByteSize(JSON.stringify(messages));

if (fullSize <= maxBytes) {
  return messages;
}

// Binary search for the minimum startIndex where remaining messages fit (works for single or multiple messages)
let left = 0;
let right = messages.length - 1;
let bestStartIndex = messages.length;

while (left <= right) {
  const mid = Math.floor((left + right) / 2);
  const remainingMessages = messages.slice(mid);
  const remainingSize = getByteSize(JSON.stringify(remainingMessages));

  if (remainingSize <= maxBytes) {
    bestStartIndex = mid;
    right = mid - 1; // Try to keep more messages
  } else {
    // If we're down to a single message and it doesn't fit, break and handle content truncation
    if (remainingMessages.length === 1) {
      bestStartIndex = mid; // Use this single message
      break;
    }
    left = mid + 1; // Need to remove more messages
  }
}

const remainingMessages = messages.slice(bestStartIndex);

// SPECIAL CASE: Single message handling (either started with 1, or reduced to 1 after binary search)
if (remainingMessages.length === 1) {
  const singleMessage = remainingMessages[0];
  const singleMessageSize = getByteSize(JSON.stringify(singleMessage));

  // If single message fits, return it
  if (singleMessageSize <= maxBytes) {
    return remainingMessages;
  }

  // Single message is too large, try to truncate its content
  if (
    typeof singleMessage === 'object' &&
    singleMessage !== null &&
    'content' in singleMessage &&
    typeof (singleMessage as { content: unknown }).content === 'string'
  ) {
    const originalContent = (singleMessage as { content: string }).content;
    const messageWithoutContent = { ...singleMessage, content: '' };
    const otherMessagePartsSize = getByteSize(JSON.stringify(messageWithoutContent));
    const availableContentBytes = maxBytes - otherMessagePartsSize;

    if (availableContentBytes <= 0) {
      return [];
    }

    const truncatedContent = truncateStringByBytes(originalContent, availableContentBytes);
    return [{ ...singleMessage, content: truncatedContent }];
  } else {
    return [];
  }
}

// Multiple messages remain and fit within limit
return remainingMessages;
}

where truncateStringByBytes also does a quick binary search.