Refactor robots.txt generation logic

[john-gitbook]

Changes robots.txt logic to block search and ask query urls from being indexed but allow icons to be indexed. I checked with Claude and Chat GPT and both recommended these changes.

Chat gpt link:
https://chatgpt.com/share/e/68b85180-8be0-800a-a531-b57aa06e58e8

[changeset-bot]

⚠️ No Changeset found

Latest commit: b8c6692

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

Summary of the deployments:

Version	URL	Status
Vercel	https://gitbook-v2-miezadvqd-gitbook.vercel.app	✅
Cloudflare	https://3fd43286-gitbook-open-v2-preview.gitbook.workers.dev	✅

Test content

Site	2v	2c
GitBook	https://gitbook-v2-miezadvqd-gitbook.vercel.app/url/gitbook.com/docs	https://3fd43286-gitbook-open-v2-preview.gitbook.workers.dev/url/gitbook.com/docs
E2E	https://gitbook-v2-miezadvqd-gitbook.vercel.app/url/gitbook.gitbook.io/test-gitbook-open	https://3fd43286-gitbook-open-v2-preview.gitbook.workers.dev/url/gitbook.gitbook.io/test-gitbook-open

[argos-ci]

The latest updates on your projects. Learn more about Argos notifications ↗︎

Build	Status	Details	Updated (UTC)
customers-v2 (Inspect)	✅ No changes detected	-	Sep 3, 2025, 2:56 PM
v2-cloudflare (Inspect)	✅ No changes detected	-	Sep 3, 2025, 3:01 PM
v2-vercel (Inspect)	⚠️ Changes detected (Review)	2 changed	Sep 3, 2025, 2:59 PM