Enable sitemap generator and add robots.txt

[devksingh4]

Summary by CodeRabbit

• New Features
  • Implemented automatic sitemap generation to improve search engine discoverability and indexing.
  • Added robots.txt configuration to control search crawler access and exclude administrative areas from indexing.
  • Enhanced page metadata handling for improved search engine optimization.

[coderabbitai]

Warning

Rate limit exceeded

@devksingh4 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 24 minutes and 25 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

Walkthrough

This pull request adds sitemap generation functionality to the website. It introduces the @astrojs/sitemap package as a dependency, configures the Astro application with a site URL and sitemap integration that excludes admin routes, creates a robots.txt file with crawler directives and sitemap reference, and updates the HeadMetadata component to include a sitemap link reference while conditionally rendering description metadata.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Enable sitemap generator and add robots.txt' directly and accurately summarizes the main changes: adding sitemap integration and creating a robots.txt file.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch seo

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Deploying with Cloudflare Pages

Name	Result
Last commit:	058f993
Preview URL:	https://b0ce35f9.acmuiuc.pages.dev
Branch Preview URL:	https://seo.acmuiuc.pages.dev

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@public/robots.txt`:
- Line 4: Update the placeholder Sitemap URL in public/robots.txt so it uses the
actual site value from astro.config.mjs instead of
https://www.example.com/sitemap-index.xml; open astro.config.mjs, copy the
exported site value and replace the hardcoded domain in the Sitemap line (the
sitemap-index.xml path can remain) so the Sitemap entry dynamically reflects the
configured site domain.

In `@src/components/HeadMetadata.astro`:
- Line 16: In HeadMetadata.astro change the standard meta tag that renders the
page description so it uses name="description" instead of property="description"
(the conditional that currently renders {description && <meta
property="description" content={description} />}): update that JSX/ASTRO
fragment to emit the standard meta description tag while keeping the content
value and leave the Open Graph tag (og:description) unchanged.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Sitemap URL points to example.com instead of the actual site domain.

The sitemap URL is a placeholder. It should match the site value configured in astro.config.mjs.

🐛 Proposed fix

-Sitemap: https://www.example.com/sitemap-index.xml
+Sitemap: https://www.acm.illinois.edu/sitemap-index.xml

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	Sitemap: https://www.example.com/sitemap-index.xml
	Sitemap: https://www.acm.illinois.edu/sitemap-index.xml

🤖 Prompt for AI Agents

In `@public/robots.txt` at line 4, Update the placeholder Sitemap URL in
public/robots.txt so it uses the actual site value from astro.config.mjs instead
of https://www.example.com/sitemap-index.xml; open astro.config.mjs, copy the
exported site value and replace the hardcoded domain in the Sitemap line (the
sitemap-index.xml path can remain) so the Sitemap entry dynamically reflects the
configured site domain.