feat: bot detection (#210)

Author: harlan-zw

docs/content/2.guides/4.bot-detection.md

@@ -0,0 +1,149 @@
+---
+title: Bot Detection
+description: Detect and classify bots with server-side header analysis and client-side browser fingerprinting.
+---
+
+## Introduction
+
+The modern web is full of [bots](https://nuxtseo.com/learn/controlling-crawlers). Start detecting them to better understand your traffic and optimize your Nuxt app for both human users and automated agents.
+
+Nuxt Robots provides bot detection that works on both server and client side, from simple heuristics using HTTP header `User-Agent` checks to advanced client-side detection using [BotD](https://github.com/fingerprintjs/BotD).
+
+## Bot Categories
+
+The module classifies bots into categories based on their intended purpose and trustworthiness. **Trusted bots** are legitimate services that respect robots.txt and provide value to websites, while **untrusted bots** include automation tools, scrapers, and potentially malicious crawlers.
+
+### Search Engine Bots (trusted)
+Search engines that index content for public search results and respect standard web protocols.
+- **Google**: `googlebot`, `google.com/bot.html`
+- **Bing**: `bingbot`, `msnbot`
+
+### Social Media Bots (trusted)
+Social platforms that crawl content for link previews, cards, and social sharing features.
+- **Facebook**: `facebookexternalhit`, `facebook.com`
+- **Twitter**: `twitterbot`, `twitter`
+
+### SEO & Analytics Bots (trusted)
+Professional SEO and analytics services that provide legitimate website analysis and insights.
+- **Ahrefs**: `ahrefsbot`, `ahrefs.com`
+- **Majestic**: `mj12bot`, `majestic12.co.uk/bot`
+
+### AI & ML Bots (trusted)
+AI companies and research organizations training models or providing AI-powered services.
+- **OpenAI**: `gptbot`, `openai.com`
+- **Anthropic**: `anthropic`
+
+### Automation Tools (untrusted)
+Browser automation and testing frameworks that may be used for legitimate testing or malicious scraping.
+- **Selenium**: `selenium`, `webdriver`
+- **Playwright**: `playwright`
+
+### HTTP Tools (untrusted)
+Command-line HTTP clients and programmatic request libraries often used for automated data extraction.
+- **cURL**: `curl`
+- **Python Requests**: `python-requests`, `python`
+
+### Security Scanners (untrusted)
+Network scanning and vulnerability assessment tools that may indicate malicious reconnaissance.
+- **Nmap**: `nmap`, `insecure.org`
+- **Nikto**: `nikto`
+
+### Scraping Tools (untrusted)
+Dedicated web scraping frameworks designed for automated data collection.
+- **Scrapy**: `scrapy`, `scrapy.org`
+- **Generic Scraper**: `scraper`
+
+Missing a bot? Submit a quick PR :)
+[View and contribute to bot definitions →](https://github.com/nuxt-modules/robots/blob/main/src/const-bots.ts)
+
+## Nitro Bot Detection
+
+Since server-side detection only uses HTTP headers, detection can only work for bots that correctly identify themselves in the `User-Agent` header.
+
+You can detect bots inside a Nitro route, middleware, or API handler.
+
+```ts
+import { getBotDetection } from '#robots/server/composables/getBotDetection'
+
+export default defineEventHandler((e) => {
+  const detection = getBotDetection(e)
+
+  if (detection.isBot) {
+    return { message: `${detection.botName} bot detected`, category: detection.botCategory }
+  }
+
+  return { message: 'Human user' }
+})
+```
+
+For full behavior, please consult the [`getBotDetection`](/docs/robots/nitro-api/get-bot-detection) API docs.
+
+## Nuxt Bot Detection
+
+When using bot detection in Nuxt, it will use the `User-Agent` header by default. You can optionally use the [BotD](https://github.com/fingerprintjs/BotD) fingerprinting library to detect advanced automation tools by setting `fingerprint: true`.
+
+```vue
+<script setup lang="ts">
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+const { isBot, botName, botCategory, trusted } = useBotDetection({
+  fingerprint: true, // detects using botd
+})
+</script>
+
+<template>
+  <div v-if="isBot">
+    Bot detected: {{ botName }} ({{ botCategory }})
+  </div>
+</template>
+```
+
+See the [`useBotDetection()`](/docs/robots/api/use-bot-detection) API docs for full usage details.
+
+## Fingerprinting with BotD
+
+When using `fingerprint: true`, the composable will load the [BotD](https://github.com/fingerprintjs/BotD)
+library when the window is idle and perform client-side fingerprinting to detect advanced bots and automation tools.
+
+### Performance Considerations
+
+This fingerprinting is computationally expensive for end users' CPUs, so you should be mindful of when you enable it. For example, you may consider only enabling it for sensitive pages where bot detection is critical.
+
+That said, the composable aims to be performant and will cache the bot result in the user's local storage under the `'__nuxt_robots:botd'` key so it will only run once.
+
+```ts
+localStorage.getItem('__nuxt_robots:botd') // returns the cached bot detection result - used internally already
+```
+
+### Watching For Fingerprinting
+
+The properties returned from the composable are all `ref`s. It's important to watch these for changes if you're using fingerprinting, as the results will not be immediately available when the composable is called.
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+import { watch } from 'vue'
+
+const { isBot } = useBotDetection({
+  fingerprint: true,
+})
+
+watch(isBot, (detected) => {
+  if (detected) {
+    console.log(`Bot detected!`)
+  }
+})
+```
+
+Alternatively you can use the `onFingerprintResult` callback to handle the result when fingerprinting completes.
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+const botd = useBotDetection({
+  fingerprint: true,
+  onFingerprintResult(result) {
+    // Fingerprinting completed
+    console.log('Detection result:', result)
+  },
+})
+```

docs/content/3.api/2.use-bot-detection.md

@@ -0,0 +1,200 @@
+---
+title: useBotDetection()
+description: A reactive composable for detecting and classifying bots with optional client-side fingerprinting.
+---
+
+## Introduction
+
+**Type:** `function useBotDetection(options?: UseBotDetectionOptions): UseBotDetectionReturn`{lang="ts"}
+
+```ts
+import type { UseBotDetectionOptions, UseBotDetectionReturn } from '@nuxtjs/robots/util'
+```
+
+Detect and classify bots using server-side header analysis and optional client-side browser fingerprinting.
+
+The composable provides reactive access to bot detection results with automatic caching. Client-side fingerprinting is opt-in due to performance costs.
+
+**🔔 Important:** Bot detection only runs when you use this composable. No automatic bot detection occurs - it's entirely opt-in based on your usage of this composable.
+
+## Usage
+
+**Basic detection:**
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+const { isBot, botName, botCategory, trusted } = useBotDetection()
+// isBot: ComputedRef<boolean>
+// botName: ComputedRef<BotName | undefined> // 'googlebot', 'facebook', etc.
+// botCategory: ComputedRef<BotCategory | undefined> // 'search-engine', 'social', etc.
+// trusted: ComputedRef<boolean | undefined>
+```
+
+**With fingerprinting:**
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+const { isBot, botName, botCategory, trusted, reset } = useBotDetection({
+  fingerprint: true,
+  onFingerprintError: (error) => {
+    console.error('Fingerprint error:', error)
+  },
+  onFingerprintResult: (result) => {
+    console.log('Fingerprinting completed:', result)
+  }
+})
+```
+
+**Watching for changes:**
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+import { watch } from 'vue'
+
+const { isBot, botName, botCategory } = useBotDetection()
+
+watch(isBot, (detected) => {
+  if (detected) {
+    console.log(`Bot: ${botName.value} (${botCategory.value})`)
+  }
+})
+```
+
+## Options
+
+```ts
+interface UseBotDetectionOptions {
+  fingerprint?: boolean
+  onFingerprintError?: (error: Error) => void
+  onFingerprintResult?: (result: BotDetectionContext | null) => void
+}
+```
+
+### `fingerprint`
+
+**Type:** `boolean`
+**Default:** `false`
+
+Enable automatic client-side fingerprinting when no bot is detected server-side.
+
+### `onFingerprintError`
+
+**Type:** `(error: Error) => void`
+
+Error handler for fingerprinting failures.
+
+### `onFingerprintResult`
+
+**Type:** `(result: BotDetectionContext | null) => void`
+
+Callback that fires when fingerprinting completes, providing the final detection result.
+
+## Return Type
+
+```ts
+interface UseBotDetectionReturn {
+  isBot: ComputedRef<boolean>
+  botName: ComputedRef<BotName | undefined>
+  botCategory: ComputedRef<BotCategory | undefined>
+  trusted: ComputedRef<boolean | undefined>
+  reset: () => void
+}
+```
+
+## Return Value
+
+### `isBot`
+
+**Type:** `ComputedRef<boolean>`
+
+Reactive boolean indicating whether a bot was detected.
+
+### `botName`
+
+**Type:** `ComputedRef<BotName | undefined>`
+
+The specific bot identity (e.g., 'googlebot', 'facebook', 'claude', 'selenium'). `undefined` if no bot detected.
+
+### `botCategory`
+
+**Type:** `ComputedRef<BotCategory | undefined>`
+
+The bot category/purpose (e.g., 'search-engine', 'social', 'ai', 'automation'). `undefined` if no bot detected.
+
+### `trusted`
+
+**Type:** `ComputedRef<boolean | undefined>`
+
+Whether the detected bot is considered trusted. `undefined` if no bot detected.
+
+### `reset()`
+
+**Type:** `() => void`
+
+Clear all detection state and cached results.
+
+## Server Side Behavior
+
+On the server, bot detection runs when you use the composable:
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+// Only runs when composable is used
+const { isBot, botName, botCategory } = useBotDetection()
+
+if (isBot.value) {
+  // Bot detected via server-side analysis
+  console.log('Bot:', botName.value, 'Category:', botCategory.value)
+}
+```
+
+## Client Side Behavior
+
+Client-side fingerprinting is automatic when enabled:
+
+```ts
+import { useBotDetection } from '#robots/app/composables/useBotDetection'
+
+const { isBot, botName, botCategory } = useBotDetection({
+  fingerprint: true,
+  onFingerprintError: (error) => {
+    console.error('Fingerprinting failed:', error)
+  }
+})
+
+// Fingerprinting runs automatically if no server detection occurred
+```
+
+## Configuration
+
+### Disabling Bot Detection
+
+You can disable the entire bot detection plugin:
+
+```ts
+// nuxt.config.ts
+import { defineNuxtConfig } from 'nuxt/config'
+
+export default defineNuxtConfig({
+  robots: {
+    botDetection: false
+  }
+})
+```
+
+When disabled, the `useBotDetection` composable will not be available.
+
+## Bot Categories
+
+The following bot types are detected:
+
+- **search-engine**: Google, Bing, Yandex crawlers
+- **social**: Twitter, Facebook, LinkedIn bots
+- **seo**: Ahrefs, SEMrush, Majestic tools
+- **ai**: GPT, Claude, Perplexity crawlers
+- **automation**: Selenium, Puppeteer, WebDriver
+- **security-scanner**: nmap, Nikto, ZGrab
+- **http-tool**: curl, wget, Python requests