feat(robots.txt): add guide for extending robots.txt functionality

.wordlist.txt

@@ -355,6 +355,8 @@ Enums
 EqualsAny
 ErrorsFacade
 EventListener
+EventListeners
+EventSubscriber
 EventSubscriberInterface
 Everytime
 ExampleAppWithCookies
@@ -424,6 +426,7 @@ GuestWishlistPage
 GuestWishlistPagelet
 HMAC
 HMR
+Hacktober
 HeaderPagelet
 Homebrew
 Hono
@@ -1852,6 +1855,7 @@ triggerDeprecated
 truthy
 tsx
 twigjs
+txt
 typeAndCheck
 typeAndCheckSearchField
 typeLegacySelectAndCheck

guides/plugins/plugins/content/seo/extend-robots-txt.md

@@ -0,0 +1,264 @@
+---
+nav:
+  title: Extend robots.txt configuration
+  position: 20
+
+---
+
+# Extend robots.txt configuration
+
+## Overview
+
+Since Shopware 6.7.1, the platform provides full `robots.txt` support with all standard directives and user-agent blocks.
+This feature was developed as an open-source contribution during Hacktober 2024 ([learn more](https://www.shopware.com/en/news/hacktoberfest-2024-outcome-a-robots-txt-for-shopware/)).
+For general configuration, refer to the [user documentation](https://docs.shopware.com/en/shopware-6-en/tutorials-and-faq/creation-of-robots-txt).
+
+::: info
+The events and features described in this guide are available since Shopware 6.7.5.
+:::
+
+You can extend the `robots.txt` functionality through events to:
+* Add custom validation rules during parsing
+* Modify or generate directives dynamically
+* Support custom or vendor-specific directives
+* Prevent warnings for known non-standard directives
+
+## Prerequisites
+
+This guide requires you to have a basic plugin running. If you don't know how to create a plugin, head over to the plugin base guide:
+
+<PageRef page="../../plugin-base-guide" />
+
+You should also be familiar with [Event listeners](../../plugin-fundamentals/listening-to-events).
+
+::: info
+This guide uses EventListeners since each example listens to a single event. If you need to subscribe to multiple events in the same class, consider using an [EventSubscriber](../../plugin-fundamentals/listening-to-events#listening-to-events-via-subscriber) instead.
+:::
+
+## Modifying parsed directives
+
+The `RobotsDirectiveParsingEvent` is dispatched after `robots.txt` content is parsed. You can modify the parsed result, add validation, or inject dynamic directives.
+
+This example shows how to dynamically add restrictions for AI crawlers:
+
+<Tabs>
+<Tab title="RobotsExtensionListener.php">
+
+```PHP
+<?php declare(strict_types=1);
+
+namespace Swag\Example\Listener;
+
+use Psr\Log\LoggerInterface;
+use Shopware\Core\Framework\Log\Package;
+use Shopware\Storefront\Page\Robots\Event\RobotsDirectiveParsingEvent;
+use Shopware\Storefront\Page\Robots\ValueObject\RobotsDirective;
+use Shopware\Storefront\Page\Robots\ValueObject\RobotsDirectiveType;
+use Shopware\Storefront\Page\Robots\ValueObject\RobotsUserAgentBlock;
+
+#[Package('storefront')]
+class RobotsExtensionListener
+{
+    public function __construct(
+        private readonly LoggerInterface $logger,
+    ) {
+    }
+
+    public function __invoke(RobotsDirectiveParsingEvent $event): void
+    {
+        $parsedRobots = $event->getParsedRobots();
+
+        // Add restrictions for AI crawlers
+        $aiCrawlers = ['GPTBot', 'ChatGPT-User', 'CCBot', 'anthropic-ai'];
+
+        $aiBlock = new RobotsUserAgentBlock(
+            userAgents: $aiCrawlers,
+            directives: [
+                new RobotsDirective(
+                    type: RobotsDirectiveType::DISALLOW,
+                    value: '/checkout/',
+                ),
+            ],
+        );
+
+        $parsedRobots->addUserAgentBlock($aiBlock);
+
+        $this->logger->info('Extended robots.txt with AI crawler rules');
+    }
+}
+```
+
+</Tab>
+
+<Tab title="services.xml">
+
+```XML
+<?xml version="1.0" ?>
+<container xmlns="http://symfony.com/schema/dic/services"
+           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+           xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">
+
+    <services>
+        <service id="Swag\Example\Listener\RobotsExtensionListener">
+            <argument type="service" id="logger"/>
+            <tag name="kernel.event_listener" event="Shopware\Storefront\Page\Robots\Event\RobotsDirectiveParsingEvent"/>
+        </service>
+    </services>
+</container>
+```
+
+</Tab>
+</Tabs>
+
+## Handling custom directives
+
+The `RobotsUnknownDirectiveEvent` is dispatched when an unknown directive is encountered. Use this to support vendor-specific directives or prevent warnings for known non-standard directives:
+
+<Tabs>
+<Tab title="CustomDirectiveListener.php">
+
+```PHP
+<?php declare(strict_types=1);
+
+namespace Swag\Example\Listener;
+
+use Shopware\Core\Framework\Log\Package;
+use Shopware\Storefront\Page\Robots\Event\RobotsUnknownDirectiveEvent;
+
+#[Package('storefront')]
+class CustomDirectiveListener
+{
+    public function __invoke(RobotsUnknownDirectiveEvent $event): void
+    {
+        // Support Google and Yandex specific directives
+        $knownCustomDirectives = ['noimageindex', 'noarchive', 'clean-param'];
+
+        if (in_array(strtolower($event->getDirectiveName()), $knownCustomDirectives, true)) {
+            $event->setHandled(true); // Prevent "unknown directive" warning
+        }
+    }
+}
+```
+
+</Tab>
+
+<Tab title="services.xml">
+
+```XML
+<?xml version="1.0" ?>
+<container xmlns="http://symfony.com/schema/dic/services"
+           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+           xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">
+
+    <services>
+        <service id="Swag\Example\Listener\CustomDirectiveListener">
+            <tag name="kernel.event_listener" event="Shopware\Storefront\Page\Robots\Event\RobotsUnknownDirectiveEvent"/>
+        </service>
+    </services>
+</container>
+```
+
+</Tab>
+</Tabs>
+
+## Validation and parse issues
+
+You can add validation warnings or errors during parsing using the `ParseIssue` class. This example shows common validation scenarios:
+
+<Tabs>
+<Tab title="RobotsValidationListener.php">
+
+```PHP
+<?php declare(strict_types=1);
+
+namespace Swag\Example\Listener;
+
+use Shopware\Core\Framework\Log\Package;
+use Shopware\Storefront\Page\Robots\Event\RobotsDirectiveParsingEvent;
+use Shopware\Storefront\Page\Robots\Parser\ParseIssue;
+use Shopware\Storefront\Page\Robots\Parser\ParseIssueSeverity;
+use Shopware\Storefront\Page\Robots\ValueObject\RobotsDirectiveType;
+
+#[Package('storefront')]
+class RobotsValidationListener
+{
+    public function __invoke(RobotsDirectiveParsingEvent $event): void
+    {
+        $parsedRobots = $event->getParsedRobots();
+
+        // Validate crawl-delay values
+        foreach ($parsedRobots->getUserAgentBlocks() as $block) {
+            foreach ($block->getDirectives() as $directive) {
+                if ($directive->getType() === RobotsDirectiveType::CRAWL_DELAY) {
+                    $value = (int) $directive->getValue();
+
+                    if ($value <= 0) {
+                        $event->addIssue(new ParseIssue(
+                            severity: ParseIssueSeverity::ERROR,
+                            message: 'Invalid crawl-delay value: must be a positive integer',
+                            lineNumber: null,
+                        ));
+                    }
+
+                    if ($value > 10) {
+                        $event->addIssue(new ParseIssue(
+                            severity: ParseIssueSeverity::WARNING,
+                            message: 'Crawl-delay value is very high. This may significantly slow down indexing.',
+                            lineNumber: null,
+                        ));
+                    }
+                }
+            }
+        }
+
+        // Check for conflicting Allow/Disallow directives
+        foreach ($parsedRobots->getUserAgentBlocks() as $block) {
+            $disallowed = [];
+            $allowed = [];
+
+            foreach ($block->getDirectives() as $directive) {
+                if ($directive->getType() === RobotsDirectiveType::DISALLOW) {
+                    $disallowed[] = $directive->getValue();
+                } elseif ($directive->getType() === RobotsDirectiveType::ALLOW) {
+                    $allowed[] = $directive->getValue();
+                }
+            }
+
+            foreach ($allowed as $allowPath) {
+                foreach ($disallowed as $disallowPath) {
+                    if ($allowPath === $disallowPath) {
+                        $event->addIssue(new ParseIssue(
+                            severity: ParseIssueSeverity::WARNING,
+                            message: sprintf('Conflicting directives: Path "%s" is both allowed and disallowed', $allowPath),
+                            lineNumber: null,
+                        ));
+                    }
+                }
+            }
+        }
+    }
+}
+```
+
+</Tab>
+
+<Tab title="services.xml">
+
+```XML
+<?xml version="1.0" ?>
+<container xmlns="http://symfony.com/schema/dic/services"
+           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+           xsi:schemaLocation="http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd">
+
+    <services>
+        <service id="Swag\Example\Listener\RobotsValidationListener">
+            <tag name="kernel.event_listener" event="Shopware\Storefront\Page\Robots\Event\RobotsDirectiveParsingEvent"/>
+        </service>
+    </services>
+</container>
+```
+
+</Tab>
+</Tabs>
+
+Issues are automatically logged when the `robots.txt` configuration is saved in the Administration. Use `WARNING` for recommendations and `ERROR` for critical problems that prevent proper generation.

guides/plugins/plugins/content/seo/index.md

@@ -7,6 +7,16 @@ nav:
 
 # SEO
 
-The Shopware SEO feature offers the ability to add custom SEO URLs to optimize the search engine visibility of the e-commerce platform.
+The Shopware SEO feature offers comprehensive tools to optimize the search engine visibility of your e-commerce platform.
 
-With the SEO plugin, you can create custom SEO URLs for product pages, categories, content pages, and other relevant sections of the website. The plugin allows businesses to customize meta tags for each page, including meta titles, descriptions, and keywords. This enables users to optimize the on-page SEO elements, providing search engines with relevant information about the content of the page.
+## SEO URLs
+
+You can create custom SEO URLs for product pages, categories, content pages, and other relevant sections of the website. The plugin allows businesses to customize meta tags for each page, including meta titles, descriptions, and keywords. This enables users to optimize the on-page SEO elements, providing search engines with relevant information about the content of the page.
+
+<PageRef page="add-custom-seo-url" title="Add custom SEO URLs" />
+
+## Robots configuration
+
+Shopware provides full support for `robots.txt` configuration, including all standard directives, user-agent blocks, and extensibility through events. You can customize how search engine crawlers interact with your shop by extending the `robots.txt` parsing and generation.
+
+<PageRef page="extend-robots-txt" title="Extend robots configuration" />

guides/plugins/plugins/framework/data-handling/add-data-translations.md

@@ -297,3 +297,4 @@ class ExampleDefinition extends EntityDefinition
         ]);
     }
 }
+```