Prevent indexing of non-production environments by default

[Copilot]

Problem

Non-production environments (development, staging, local) were being indexed by search engines by default, which can cause:

• Duplicate content SEO penalties when both production and staging sites are indexed
• Confusion for users who land on staging/development sites via search engines
• Potential exposure of unreleased features or content

Solution

This PR adds a new allow_indexing configuration option to the SEO module that controls whether search engines can index the site. The default behavior is:

• Production environments: Indexing is allowed (default allow_indexing = true)
• Non-production environments: Indexing is blocked (default allow_indexing = false)

When indexing is disabled, the robots.txt file is automatically modified to include:

User-agent: *
Disallow: /

Implementation

Changes Made

1. load.php: Added allow_indexing to default settings that evaluates to true only for production environments using Altis\get_environment_type()

2. inc/namespace.php:

   • Added logic in bootstrap() to check the allow_indexing setting
   • When allow_indexing is false, adds a robots_txt filter to disallow indexing
   • Added new disallow_indexing() function to modify the robots.txt output

3. docs/README.md:

   • Added documentation for the new allow_indexing configuration option
   • Fixed JSON formatting issue (missing comma)
   • Added "Configuration Options" section explaining all settings

Configuration Override

Projects that need to allow indexing in non-production environments can override this setting in their composer.json:

{
    "extra": {
        "altis": {
            "modules": {
                "seo": {
                    "allow_indexing": true
                }
            }
        }
    }
}

Benefits

• Secure by default: prevents accidental indexing of non-production sites
• Protects against SEO duplicate content issues
• Fully configurable per project
• Backward compatible (uses null coalescing operator to default to true if not set)
• Minimal performance impact (filter only added when needed)

Fixes https://github.com/humanmade/terraform-app-stack/issues/1484

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Non-production environments should not allow indexing by default</issue_title>
<issue_description>Suggested fix

$default_config['seo']['index'] = Altis\get_environment_type() === 'production';
// ...
if ( get_config()['seo']['index'] ?? false ) {
    add_filter( 'robots_txt', function ( $text ) {
        $text .= "User-agent: *\nDisallow: /\n";
        return $text;
    } );
}

Came from: https://github.com/humanmade/terraform-app-stack/issues/1484</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #370

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[rmccue]

@copilot Let's rename the configuration item to allow_indexing instead.

[Copilot]

@copilot Let's rename the configuration item to allow_indexing instead.

Renamed the configuration from index to allow_indexing in all files (load.php, inc/namespace.php, and docs/README.md). Commit: 5f9a82b