HCS.Meta.Robots

Easily configurable Robots.txt defaulting to a deny all unless enabled. Perfect for when you have multiple environments.

Installation

Add the package to an existing Umbraco website (v10.4+) from nuget:

dotnet add package HCS.Meta.Robots

To enable robots.txt and prevent it from returning a deny all add the following to your Appsettings.json

"HCS": {
    "Meta": {
        "RobotsEnabled" : true,
        "RobotsEntries" : [],
        "RobotsAddToDefault" : false
    }
}

When enabled with an emtpy RobotsEntries array the response will be:

User-agent: *
Disallow: /app_data
Disallow: /app_plugins/
Disallow: /install
Disallow: /bin
Disallow: /umbraco/

Additional options

RobotsAddToDefault if set to true, the entries in RobotsEntries will be merged with the default response rather than replacing it note, they are merged AFTER the default.

Contributing

Contributions to this package are most welcome! Please read the Contributing Guidelines.

Acknowledgments

Lottie Pitcher for the Opinionated Starter kit for helping me get this package off the ground.