Skip to content

bug: robots.txt for dev site. #307

@smagdali

Description

@smagdali

Basically, if it's not the live site, we need a robots.txt at the root that contains:

"User-Agent: *
Disallow: /"

Rationale: We only want bots to index the main site, https://frameworks.securityalliance.org/, not the dev site(s). It's confusing to users when dev sites show up in the results, and it dilutes pagerank across one or more very similar work-in-progress sites.

You can do some better optimization of this with rel=canonical but this is a simple start.

For the public frameworks site, I'd use the same robots.txt as the main site https://securityalliance.org/robots.txt

Not sure how you do conditional includes in this framework.

Metadata

Metadata

Assignees

No one assigned

    Labels

    local setupImprovements or additions to the local setup

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions