-
Notifications
You must be signed in to change notification settings - Fork 48
Open
Labels
local setupImprovements or additions to the local setupImprovements or additions to the local setup
Description
Basically, if it's not the live site, we need a robots.txt at the root that contains:
"User-Agent: *
Disallow: /"
Rationale: We only want bots to index the main site, https://frameworks.securityalliance.org/, not the dev site(s). It's confusing to users when dev sites show up in the results, and it dilutes pagerank across one or more very similar work-in-progress sites.
You can do some better optimization of this with rel=canonical but this is a simple start.
For the public frameworks site, I'd use the same robots.txt as the main site https://securityalliance.org/robots.txt
Not sure how you do conditional includes in this framework.
Metadata
Metadata
Assignees
Labels
local setupImprovements or additions to the local setupImprovements or additions to the local setup