Support for setting priority using regular expressions

[Koen1999]

Related to issue #97, I think it could be highly beneficial to be able to prioritize certain pages over others. Specifically, to prioritize manually written documentation over automatically generated documentation.

Currently the automatically generated documentation may show up first, which is not desirable from a user perspective.

I think having something like a dictionary in the configuration that allows you to map URL patterns to a priority should do the trick. In case multiple patterns match the same URL, we can just use the longest pattern.

URL_PRIORITY = {
    "_modules.*": 0.2,
    "*": 0.5,
}

@jdillard , let me know if you need help implementing something like this.

[jdillard]

That sounds like a good addition if you want to give it a shot.

Here are the docs on the <priority> tag:

The priority of this URL relative to other URLs on your site. Valid values range from 0.0 to 1.0. This value does not affect how your pages are compared to pages on other sites—it only lets the search engines know which pages you deem most important for the crawlers.

The default priority of a page is 0.5.

Please note that the priority you assign to a page is not likely to influence the position of your URLs in a search engine's result pages. Search engines may use this information when selecting between URLs on the same site, so you can use this tag to increase the likelihood that your most important pages are present in a search index.

Also, please note that assigning a high priority to all of the URLs on your site is not likely to help you. Since the priority is relative, it is only used to select between URLs on your site.

Note that the valid range is 0.0 to 1.0.

[Koen1999]

@jdillard , Happy to hear you like the proposal. I'm a bit occupied at the moment, but in a few weeks I will make time to work on this feature.

FYI: To me this feature is more important than #97, since assigning a low priority is a good alternative to completely preventing indexing of certain pages.