Skip to content

Commit 49cdf15

Browse files
authored
Merge pull request #817 from flairNLP/update-ua-docs
update user-agent docs
2 parents 93f495b + 0f33e59 commit 49cdf15

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

docs/how_to_add_a_publisher.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ will exclude all sitemap URLs not containing the substring `sitemap-content-`.
257257
### Finishing the Publisher Specification
258258
259259
1. If your publisher requires to use custom request headers to work properly you can alter it by using the `request_header` parameter of `PublisherSpec`.
260-
The default is: `{"user_agent": "Fundus"}`.
260+
The default is: `{"user-agent": "Fundus/2.0 (contact: github.com/flairnlp/fundus)"}`.
261261
2. If you want to block URLs for the entire publisher use the `url_filter` parameter of `Publisher`.
262262
3. In some cases it can be necessary to append query parameters to the end of the URL, e.g. to load the article as one page. This can be achieved by adding the `query_parameter` attribute of `PublisherSpec` and assigning it a dictionary object containing the key - value pairs: e.g. `{"page": "all"}`. These key - value pairs will be appended to all crawled URLs.
263263

0 commit comments

Comments
 (0)