Skip to content

Commit 0f33e59

Browse files
committed
update user-agent docs
1 parent 925990e commit 0f33e59

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

docs/how_to_add_a_publisher.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ will exclude all sitemap URLs not containing the substring `sitemap-content-`.
257257
### Finishing the Publisher Specification
258258
259259
1. If your publisher requires to use custom request headers to work properly you can alter it by using the `request_header` parameter of `PublisherSpec`.
260-
The default is: `{"user_agent": "Fundus"}`.
260+
The default is: `{"user-agent": "Fundus/2.0 (contact: github.com/flairnlp/fundus)"}`.
261261
2. If you want to block URLs for the entire publisher use the `url_filter` parameter of `Publisher`.
262262
3. In some cases it can be necessary to append query parameters to the end of the URL, e.g. to load the article as one page. This can be achieved by adding the `query_parameter` attribute of `PublisherSpec` and assigning it a dictionary object containing the key - value pairs: e.g. `{"page": "all"}`. These key - value pairs will be appended to all crawled URLs.
263263

0 commit comments

Comments
 (0)