Watcher fails to retrieve information from several sources

[ludoComp9]

Describe the bug
Requests failed due to user-agent

To Reproduce
Steps to reproduce the behavior:

1. See watcher logs.
   Example:
   watcher | WARNING 2026-02-03 13:30:54,296 | Feed: https://www.bleepingcomputer.com/feed/ => Error: Status code: 403
   watcher | WARNING 2026-02-03 13:30:55,027 | Feed: https://www.phishlabs.com/feed/ => Error: Status code: 403

Expected behavior
From watcher service, I can see wget https://www.bleepingcomputer.com/feed/is forbidden (error code 403)

wget https://www.bleepingcomputer.com/
--2026-02-03 13:38:00--  https://www.bleepingcomputer.com/
Resolving www.bleepingcomputer.com (www.bleepingcomputer.com)... 172.66.135.165, 172.66.139.132
Connecting to www.bleepingcomputer.com (www.bleepingcomputer.com)|172.66.135.165|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2026-02-03 13:38:01 ERROR 403: Forbidden.

But, with a user-agent defined, it works:

get --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://www.bleepingcomputer.com/
--2026-02-03 13:37:39--  https://www.bleepingcomputer.com/
Resolving www.bleepingcomputer.com (www.bleepingcomputer.com)... 172.66.135.165, 172.66.139.132
Connecting to www.bleepingcomputer.com (www.bleepingcomputer.com)|172.66.135.165|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                       [ <=>                                                                                        ] 113.61K  --.-KB/s    in 0.09s   

2026-02-03 13:37:39 (1.17 MB/s) - ‘index.html’ saved [116334]

Desktop (please complete the following information):

• OS: Debian 12 bookworm
• Watcher version: 3.2.2
• Watcher mode: docker

Additional context
How we could specify a user agent in remote requests to hope to retrieve information from all / almost all remote sites ?

Regards,

[segunakinfenwa-sketch]

Good day @ludoComp9
Can I jump on this?

[ygalnezri]

Hi @ludoComp9,

Thanks for the report and for the detailed reproduction steps, very helpful for me.

Some remote sources return a 403 Forbidden when Watcher tries to fetch feeds without a proper User-Agent. As you noticed with wget, several websites block default clients and expect requests that look more like a regular browser.

We updated core.py to make remote requests more reliable. Watcher now sends browser-like headers including User-Agent and Accept-Language for both RSS feed fetching and redirect resolution. SSL handling has also been adjusted to prevent failures on endpoints with stricter configurations.

Most feeds should now be retrieved correctly. For example, the feed you mentioned, https://www.bleepingcomputer.com/feed/, now works properly in Watcher. Please note that some websites may still return 403 due to advanced bot protections like Cloudflare, this is expected and cannot always be bypassed with headers only...

These changes will be included in the upcoming Watcher 3.2.3 release, which also addresses several other bugs detected in previous versions to improve overall stability.

@segunakinfenwa-sketch, as you asked, this should keep you updated on the fix.

Thanks again for reporting this, it helps us make Watcher more robust.

Kind regards,
Ygal

[ygalnezri]

Hi @ludoComp9, Hi @segunakinfenwa-sketch,

Thanks for the detailed report and reproduction steps — very helpful.

You identified the issue correctly: some remote sources return a 403 Forbidden when requests are sent without a proper User-Agent, as many websites now block default HTTP clients.

This has been fully fixed in the latest release v3.3.0:
https://github.com/thalesgroup-cert/Watcher/releases/tag/v3.3.0

Watcher now sends browser-like headers (including User-Agent and Accept-Language) and improves SSL handling, making feed retrieval significantly more reliable. The feed you mentioned should now work as expected, although some sites may still block requests due to advanced bot protections.

Please pull the latest Docker image and follow the update steps in the release notes, this should resolve the behavior entirely.

Thanks again for your report; feedback like this helps us keep Watcher robust.

Kind regards,
Ygal