"403 Client Error: Forbidden for url" When trying popular websites

[thehorse2000]

Most of the time, if I am applying to a popular website blog, I get a 403 Error.
You can replicate by trying to generate a podcast from this OpenAI blog post: https:/openai.com/index/introducing-upgrades-to-codex.
I believe this issue is because the package is getting the website content using "request", most popular websites have some sort of bot detection and blocking, hence the 403 Error.

How to reproduce:

1. Use the Gradio app to reproduce (https://thatupiso-podcastfy-ai-demo.hf.space/)
2. Use this OpenAI blog post URL: https:/openai.com/index/introducing-upgrades-to-codex.
3. You get this error in the response

{
    "detail": "File not allowed: /home/user/app/Failed to extract content from https:/openai.com/index/introducing-upgrades-to-codex/: 403 Client Error: Forbidden for url: https:/openai.com/index/introducing-upgrades-to-codex."
}

[thehorse2000]

I suggest using Playwright to fetch the website's HTML instead of just making a normal request, this will resolve the issue