today I had the bright idea of using jina as a proxy to read pages Claude can't fetch for some reason but apparently fetch by Claude is blocked by robots.txt?

I thought the point of the project was to make the web more readable for agents, but seems like modern LLMs get caught in the overly broad robots.txt block jina reader has?

Just wondering if it's intentional.