robots.txt - which generated directories should have a Disallow rule?

[mbifulco]

Hi there,

I've been doing some inspection of my site (mikebifulco.com's SEO using the ahrefs webmaster tools to occasionally run audits on my site. In doing so, I noticed that their crawler is checking the _next directory on my site:

My understanding is that I probably don't want /_next and /api to be indexed for search, but I haven't found any real guidelines in the docs for next on this. (this is all I found: static file serving and "what is a robots.txt file?"

Also, I checked robots.txt for some sites that I thought might show recommended practices... and was a bit surprised.

vercel.com

link

User-Agent: *
Disallow:
Disallow: /api
Disallow: /oauth
Disallow: /confirm
Disallow: /notifications
Disallow: /old-browser.html

nextjs.org

link

(404 page)

mikebifulco.com (for comparison)

link

# *
User-agent: *
Allow: /

# Host
Host: https://mikebifulco.com

# Sitemaps
Sitemap: https://mikebifulco.com/sitemap.xml

should I add a disallow rule for crawlers on _next? What about /api, presuming I don't want any of my APIs to show up in search results? Nothing special there, but there's no reason anyone would ever want to browse directly to one of those routes.

[vpaflah]

Did you find anything useful?

[HamedMP]

Same question here, Claude 3.5 recommends having _vercel but not sure if that will impact showing the images on the search results

[olaj]

I'm also curious, the AI's (Chat GPT, Gemeni) seems super confidence in that you should block the entire _next folder. But since it also blocks CSS file it seems it can't render the pages (i.e if you do a live test thing in GSC). Like they talk about here.

https://webmasters.stackexchange.com/questions/137447/should-i-block-the-next-directory-in-a-site-using-next-js-with-robots-txt

Maybe it's ok to block all but the .css files or something?

[fawredd]

Same question.