Nested(multiple) crawling policy?

[flrngel]

Hi, I'm recently working on a transform/sitemap's sitemapindex protocol support.

For example, CNN's Sitemap Index is the case. This index contains multiple sitemaps. To build RSS of sitemaps' URLs, it needs to crawl the multiple URLs again.

I'm not sure if this is okay or not. Do you have any policy for this kind of pattern?

My current WIP code is here.

[TonyRL]

It will be fine as long as the target site doesn't block you.

[flrngel]

Great. Will PR soon.