sitemap.xml.gz

[MJomaa]

Hi :)

First of all, thank you for this project.
I didn't see any route that handles "/sitemap.xml.gz" in the MVC6 boilerplate.
Would be cool to add that :)
Example Implementation:

        [ResponseCache(CacheProfileName = CacheProfileName.SitemapNodes)]
        [Route("sitemap.xml.gz", Name = HomeControllerRoute.GetSitemapXmlGz)]
        public async Task<IActionResult> SitemapXmlGz(int? index = null)
        {
            var content = await _sitemapService.GetSitemapXml(index);

            if (content == null)
            {
                return BadRequest("Sitemap index is out of range.");
            }

            var uncompressed = new MemoryStream(Encoding.UTF8.GetBytes(content));
            byte[] compressedBytes;

            using (var compressed = new MemoryStream())
            {
                using (var compressor = new GZipStream(compressed, CompressionMode.Compress))
                {
                    await uncompressed.CopyToAsync(compressor);
                }

                compressedBytes = compressed.ToArray();
            }

            Response?.Headers?.Add("Content-Encoding", "gzip");
            Response?.Headers?.Add("Content-Length", compressedBtes?.Length.ToString());

            return new FileContentResult(compressedBytes, "application/x-gzip-compressed");
        }

[RehanSaeed]

This is an interesting issue. Requesting \sitemap.xml with a Accept-Encoding: gzip HTTP header should return the same result as \sitemap.xml.gz if you enable GZIP compression in IIS or NGINX (Which boilerplate does).

After some reading \sitemap.xml has wider support among bots but due to the closed nature of search engine and social website bots, we can't be sure if they all support \sitemap.xml.gz (They probably do as it's a standard). However, these bots are probably already sending the Accept-Encoding: gzip` HTTP header, so what benefit does this new action give us if they are getting a GZIP'ed sitemap using a header rather than a URL?

[MJomaa]

Ah I didn't know.
I guess it's for the case when a (bad?) robot is explicitly requesting \sitemap.xml.gz.

There are many sites with kinda bad content .. basicly what their bot crawls and spits out. However those sites are indexed by google and often under the first result site. If their content contains your links then google will have more links to your site.

To be clear, I don't want to replace \sitemap.xml, just add another route \sitemap.xml.gz. (So you can request the gzipped file via HTTP Header or Route)

[RehanSaeed]

I understand that we could support both but the sitemap is linked to in the robots.txt file, so in reality only one is used unless someone specifically visits that URL. You have the choice of linking to a GZIP'ed or non-GZIP'ed version in robots.txt.

One easy way to see what to do would be to look at the robots.txt from the top 100 websites and see what they are doing. I want to understand if it's worth the extra code in the thousands of projects created from this template. It's only worth it if it will be used.

[MJomaa]

Google and Microsoft are using multiple sitemaps.
Sitemap: [MyDomain]/sitemap.xml

However neither are linking the .gz version.
My suggestion would be do include the sitemap.xml.gz route, but not linking it in the robots.txt.

I mean.. what's the matter? They are both using the same SitemapService.

[RehanSaeed]

Agreed that gz should not be the default in robots.txt but then if it's not there I don't see any use cases for it's use unless I'm missing something. I don't want to add code that is not going to be used, it adds bloat and is more to maintain etc.

[MJomaa]

If you think that it is not gonna be used that often to be an valuable addition, then I'm fine with it.

[RehanSaeed]

Solid issue, I was not aware this thing even existed.