CDN Hosting at Google Cloud Storage

Note: these are notes about using Google Cloud Storage as a CDN service. We use GCS in combination with CloudFlare which means we can get away with some things..

Preliminaries

• The documentation home is https://developers.google.com/storage/index.
• The main tool is gsutil.
• Note: due to rapid development a lot of information on the web uses outdated syntax (but is still useful, e.g., https://gist.github.com/mhulse/4362104).

Copying

Notes:

• gsutil option: -m multithreaded (higher load+cost but faster)

• cp options: recursively (-R), compressed (-z), verbose (-v) https://developers.google.com/storage/docs/gsutil/commands/cp

• Note: compressed is necessary to get gzipped delivery. Google Cloud storage does not compress on the fly but should decompress on the fly for old browsers that can't accept decompressed files.

    gsutil -m cp -Rzv mathjax gs://our-bucket/
  

This also works within GCS, i.e., bucket to bucket.

Note that metadata isn't preserved by default (public access, caching, custom headers etc); use -p option to preserve them (warning: this incurs higher costs).

gsutil -m cp -Rp gs://bucket1/file1 gs://bucket2/

(You'll still need to set the CORS for each bucket though; see below.)

make files public

Notes:

• https://developers.google.com/storage/docs/gsutil/commands/acl#set

This has to be done whenever files are uploaded (TODO: automate!)

    gsutil -m acl set -R -a public-read gs://our-bucket/path/to/file_or_wildcard

set CORS

Notes:

• should only be needed once per bucket
• https://developers.google.com/storage/docs/cross-origin

Create an XML file with

<?xml version="1.0" encoding="UTF-8"?>
<CorsConfig>
  <Cors>
        <Origins>
            <Origin>*</Origin>
        </Origins>
        <Methods>
            <Method>GET</Method>
            <Method>POST</Method>
            <Method>HEAD</Method>
        </Methods>
        <ResponseHeaders>
            <ResponseHeader>*</ResponseHeader>
        </ResponseHeaders>
        <MaxAgeSec>86400</MaxAgeSec>
    </Cors>
</CorsConfig>

Save as cors.xml and run

gsutil cors set cors.xml gs://our-bucket

mime-type headers

Notes:

• https://developers.google.com/storage/docs/gsutil/commands/setmeta

• the ** wildcard includes subdirectories, https://developers.google.com/storage/docs/gsutil/addlhelp/WildcardNames

• EOT and SVG content-type headers are detected correctly during upload to GCS.

• WOFF is set to the old mime type x-font/woff; that works but we decided to go with the future-proof font-woff. Either change it as below or see advice here to do this automatically on upload.

• For otf:

    gsutil setmeta -r -h "Content-Type:font/opentype" gs://our-bucket/mathjax/**.otf
  

• For otf:

    gsutil setmeta -r -h "Content-Type:application/font-woff" gs://our-bucket/mathjax/**.woff
  

Caching

Note Since we are combining GCS with CloudFlare's caching, we will have short caching times on GCS.

This has to be repeated whenever files get uploaded from local.

    $ gsutil -m setmeta -r -h "Cache-Control:public, max-age=3600" gs://our-bucket/mathjax/

CNAME

To use a CNAME for a bucket, the bucket needs to be named like the CNAME, see https://developers.google.com/storage/docs/reference-uris#cname.

You also need to verify domain ownership; see https://support.google.com/webmasters/answer/35179?hl=en.

Testing

• CORS allow-origin [vital]
  • A simple test: switch mathjax.org over to the GCS origin (bucket.storage.googlapis.com) and use saucelabs to check rendering across IE, FF, Chrome, Safari.
  • Since CORS is set on bucket level, we should not have to worry about it.
• font mimetypes [non-vital]
  • TODO write script to curl -I all woff and otf files, compare header.

Logging

We don't log on GCS anymore since we combine it with CloudFlare (and get their stats).

For the record, instructions are at https://developers.google.com/storage/docs/gsutil/commands/logging