Merge pull request #460 from ampproject/master

Snapshot release v4

Author: banaag

.travis.yml

@@ -1,3 +1,15 @@
 # https://docs.travis-ci.com/user/languages/go/
 language: go
-go: ["1.10"]
+
+# Use ".x" to ask gimme to choose latest stable minor version of each Go
+# release: https://github.com/travis-ci/gimme
+go:
+  - 1.11.x
+  - 1.12.x
+  - 1.13.x
+  - 1.14.x
+
+env:
+  # Necessary for Go 1.11 and 1.12, so they'll respect the version constraints
+  # in go.mod.
+  - GO111MODULE=on

LICENSE

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2018 Google LLC
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -34,10 +34,29 @@ own and can obtain certificates for.
   1. Install Go version 1.10 or higher. Optionally, set
      [$GOPATH](https://github.com/golang/go/wiki/GOPATH) to something (default
      is `~/go`) and/or add `$GOPATH/bin` to `$PATH`.
-  2. `go get -u -mod=vendor github.com/ampproject/amppackager/cmd/amppkg`
-
-     Optionally, move the built `~/go/bin/amppkg` wherever you like.
-  3. Create a file `amppkg.toml`. A minimal config looks like this:
+  1. Get amppackager.
+
+     Check your Go version by running `go version`.
+  
+     For Go 1.14 and higher versions run:
+
+       ```
+       go get -u github.com/ampproject/amppackager/cmd/amppkg
+       ```
+     
+     For Go 1.13 and earlier versions run:
+
+       ```
+       go get -u -mod=vendor github.com/ampproject/amppackager/cmd/amppkg
+       ```
+
+  1. Optionally, move the built `~/go/bin/amppkg` wherever you like.
+  1. Prepare a temporary certificate and private key pair to use for signing the
+     exchange when testing your config. Follow WICG
+     [instructions](https://github.com/WICG/webpackage/tree/master/go/signedexchange#creating-our-first-signed-exchange)
+     to ensure compliance with the [WICG certificate
+     requirements](https://wicg.github.io/webpackage/draft-yasskin-httpbis-origin-signed-exchanges-impl.html#cross-origin-cert-req).
+  1. Create a file `amppkg.toml`. A minimal config looks like this:
      ```
      LocalOnly = true
      CertFile = 'path/to/fullchain.pem'
@@ -49,7 +68,7 @@ own and can obtain certificates for.
          Domain = "amppackageexample.com"
      ```
      More details can be found in [amppkg.example.toml](amppkg.example.toml).
-  4. `amppkg -development`
+  1. `amppkg -development`
 
      If `amppkg.toml` is not in the current working directory, pass
      `-config=/path/to/amppkg.toml`.
@@ -61,12 +80,17 @@ container.
 
 #### Test your config
 
-  1. Run Chrome with the following commandline flags:
+  1. Run Chrome with the following command line flags:
      ```
-     --user-data-dir=/tmp/udd
-     --ignore-certificate-errors-spki-list=$(openssl x509 -pubkey -noout -in path/to/fullchain.pem | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64)
-     --enable-features=SignedHTTPExchange
-     'data:text/html,<a href="https://localhost:8080/priv/doc/https://amppackageexample.com/">click me'
+     alias chrome = [FULL PATH TO CHROME BINARY]
+     PATH_TO_FULLCHAIN_PEM = [FULL PATH TO fullchain.pem]
+     chrome --user-data-dir=/tmp/udd\
+         --ignore-certificate-errors-spki-list=$(\
+            openssl x509 -pubkey -noout -in $PATH_TO_FULLCHAIN_PEM |\
+            openssl pkey -pubin -outform der |\
+            openssl dgst -sha256 -binary | base64)\
+         --enable-features=SignedHTTPExchange\
+            'data:text/html,<a href="https://localhost:8080/priv/doc/https://amppackageexample.com/">click me'
      ```
   2. Open DevTools. Check 'Preserve log'.
   3. Click the `click me` link.
@@ -148,13 +172,15 @@ You may also want to:
      before publication, or with a regular audit of a sample of documents. The
      [transforms](transformer/) are designed to work on valid AMP pages, and
      may break invalid AMP in small ways.
+  4. Setup
+     [monitoring](#monitoring-amppackager-in-production-via-its-prometheus-endpoints)
+     of `amppackager` and related requests to AMP document server.
 
 Once you've done the above, you should be able to test by launching Chrome
-without any comamndline flags; just make sure
-chrome://flags/#enable-signed-http-exchange is enabled. To test by visiting the
-packager URL directly, first add a Chrome extension to send an
-`AMP-Cache-Transform: any` request header. Otherwise, follow the above
-"Demonstrate privacy-preserving prefetch" instructions.
+without any command line flags. To test by visiting the packager URL directly,
+first add a Chrome extension to send an `AMP-Cache-Transform: any` request
+header. Otherwise, follow the above "Demonstrate privacy-preserving prefetch"
+instructions.
 
 ##### Security Considerations
 
@@ -175,8 +201,9 @@ that:
 
 It is possible to test an otherwise fully production configuration without
 obtaining a certificate with the `CanSignHttpExchanges` extension. `amppkg`
-still needs to perform OCSP verification, so the Issuer CA must be valid (i.e. no
-self-signed certificates). e.g. You can use a certificate from [Let's Encrypt](https://letsencrypt.org/).
+still needs to perform OCSP verification, so the Issuer CA must be valid (i.e.
+no self-signed certificates). e.g. You can use a certificate from [Let's
+Encrypt](https://letsencrypt.org/).
 
 Running `amppkg` with the `-invalidcert` flag will skip the check for
 `CanSignHttpExchanges`. This flag is not necessary when using the
@@ -204,8 +231,9 @@ eligible for use in the AMP viewer in other browsers.
 
 ### Limitations
 
-Currently, the packager will refuse to sign any AMP documents larger than 4 MB.
-Patches that allow for streamed signing are welcome.
+Currently, the packager will refuse to sign any AMP documents that hit the size
+limit of 4MB. You can [monitor](monitoring.md#available-metrics) the size of
+your documents that have been signed, to see how close you are to the limit.
 
 The packager refuses to sign any URL that results in a redirect. This is by
 design, as neither the original URL nor the final URL makes sense as the signed
@@ -225,6 +253,65 @@ amp-install-serviceworker will still succeed in the unsigned AMP viewer case,
 and crawlers may reuse the contents of the signed exchange when displaying an
 AMP viewer to browser versions that don't support SXG.
 
+#### `<amp-script>`
+
+If you have any inline `<amp-script>`s (those with a `script` attribute), then
+the expiration of the SXG will be set based on the minimum `max-age` of those
+`<amp-script>`s, minus one day (due to
+[backdating](https://github.com/ampproject/amppackager/issues/397)). If
+possible, prefer external `<amp-script>`s (those with a `src` attribute), which
+do not have this limitation.
+
+If inline is necessary, you will need to weigh the [security
+risks](https://wicg.github.io/webpackage/draft-yasskin-http-origin-signed-responses.html#seccons-downgrades)
+against the [AMP Cache requirement](docs/cache_requirements.md) for a minimum
+`max-age` of `345600` (4 days). For SXGs shorter than that, the Google AMP Cache
+will treat them as if unsigned (by showing an AMP Viewer).
+
+#### How does `amppackager` process a document it cannot sign?
+
+Packager will respond to every request with either a signed document, an
+unsigned document, or an error.
+
+It will sign every document it can. It may, however, decide not to,
+for a number of reasons: the certificate may be invalid, the page may not be a
+valid AMP page, the page may not be an AMP page at all, the page may be 4MB or
+larger, etc. 
+
+If packager cannot sign the document but can fetch it, it will proxy the
+document unsigned.
+
+If there was a problem with the gateway fetch request, or with the original
+request, packager will respond with an HTTP error, and log the problem to
+stdout.
+
+You can monitor the packager's error rates, as well as the rates of signed
+vs unsigned documents, via the tools discussed in the next section.
+
+Specifically, you can monitor the requests that resulted in a signed or an
+unsigned document via `documents_signed_vs_unsigned` metric, and the ones that
+resulted in an error - via `total_requests_by_code_and_url` metric.
+
+#### Monitoring `amppackager` in production via its Prometheus endpoints
+
+Once you've run the `amppackager` server in production, you may want to
+[monitor](monitoring.md) its health and performance. You may also monitor the
+performance of the underlying requests to the AMP document server. You can
+monitor both servers via the [Prometheus](https://prometheus.io/) endpoints
+provided by `amppackager`. A few examples of questions you can answer:
+
+*  Is `amppackager` up and running?
+*  How many requests has it processed since it's been up?
+*  What was the 0.9 percentile latency of handling those request?
+*  How many of those requests have triggered a gateway request to the
+   AMP document server? 
+*  For those gateway requests, what was the 0.9 percentile latency of 
+   the AMP document server?
+
+You can perform one-off manual health inspections, visualize the real-time
+stats, set up alerts, and more. To learn what are all the things you can
+monitor, and how to do it, check the [monitoring manual](monitoring.md).
+
 ## Local Transformer
 
 The local transformer is a library within the AMP Packager that transforms AMP

amppkg.example.toml

@@ -255,6 +255,7 @@ ForwardedRequestHeaders = []
     # For the DNS challenge, go-acme/lego, there are certain environment variables that need to be set up which depends on
     # the DNS provider that you use to fulfill the DNS challenge.  See:
     # 	https://go-acme.github.io/lego/dns/
+    # To use this, build amppkg with `go build -tags dns01`; it is disabled by default because it bloats the binary.
     # DnsProvider = "gcloud"
 
   # This config will be used if 'autorenewcert' is turned on and 'development' is turned on.

cmd/amppkg/main.go

@@ -28,6 +28,7 @@ import (
 	"time"
 
 	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
 
 	"github.com/ampproject/amppackager/packager/certcache"
 	"github.com/ampproject/amppackager/packager/certloader"
@@ -42,6 +43,7 @@ import (
 var flagConfig = flag.String("config", "amppkg.toml", "Path to the config toml file.")
 var flagDevelopment = flag.Bool("development", false, "True if this is a development server.")
 var flagInvalidCert = flag.Bool("invalidcert", false, "True if invalid certificate intentionally used in production.")
+var flagStaging = flag.String("staging", "", "URL that overrides the base URL used to host certs, used for testing. Can only be used with -development flag.")
 
 // IMPORTANT: do not turn on this flag for now, it's still under development.
 var flagAutoRenewCert = flag.Bool("autorenewcert", false, "True if amppackager is to attempt cert auto-renewal.")
@@ -104,7 +106,7 @@ func main() {
 		} else {
 			die(errors.Wrap(err, "initializing cert cache"))
 		}
-        }
+	}
 
 	healthz, err := healthz.New(certCache)
 	if err != nil {
@@ -119,15 +121,21 @@ func main() {
 	defer rtvCache.StopCron()
 
 	var overrideBaseURL *url.URL
-	if *flagDevelopment {
+	if *flagStaging != "" {
+		overrideBaseURL, err = url.Parse(*flagStaging)
+		if err != nil {
+			die(errors.Wrap(err, "parsing staging URL"))
+		}
+	} else if *flagDevelopment {
 		overrideBaseURL, err = url.Parse(fmt.Sprintf("https://localhost:%d/", config.Port))
 		if err != nil {
 			die(errors.Wrap(err, "parsing development base URL"))
 		}
 	}
 
+	signerRequireHeaders := !*flagDevelopment
 	signer, err := signer.New(certCache, key, config.URLSet, rtvCache, certCache.IsHealthy,
-		overrideBaseURL, /*requireHeaders=*/!*flagDevelopment, config.ForwardedRequestHeaders)
+		overrideBaseURL, signerRequireHeaders, config.ForwardedRequestHeaders, time.Now)
 	if err != nil {
 		die(errors.Wrap(err, "building signer"))
 	}
@@ -143,7 +151,7 @@ func main() {
 		Addr: addr,
 		// Don't use DefaultServeMux, per
 		// https://blog.cloudflare.com/exposing-go-on-the-internet/.
-		Handler:           logIntercept{mux.New(certCache, signer, validityMap, healthz)},
+		Handler:           logIntercept{mux.New(certCache, signer, validityMap, healthz, promhttp.Handler())},
 		ReadTimeout:       10 * time.Second,
 		ReadHeaderTimeout: 5 * time.Second,
 		// If needing to stream the response, disable WriteTimeout and

cmd/gateway_server/server.go

@@ -74,7 +74,7 @@ func (s *gatewayServer) GenerateSXG(ctx context.Context, request *pb.SXGRequest)
 	}
 
 	// Note: do not initialize certCache, we just want it to hold the certs for now.
-	certCache := certcache.New(certs, nil, []string{""}, "", "", "", nil);
+	certCache := certcache.New(certs, nil, []string{""}, "", "", "", nil, time.Now)
 
 	privateKey, err := util.ParsePrivateKey(request.PrivateKey)
 	if err != nil {
@@ -116,7 +116,7 @@ func (s *gatewayServer) GenerateSXG(ctx context.Context, request *pb.SXGRequest)
 		},
 	}
 
-	packager, err := signer.New(certCache, privateKey, urlSets, s.rtvCache, shouldPackage, signUrl, false, []string{})
+	packager, err := signer.New(certCache, privateKey, urlSets, s.rtvCache, shouldPackage, signUrl, false, []string{}, time.Now)
 
 	if err != nil {
 		return errorToSXGResponse(err), nil

deploy/gcloud/Dockerfile.consumer

@@ -0,0 +1,78 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Used https://medium.com/@chemidy/create-the-smallest-and-secured-golang-docker-image-based-on-scratch-4752223b7324
+# as a guide.
+
+# Use an official Go runtime as a parent image
+FROM golang:1.13 as builder
+
+ENV GO111MODULE=on
+
+# Install git.
+# Git is required for fetching the dependencies.
+RUN apt-get update \
+    && apt-get install -y git
+
+WORKDIR /data
+
+# Run this if you clone from master branch.
+RUN git clone -b master https://github.com/ampproject/amppackager.git /data/amppackager
+# RUN git clone https://github.com/ampproject/amppackager.git /data/amppackager
+
+WORKDIR /data/amppackager/cmd/amppkg
+
+# Build the binary.
+# See: https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b
+#      https://github.com/kelseyhightower/contributors
+# Avoid "x509: failed to load system roots and no roots provided" by bundling root certificates.
+# Avoid dynamic linking by using the pure Go net package (-tags netgo)
+# Avoid dynamic linking by disabling cgo (CGO_ENABLED=0)
+# Reduce binary size by omitting dwarf information (-ldflags '-w')
+RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' -o /go/bin/amppkg
+
+FROM alpine:latest as certs
+
+RUN apk --update add ca-certificates
+
+# Build a small executable from docker alpine. Docker alpine is needed because
+# bash is required to be present by the deployer as it prints the env variables
+# listed below for verifiying the docker image.
+FROM alpine:latest 
+
+# Copy the certs from certs image.
+ENV PATH=/bin
+COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
+
+# Copy the AMP packager binary into our app dir inside the container.
+COPY --from=builder /go/bin/amppkg .
+
+# Make port 8080 available to the world outside this container. This
+# port must match the AMP Packager port configured in the toml file.
+EXPOSE 8080
+
+ENV PATH=$PATH:.
+
+# This env var is used by click-to-deploy deployer, right now needs to be
+# manually synced with C2D_RELEASE in the other Docker images for AMP Packager
+# Deployer currently in https://github.com/banaag/click-to-deploy but will move
+# to https://github.com/GoogleCloudPlatform/click-to-deploy.
+ENV C2D_RELEASE=0.0.1
+
+# Start the AMP Packager
+ENTRYPOINT ["amppkg"]
+
+# Set default flags to run in development mode.
+CMD ["-config=/consumer/amppkg_consumer.toml"]
+