shttp, skip: use net/http instead of quic-go/http3, support tunneling (#198)

The "default" HTTP package for HTTP over SCION, shttp, now uses HTTP/1
or HTTP/2 based on the net/http standard library.
This allows our applications to talk to "normal" (i.e. non-HTTP/3)
web servers / clients when suitably proxied/tunneled.
Also, this now supports http://, which is useful for quickly running the
demo applications without nasty tricks or having to hassle with TLS
setup.

The existing library for HTTP/3 over SCION is moved to a separate
package shttp3, overhauled and reduced to the bare necessities.
This will need some additional love to make it useful (and used).
Normally HTTP/3 is using the same port as HTTPS but on UDP instead of
TCP -- we run both over UDP so we'll have to come up with something to
disambiguate.

Implement CONNECT in skip to support HTTPS. Change the way that SCION
hosts are identified, as the `.scion`-TLD approach is incompatible with
HTTPS (wrong name for certificates). Instead, it just looks at the list
of hosts for which a SCION address is configured in `/etc/hosts` or
`/etc/scion/hosts`.

Add a web gateway that acts as a proxy/gateway to allow mirroring some
content on the TCP/IP web into the SCION based web.

Author: matzf

Makefile

@@ -24,6 +24,7 @@ build: scion-bat \
 	scion-skip \
 	scion-ssh scion-sshd \
 	scion-webapp \
+	scion-web-gateway \
 	example-helloworld \
 	example-hellodrkey \
 	example-shttp-client example-shttp-server example-shttp-fileserver example-shttp-proxy
@@ -33,15 +34,15 @@ clean:
 	rm -f bin/*
 
 test:
-	go test -v -tags=$(TAGS) ./...
+	go test -tags=$(TAGS) ./...
 
 setup_lint:
 	@# Install golangci-lint (as dumb as this looks, this is the recommended way to install)
 	curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s -- -b ${DESTDIR} v1.37.1
 
 lint:
 	@type golangci-lint > /dev/null || ( echo "golangci-lint not found. Install it manually or by running 'make setup_lint'."; exit 1 )
-	golangci-lint run --build-tags=$(TAGS) --timeout=2m0s -v
+	golangci-lint run --build-tags=$(TAGS) --timeout=2m0s
 
 install: all
   # Note: install everything but the examples
@@ -107,6 +108,10 @@ scion-sshd:
 scion-webapp:
 	go build -tags=$(TAGS) -o $(BIN)/$@ ./webapp/
 
+.PHONY: scion-web-gateway
+scion-web-gateway:
+	go build -tags=$(TAGS) -o $(BIN)/$@ ./web-gateway/
+
 .PHONY: example-helloworld
 example-helloworld:
 	go build -tags=$(TAGS) -o $(BIN)/$@ ./_examples/helloworld/

README.md

@@ -139,8 +139,9 @@ netcat contains a SCION port of the netcat application. See the [netcat README](
 Pkg contains underlaying library code for scion-apps.
 
 - appnet: simplified and functionally extended wrapper interfaces for the SCION core libraries
-- appquic:  a simple interface to use QUIC over SCION
-- shttp: a client/server implementation of HTTP/3 over SCION/QUIC
+- appquic: a simple interface to use QUIC over SCION
+- shttp: glue library to use net/http libraries for HTTP over SCION
+- shttp3: glue library to use quic-go/http3 for HTTP/3 over SCION
 - integration: a simple framework to support intergration testing for the demo applications in this repository
 
 
@@ -161,6 +162,10 @@ Directory ssh contains a SSH client and server running over SCION network.
 More documentation is available in the [ssh README](ssh/README.md).
 
 
+## web-gateway
+
+web-gateway is a SCION web server that proxies web content from the TCP/IP web to the SCION web.
+
 ## webapp
 
 Webapp is a Go application that will serve up a static web portal to make it easy to experiment with SCIONLab test apps on a virtual machine.

_examples/shttp/README.md

@@ -2,129 +2,140 @@
 
 This directory contains small example programs that show how HTTP can be used over SCION/QUIC for servers, proxies, and clients:
 
-- fileserver: a server that serves the files from its working directory
-- proxy: a proxy server that can translate between HTTP and HTTP-over-SCION
 - server: a server with friendly greetings and other examples
 - client: a client that talks to the example server
+- fileserver: a server that serves the files from its working directory.
+    This includes an example for serving both HTTP and HTTPS.
+- proxy: a proxy server that can translate between HTTP and HTTP-over-SCION
 
 See also the package [shttp](../../pkg/shttp/README.md) for the underlaying library code.
 
 ## Build:
 
-Clone the repository netsec-ethz/scion-apps and build the eaxample applications:
-
 ```
-git clone https://github.com/netsec-ethz/scion-apps.git
-cd scion-apps
 make example-shttp-fileserver \
         example-shttp-proxy \
         example-shttp-server \
-        example-shttp-client
+        example-shttp-client \
+        scion-bat
 ```
 
 ## Running:
 
 All examples require a running SCION endhost stack, i.e. a running SCION dispatcher and SCION daemon. Please refer to '[Running](../../README.md#Running)' in this repository's main README and the [SCIONLab tutorials](https://docs.scionlab.org) to get started.
+See '[Environment](../../README.md#Environment)' on how to set the dispatcher and sciond environment variables when e.g. running multiple local ASes.
 
-### Generic file server example
+### Simple server example
 
-Run `example-shttp-fileserver`:
+Open a shell in the root of the scion-apps repository and run the `example-shttp-server`:
 
 ```
-bin/example-shttp-fileserver
+bin/example-shttp-server
 ```
 
-See '[Environment](../../README.md#Environment)' on how to set the dispatcher and sciond environment variables in the server's AS.
-
-Build `scion-bat` as a client for `example-shttp-fileserver`:
+Open a new shell and run the custom `example-shttp-client` to interact with the `example-shttp-server`:
 
 ```
-make scion-bat
+bin/example-shttp-client -s 17-ffaa:1:a,127.0.0.1
 ```
+Replace '17-ffaa:1:a' with your server's ISD and AS numbers.
 
-See also the application '[bat](../../bat/README.md)' for more details on the cURL-like CLI tool `scion-bat`.
-
-Access `example-shttp-fileserver` with `scion-bat`:
+Alternatively, we can also use the more generic command line HTTP client
+`scion-bat` to interact with the `example-shttp-server`. See also the
+application '[bat](../../bat/README.md)' for more details on the cURL-like CLI
+tool `scion-bat`.
 
 ```
-bin/scion-bat 17-ffaa:1:a,[127.0.0.1]:443/
+bin/scion-bat 17-ffaa:1:a,127.0.0.1/hello
+bin/scion-bat 17-ffaa:1:a,127.0.0.1/json
+bin/scion-bat -f 17-ffaa:1:a,127.0.0.1/form foo=bar
 ```
 
-Replace '17-ffaa:1:a' with your server's ISD and AS numbers and see '[Environment](../../README.md#Environment)' on how to set the dispatcher and sciond environment variables in the client's (or proxy's) AS.
+### File server example
 
-Run `example-shttp-proxy` to provide `example-shttp-fileserver` functionality via HTTP:
+Run `example-shttp-fileserver`:
 
 ```
-bin/example-shttp-proxy --remote=17-ffaa:1:a,[127.0.0.1]:443 --local=0.0.0.0:8080
+bin/example-shttp-fileserver
 ```
 
-Access `example-shttp-fileserver` via HTTP with `cURL`:
+Access `example-shttp-fileserver` with `scion-bat`:
 
 ```
-curl -v http://127.0.0.1:8080/
+bin/scion-bat http://17-ffaa:1:a,127.0.0.1/
 ```
 
-(Or navigate to http://127.0.0.1:8080/ in a web browser.)
-
-`example-shttp-proxy` can also be used as a proxy from SCION to HTTP, from SCION to SCION, and from HTTP to HTTP. See package [shttp](../../pkg/shttp/README.md) for more details.
 
-### Simple shttp-based server example
+### File server example with HTTPS
 
-Open a shell in the root of the scion-apps repository and run `example-shttp-server`:
+The file server optionally supports serving via HTTPS.
+For this, we need a **hostname** for the server, as a raw SCION address cannot
+(currently) be used as the subject of a TLS certificate.
+Then, we'll need to create a **key** and obtain a **certificate** for our server.
+We use a self signed certificate here and we cheat by installing the self
+signed certificate to the host's root CA list.
 
 ```
-cd _examples/shttp/server
-go run .
+# echo "1-ff00:0:111,[127.0.0.1] foo-server" >> /etc/scion/hosts
+$ mkdir certs; openssl req -newkey rsa:2048 -nodes -keyout certs/server.key -x509 -days 365 -subj '/CN=foo-server' -addext "subjectAltName = DNS:foo-server" -out certs/server.crt
+# cp -n certs/server.crt /etc/ssl/certs/ # for ubuntu/debian etc.
 ```
 
-Open a new shell in the scion-apps repository and access `example-shttp-server` with `scion-bat`:
-
+Then we provide the key/certs for the server at startup:
 ```
-bin/scion-bat 17-ffaa:1:a,[127.0.0.1]:443/hello
+bin/example-shttp-fileserver -cert certs/server.crt -key certs/server.key
 ```
 
-or
-
+And then access it with bat:
 ```
-bin/scion-bat 17-ffaa:1:a,[127.0.0.1]:443/json
+bin/scion-bat https://foo-server
 ```
 
-or
+Don't forget to remove `/etc/ssl/certs/server.crt` once you're done.
 
-```
-bin/scion-bat -f 17-ffaa:1:a,[127.0.0.1]:443/form foo=bar
-```
+**Note**: Instead of using a hostname and installing the certificate in the
+root CA store, we can also use `scion-bat`'s flag `-insecure=true`, to allow
+connections with unchecked certificates. But that's a bit boring, right?
 
-Run the custom `example-shttp-client` for `example-shttp-server`:
 
-```
-bin/example-shttp-client -s 17-ffaa:1:a,[127.0.0.1]:443
-```
+### Proxy example: SCION server, TCP/IP client
+
+The `example-shttp-proxy` is a reverse proxy that can proxy requests on TCP/IP to a SCION web server, or vice versa.
 
-Run `example-shttp-proxy` to provide `bin/example-shttp-server` functionality via HTTP:
+Listen on TCP/IP port 8888 and proxy request to a SCION URL, e.g. start the `example-shttp-server` as described above and then
 
 ```
-bin/example-shttp-proxy --remote=17-ffaa:1:a,[127.0.0.1]:443 --local=0.0.0.0:8080
+bin/example-shttp-proxy --port 8888 --remote=http://17-ffaa:1:a,127.0.0.1
 ```
 
-Access `example-shttp-server` via HTTP with `cURL`:
+Now we can access `example-shttp-server` via TCP/IP with `cURL`:
 
 ```
-curl http://127.0.0.1:8080/hello
+curl -sfS http://127.0.0.1:8888/hello
+curl -sfS http://127.0.0.1:8888/json
+curl -sfS -d foo=bar http://127.0.0.1:8888/form
 ```
 
-or
+And, finally, to see the cute dog picture:
+
+Navigate to http://127.0.0.1:8888/image in a web browser.
+
+### Proxy example: TCP/IP server, SCION client
+
+Listen on SCION port 8888 and proxy request to TCP/IP URL, e.g. https://www.scionlab.org
 
 ```
-curl http://127.0.0.1:8080/json
+bin/example-shttp-proxy --listen-scion --port 8888 --remote=https://www.scionlab.org
 ```
 
-or
+Now we can access www.scionlab.org via SCION with `scion-bat` (note the `Host:www.scionlab.org` directive, alternatively we could add a corresponding hostname entry in the hosts file).
 
 ```
-curl -d foo=bar http://127.0.0.1:8080/form
+bin/scion-bat http://17-ffaa:1:a,127.0.0.1:8888/ Host:www.scionlab.org
 ```
 
-And, finally, to see the cute dog picture:
+or alternatively
 
-Navigate to http://127.0.0.1:8080/image in a web browser.
+```
+bin/scion-bat --proxy http://17-ffaa:1:a,127.0.0.1:8888/ http://www.scionlab.org
+```

_examples/shttp/client/main.go

@@ -15,7 +15,6 @@
 package main
 
 import (
-	"crypto/tls"
 	"flag"
 	"fmt"
 	"io/ioutil"
@@ -37,16 +36,14 @@ func main() {
 		os.Exit(2)
 	}
 
-	// Create a standard server with our custom RoundTripper
+	// Create a standard client with our custom Transport/Dialer
 	c := &http.Client{
-		Transport: shttp.NewRoundTripper(&tls.Config{InsecureSkipVerify: true}, nil),
+		Transport: shttp.DefaultTransport,
 	}
-	// (just for demonstration on how to use Close. Clients are safe for concurrent use and should be re-used)
-	defer c.Transport.(shttp.RoundTripper).Close()
 
 	// Make a get request
 	start := time.Now()
-	query := fmt.Sprintf("https://%s/hello", *serverAddrStr)
+	query := fmt.Sprintf("http://%s/hello", *serverAddrStr)
 	resp, err := c.Get(shttp.MangleSCIONAddrURL(query))
 	if err != nil {
 		log.Fatal("GET request failed: ", err)
@@ -57,8 +54,11 @@ func main() {
 	log.Printf("\nGET request succeeded in %v seconds", end.Sub(start).Seconds())
 	printResponse(resp)
 
+	// (just for demonstration on how to use Close. Clients are safe for concurrent use and should be re-used)
+	c.CloseIdleConnections()
+
 	start = time.Now()
-	query = fmt.Sprintf("https://%s/form", *serverAddrStr)
+	query = fmt.Sprintf("http://%s/form", *serverAddrStr)
 	resp, err = c.Post(
 		shttp.MangleSCIONAddrURL(query),
 		"application/x-www-form-urlencoded",

_examples/shttp/fileserver/main.go

@@ -18,7 +18,6 @@ package main
 
 import (
 	"flag"
-	"fmt"
 	"log"
 	"net/http"
 	"os"
@@ -28,12 +27,16 @@ import (
 )
 
 func main() {
-	port := flag.Uint("p", 443, "port the server listens on")
+	certFile := flag.String("cert", "", "Path to TLS server certificate for optional https")
+	keyFile := flag.String("key", "", "Path to TLS server key for optional https")
 	flag.Parse()
 
 	handler := handlers.LoggingHandler(
 		os.Stdout,
 		http.FileServer(http.Dir("")),
 	)
-	log.Fatal(shttp.ListenAndServe(fmt.Sprintf(":%d", *port), handler, nil))
+	if *certFile != "" && *keyFile != "" {
+		go func() { log.Fatal(shttp.ListenAndServeTLS(":443", *certFile, *keyFile, handler)) }()
+	}
+	log.Fatal(shttp.ListenAndServe(":80", handler))
 }