webapp: update to current scionlab (#179)

Update to netsec-ethz/scion:v2020.12 / netsec-ethz/scion-apps:v0.3.0.
Adapt to various changes in SCION and SCIONLab, in particular to changed
directory structures (update how configuration files are found etc.).
Update parameters for invocation of tools and apps.

Remove unnecessary helper "config" files `client_user.json` and
`server_ias.json` and instead pass the relevant information about the currently
selected AS explicitly to the page handlers.
As a consequence, the selected AS is no longer persistent accross restart of
the webapp.
Streamline the discovery of the relevant AS configuration files (find relevant
files and addresses once and remember them, instead of finding them again and
again).

Simplify invocation of the webapp by removing various command line parameters
* remove -sbin and -sabin, instead find binaries for `scion` tool and other "apps" on PATH
* remove -slogs, file log not always available and no longer needed as health
  checks were changed to inspect metrics instead of logs
* remove -sroot

Update health checks
* simplify scripts by explicitly passing path to topology file
* update keys for parsing topology.json file
* inspect metrics instead of log files to determine if beacons are arriving

Misc:
* replace BurntSushi/toml with pelletier/go-toml, allows extracting information
  from toml files without clunky helper structs.
* increase linter timeout

Author: mmalesev

Makefile

@@ -21,6 +21,7 @@ build: scion-bat \
 	scion-sensorfetcher scion-sensorserver \
 	scion-skip \
 	scion-ssh scion-sshd \
+	scion-webapp \
 	example-helloworld \
 	example-hellodrkey \
 	example-shttp-client example-shttp-server example-shttp-fileserver example-shttp-proxy
@@ -38,7 +39,7 @@ setup_lint:
 
 lint:
 	@type golangci-lint > /dev/null || ( echo "golangci-lint not found. Install it manually or by running 'make setup_lint'."; exit 1 )
-	golangci-lint run --build-tags=$(TAGS)
+	golangci-lint run --build-tags=$(TAGS) --timeout=2m0s -v
 
 install: all
   # Note: install everything but the examples

go.mod

@@ -14,6 +14,7 @@ require (
 	github.com/mattn/go-sqlite3 v1.9.1-0.20180719091609-b3511bfdd742
 	github.com/msteinert/pam v0.0.0-20190215180659-f29b9f28d6f9
 	github.com/netsec-ethz/rains v0.2.0
+	github.com/pelletier/go-toml v1.8.1-0.20200708110244-34de94e6a887
 	github.com/scionproto/scion v0.6.0
 	github.com/smartystreets/goconvey v1.6.4
 	golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a

webapp/README.md

@@ -61,42 +61,34 @@ Install all [SCIONLab apps](https://github.com/netsec-ethz/scion-apps) and depen
 ```shell
 cd scion-apps
 ./deps.sh
-make install
+make build
 ```
 
-Development Run on Local Topology:
+Development Run on ScionLab Topology:
 You can alter the defaults on the command line, all of which are listed below:
 ```shell
-webapp \
+./bin/scion-webapp \
 -a 127.0.0.1 \
--p 8081 \
--r . \
+-p 8000 \
+-r $GOPATH/src/github.com/netsec-ethz/scion-apps/webapp/web/data \
 -srvroot $GOPATH/src/github.com/netsec-ethz/scion-apps/webapp/web \
--sabin $GOPATH/bin \
--sroot $GOPATH/src/github.com/scionproto/scion \
--sbin $GOPATH/src/github.com/scionproto/scion/bin \
--sgen $GOPATH/src/github.com/scionproto/scion/gen \
--sgenc $GOPATH/src/github.com/scionproto/scion/gen-cache \
--slogs $GOPATH/src/github.com/scionproto/scion/logs
+-sgen /etc/scion \
+-sgenc /var/lib/scion 
 ```
 or can you run `webapp` like this, which will use the defaults above:
 ```shell
-webapp
+./bin/scion-webapp
 ```
 
-Development Run on SCIONLab Topology:
+Development Run on Local Topology:
 ```shell
-webapp \
--a 0.0.0.0 \
+./bin/scion-webapp \
+-a 127.0.0.1 \
 -p 8080 \
--r ~/go/src/github.com/netsec-ethz/scion-apps/webapp/web/data \
--srvroot ~/go/src/github.com/netsec-ethz/scion-apps/webapp/web \
--sabin $GOPATH/bin \
--sroot /etc/scion \
--sbin /usr/bin \
--sgen /etc/scion/gen \
--sgenc /var/lib/scion \
--slogs /var/log/scion
+-r $GOPATH/src/github.com/netsec-ethz/scion-apps/webapp/web/data \
+-srvroot $GOPATH/src/github.com/netsec-ethz/scion-apps/webapp/web \
+-sgen $HOME/scion/gen \
+-sgenc  $HOME/scion/gen-cache
 ```
 
 ## Dependencies
@@ -110,21 +102,13 @@ Usage of webapp:
   -p int
         Port of server host. (default 8000)
   -r string
-        Root path to read/browse from, CAUTION: read-access granted from -a and -p. (default ".")
-  -sabin string
-        Path to execute the installed scionlab apps binaries (default "/home/ubuntu/go/bin")
-  -sbin string
-        Path to execute SCION bin directory of infrastructure tools (default "/home/ubuntu/go/src/github.com/scionproto/scion/bin")
+        Root path to read/browse from, CAUTION: read-access granted from -a and -p. (default "$GOPATH/src/github.com/netsec-ethz/scion-apps/webapp /web/data")
   -sgen string
-        Path to read SCION gen directory of infrastructure config (default "/home/ubuntu/go/src/github.com/scionproto/scion/gen")
+        Path to read SCION gen directory of infrastructure config (default "/etc/scion")
   -sgenc string
-        Path to read SCION gen-cache directory of infrastructure run-time config (default "/home/ubuntu/go/src/github.com/scionproto/scion/gen-cache")
-  -slogs string
-        Path to read SCION logs directory of infrastructure logging (default "/home/ubuntu/go/src/github.com/scionproto/scion/logs")
-  -sroot string
-        Path to read SCION root directory of infrastructure (default "/home/ubuntu/go/src/github.com/scionproto/scion")
+        Path to read SCION gen-cache directory of infrastructure run-time config (default "/var/lib/scion")
   -srvroot string
-        Path to read/write web server files. (default "/home/ubuntu/go/src/github.com/netsec-ethz/scion-apps/webapp/web")
+        Path to read/write web server files. (default "$GOPATH/src/github.com/netsec-ethz/scion-apps/webapp/web")
 ```
 
 ## Related Links

webapp/dependencies.md

@@ -27,22 +27,21 @@ webapp -h
 - cat
 - systemctl
 
-## SCION Binaries Used/Checked
-- $sbin/scmp
-- $sabin/bwtestclient
-- $sabin/imagefetcher
-- $sabin/sensorfetcher
+## SCION Binaries Used/Checked (should be somewhere on $PATH)
+- scion (for scion ping and scion traceroute)
+- scion-bwtestclient
+- scion-imagefetcher
+- scion-sensorfetcher
 
 ## Scripts/Directories Used/Checked
-- $sroot/scion.sh (deprecated)
-- $slogs/bs[IA]-1.log
-- $sgen/ISD*/AS* (scanning for local IAs)
-- $sgen/ISD[ISD]/AS[AS]/endhost/topology.json
+- $SCION_ROOT/scion.sh (only for local topologies (ISD < 16))
+- $sgen/.*topology.json (checking all subdirectories of $sgen for topology.json)
+- $sgen/.*(sciond|sd).toml (for reading sciond address)
+- $sgen/*cs*.toml (for reading prometheus server address)
 - $sgenc/*.crt
 - $sgenc/*.trc
 - $sgenc/ps[IA]-1.path.db
 - /run/shm/dispatcher/default.sock
-- /run/shm/sciond/default.sock
 
 ## Static Webserver Required
 - $srvroot/favico.ico
@@ -59,7 +58,3 @@ webapp -h
 - $srvroot/data/
 - $srvroot/data/images/
 - $srvroot/logs/
-
-
-
-

webapp/development.md

@@ -74,6 +74,7 @@ Add alternate test forwarding port line in `Vagrantfile`:
 Update Go Paths:
 ```shell
 echo 'export GOPATH="$HOME/go"' >> ~/.profile
+echo 'export GOBIN="$GOPATH/bin"' >> ~/.profile
 echo 'export PATH="$HOME/.local/bin:$GOPATH/bin:/usr/local/go/bin:$PATH"' >> ~/.profile
 source ~/.profile
 mkdir -p "$GOPATH"
@@ -91,35 +92,25 @@ Build and install `scion-apps`:
 ```shell
 sudo apt install make gcc libpam0g-dev
 cd ~
-git clone -b scionlab https://github.com/netsec-ethz/scion-apps
+git clone -b master https://github.com/netsec-ethz/scion-apps
 cd scion-apps
+make setup_lint
 make install
 ```
 
-Download scionlab's fork of scion and build and install `sig`:
-```shell
-cd ~
-git clone -b scionlab https://github.com/netsec-ethz/scion
-go build -o $GOPATH/bin/sig ~/scion/go/sig/main.go
-```
-
-Install Go Watcher:
+(Optional) Install Go Watcher:
 ```shell
 go get -u github.com/mitranim/gow 
 ```
 
-Development Run:
+Development Run (check dependencies.md for the meaning of the flags):
 ```shell
 cd ~/scion-apps/webapp
 gow run . \
 -a 0.0.0.0 \
 -p 8080 \
 -r ./web/data \
 -srvroot ./web \
--sabin /usr/bin/scion \
--sroot /etc/scion \
--sbin /usr/bin \
 -sgen  /etc/scion/gen \
 -sgenc /var/lib/scion \
--slogs /var/log/scion
 ```