Collect basic request metrics and expose using GoLang's expvar

[yinonavraham]

For every request, the following metrics are updated:

• totalRequestCount - the total number of requests since the application started
• concurrentRequestCount - the current number of concurrent requests
• maxConcurrentRequestCount - the maximum number of concurrent requests since the application started

Metrics are disabled by default, to preserve the current behavior and performance. To enable metrics, run the application with the '-metrics' command line flag. To get the current metrics values, use the following request path:

GET /debug/vars

The impact of collecting request metrics when enabled on the performance is very low (see example below). When metrics are disabled there is no impact what-so-ever.

Example

Benchmark without metrics:

$ ./whoami -port=8000 &
Starting up on port 8000

$ go-wrk -d 10 -c 500 -T 10000 http://localhost:8000/bench
Running 10s test @ http://localhost:8000/bench
  500 goroutine(s) running concurrently
375669 requests in 9.971437293s, 39.05MB read
Requests/sec:		37674.51
Transfer/sec:		3.92MB
Avg Req Time:		13.271573ms
Fastest Request:	78.579µs
Slowest Request:	395.278902ms
Number of Errors:	0
$ curl localhost:8000/debug/vars -s | jq .metrics
null

Benchmark with metrics enabled:

$ ./whoami -port=8000 -metrics &
Starting up on port 8000

$ go-wrk -d 10 -c 500 -T 10000 http://localhost:8000/bench
Running 10s test @ http://localhost:8000/bench
  500 goroutine(s) running concurrently
357041 requests in 9.977480102s, 37.11MB read
Requests/sec:		35784.69
Transfer/sec:		3.72MB
Avg Req Time:		13.972457ms
Fastest Request:	108.235µs
Slowest Request:	402.588444ms
Number of Errors:	0
$ curl localhost:8000/debug/vars -s | jq .metrics
{
  "concurrentRequestCount": 0,
  "maxConcurrentRequestCount": 192,
  "totalRequestCount": 357041
}

$ go-wrk -d 10 -c 500 -T 10000 http://localhost:8000/?wait=1s
Running 10s test @ http://localhost:8000/?wait=1s
  500 goroutine(s) running concurrently
5000 requests in 10.211386043s, 1.86MB read
Requests/sec:		489.65
Transfer/sec:		186.97KB
Avg Req Time:		1.021138604s
Fastest Request:	1.000358045s
Slowest Request:	1.125278s
Number of Errors:	0
$ curl localhost:8000/debug/vars -s | jq .metrics
{
  "concurrentRequestCount": 0,
  "maxConcurrentRequestCount": 500,
  "totalRequestCount": 362041
}

[yinonavraham]

Can someone please review this PR, approve or reject? Any feedback is welcome... I really don't like to keep PRs open for long time - they don't age well...