_content/doc: add documentation for Go telemetry

Add documentation for Go telemetry, aimed at Go users who want to
understand telemetry better. This article goes into some detail about
the telemetry data flow, but doesn't get into details about x/telemetry
APIs, as those are mostly interesting only for Go toolchain developers,
and therefore belong in x/telemetry documentation.

For golang/go#63883

Change-Id: Ie7e82960a2dbe5e0c0c81e46dc0b0c86174365fe
Reviewed-on: https://go-review.googlesource.com/c/website/+/558196
Reviewed-by: Alan Donovan <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: Russ Cox <[email protected]>
Auto-Submit: Robert Findley <[email protected]>

Author: findleyr

_content/doc/telemetry.md

@@ -0,0 +1,376 @@
+---
+title: "Go Telemetry"
+layout: article
+breadcrumb: true
+date: 2024-02-07:00:00Z
+---
+
+Table of Contents:
+
+ [Background](#background)\
+ [Overview](#overview)\
+ [Configuration](#config)\
+ [Counters](#counters)\
+ [Reporting and Uploading](#reports)\
+ [Charts](#charts) \
+ [IDE Prompting](#ide) \
+ [Frequently Asked Questions](#faq)
+
+## Background {#background}
+
+Go telemetry is a way for Go toolchain programs to collect data about their
+performance and usage. Here "Go toolchain" means developer tools maintained
+by the Go team, including the `go` command and supplemental tools such as the
+Go language server `gopls` or Go security tool `govulncheck`. Go telemetry is
+only intended for use in programs maintained by the Go team.
+
+By default, telemetry data is kept only on the local computer, but users may
+opt in to uploading an approved subset of telemetry data to [telemetry.go.dev].
+Uploaded data helps the Go team improve the Go language and its tools,
+by helping us understand usage and breakages.
+
+The word "telemetry" has acquired negative connotations in the world of open
+source software, in many cases deservedly so. Yet measuring the user experience
+is an important element of modern software engineering, and data sources such
+as GitHub issues or annual surveys are coarse and lagging indicators,
+insufficient for the types of questions the Go team needs to be able to answer.
+Go telemetry is designed to help programs in the toolchain collect useful data
+about their reliability, performance, and usage, while maintaining the
+transparency and privacy that users expect from the Go project. To learn more
+about the design process and motivation for telemetry, please see the
+[telemetry blog posts](https://research.swtch.com/telemetry).
+To learn more about telemetry and privacy, please see the
+[telemetry privacy policy](https://telemetry.go.dev/privacy).
+
+This page explains how Go telemetry works, in some detail. For quick answers to
+frequently asked questions, see the [FAQ](#faq).
+
+## Overview {#overview}
+
+Go telemetry uses three core data types:
+
+- [_Counters_](#counters) are lightweight counts of named events, instrumented
+  in the toolchain program. If collection is enabled (the [mode](#config)
+  is **local** or **on**), counters are written to a memory-mapped file in the
+  local file system.
+- [_Reports_](#reports) are aggregated summaries of counters for a given week.
+  If uploading is enabled (the [mode](#config) is **on**), reports for
+  [approved counters](#proposals) are uploaded to [telemetry.go.dev], where
+  they are publicly accessible.
+- [_Charts_](#charts) summarize uploaded reports for all users.
+  Charts can be viewed at [telemetry.go.dev].
+
+All local Go telemetry data and configuration is stored in the directory
+<code>[os.UserConfigDir()](/pkg/os#UserConfigDir)/go/telemetry</code>
+directory. Below, we'll refer to this directory as `<gotelemetry>`.
+
+The diagram below illustrates this data flow.
+
+<div class="image">
+  <center>
+    <img max-width="800px" src="/doc/telemetry/dataflow.png" />
+  </center>
+</div>
+
+In the rest of this document, we'll explore the components of this diagram. But
+first, let's learn more about the configuration that controls it.
+
+## Configuration {#config}
+
+The behavior of Go telemetry is controlled by a single value: the telemetry
+_mode_. The possible values for `mode` are `local` (the default), `on`, or
+`off`:
+
+- When `mode` is `local`, telemetry data is collected and stored on the local
+  computer, but never uploaded to remote servers.
+- When `mode` is `on`, data is collected, and may be uploaded depending on
+  [sampling](#uploads).
+- When `mode` is `off`, data is neither collected nor uploaded.
+
+The [`gotelemetry`](/pkg/golang.org/x/telemetry/cmd/gotelemetry) command
+configures the telemetry mode and manages local telemetry data. Use this
+command to install it:
+
+```
+go install golang.org/x/telemetry/cmd/gotelemetry@latest
+```
+
+The following commands interact with the telemetry mode:
+
+- `gotelemetry local`: set the mode to `local`.
+- `gotelemetry on`: set the mode to `on`.
+- `gotelemetry off`: set the mode to `off`.
+- `gotelemetry env`: see the current mode.
+
+For the complete usage information of the `gotelemetry` command line tool,
+see its [package documentation](/pkg/golang.org/x/telemetry/cmd/gotelemetry).
+
+Telemetry may also be enabled by accepting an [IDE prompt](#ide).
+
+## Counters {#counters}
+
+As mentioned above, Go telemetry is instrumented via _counters_. Counters come
+in two variants: basic counters and stack counters.
+
+### Basic counters
+
+A _basic counter_ is an incrementable value with a name that describes the
+event that it counts. For example, the `gopls/client:vscode` counter records
+the number of times a `gopls` session is initiated by VS Code. Alongside this
+counter we may have `gopls/client:neovim`, `gopls/client:eglot`, and so on, to
+record sessions with different editors or language clients. If you used
+multiple editors throughout the week, you might record the following counter
+data:
+
+    gopls/client:vscode 8
+    gopls/client:neovim 5
+    gopls/client:eglot  2
+
+When counters are related in this way, we sometimes refer to the part before
+the `:` the _chart name_ (`gopls/client` in this case), and the part after `:`
+as the _bucket name_ (`vscode`). We'll see why this matters when we discuss
+[charts](#charts).
+
+Basic counters can also represent a _histogram_. For example, the {{raw
+`<code>gopls/completion/latency:&lt;50ms</code>`}} counter records the number
+of times an autocompletion takes less than 50ms.
+
+{{raw `
+<pre>
+gopls/completion/latency:&lt;10ms
+gopls/completion/latency:&lt;50ms
+gopls/completion/latency:&lt;100ms
+...
+</pre>
+`}}
+
+This pattern for recording histogram data is a convention: there's nothing
+special about the {{raw `<code>&lt;50ms</code>`}} bucket name. These types of
+counters are commonly used to measure performance.
+
+### Stack counters
+
+A _stack counter_ is a counter that also records the current call stack of the
+Go toolchain program when the count is incremented. For example, the
+`crash/crash` stack counter records the call stack when a toolchain program
+crashes:
+
+    crash/crash
+    golang.org/x/tools/gopls/internal/golang.hoverBuiltin:+22
+    golang.org/x/tools/gopls/internal/golang.Hover:+94
+    golang.org/x/tools/gopls/internal/server.Hover:+42
+    ...
+
+Stack counters typically measure events where program invariants are violated.
+The most common example of this is a crash, but another example is the
+`gopls/bug` stack counter, which counts unusual situations identified in
+advance by the programmer, such as a recovered panic or an error that "can't
+happen". Stack counters include only the names and line numbers of functions
+within Go toolchain programs. They don't include any information about user
+inputs, such as the names or contents of a user's source code.
+
+Stack counters can help track down rare or tricky bugs that don't get reported
+by other means. Since introducing the `gopls/bug` counter, we've found
+[dozens of instances](https://github.com/golang/go/issues?q=label%3Agopls%2Ftelemetry-wins)
+of "unreachable" code that was reached in practice, and tracking down these
+exceptions has led to the discovery (and fix) of many user-visible bugs that
+were either not obvious to the user or too difficult to report. Especially with
+prerelease testing, stack counters can help us improve the product more
+efficiently than we could without automation.
+
+### Counter files
+
+All counter data is written to the `<gotelemetry>/local` directory, in
+files named according to the following schema:
+
+```
+[program name]@[program version]-[go version]-[GOOS]-[GOARCH]-[date].v1.count
+```
+
+- The **program name** is the basename of the program's package path, as reported
+  by [debug.BuildInfo].
+- The **program version** and **go version** are also reported by [debug.BuildInfo].
+- The **GOOS** and **GOARCH** values are reported by
+  [`runtime.GOOS`](/pkg/runtime#GOOS) and
+  [`runtime.GOARCH`](/pkg/runtime#GOARCH).
+- The **date** is the date the counter file was created, in `YYYY-MM-DD` format.
+
+These files are memory mapped into each running instance of the instrumented
+programs. The use of a memory-mapped file means that even if the program
+immediately crashes, or several copies of instrumented tools are running
+simultaneously, the counters are recorded safely.
+
+## Reporting and uploading {#reports}
+
+Approximately once a week, counter data gets aggregated into reports named
+`<date>.json` in the `<gotelemetry>/local` directory. These reports sum all of
+counts for the previous week, grouped by the same program identifiers used for
+the counter file (program name, program version, go version, GOOS, and GOARCH).
+
+Local reports can be viewed as charts with the
+[`gotelemetry view`](/pkg/golang.org/x/telemetry/cmd/gotelemetry) command.
+Here's an example summary of the `gopls/completion/latency` counter:
+
+<div class="image">
+  <center>
+    <img max-width="800px" src="/doc/telemetry/gopls-latency.png" />
+  </center>
+</div>
+
+### Uploading {#uploads}
+
+If telemetry uploading is enabled, the weekly reporting process will also
+generate reports containing the subset of counters present in the
+[upload config](https://telemetry.go.dev/config). These counters must be
+approved by the public review process described in the next section. After it
+has been successfully uploaded, a copy of the uploaded reports are stored in
+the `<gotelemetry>/upload` directory.
+
+Once enough users opt in to uploading telemetry data, the upload process will
+randomly skip uploading for a fraction of reports, to reduce collection amounts
+and increase privacy while maintaining statistical significance.
+
+### The telemetry proposal process {#proposals}
+
+Counters may be added to the upload configuration only through the _telemetry
+proposal process_, which proceeds as follows:
+
+1. The proposer files a [proposal] to upload new data. This is expressed in the
+   form of a specific [chart](#charts) that will be displayed on
+   [telemetry.go.dev].
+2. Once discussion on the issue resolves, the proposal is approved or declined
+   by a member of the Go team.
+3. The proposer sends a CL modifying the internal
+   [chart config](https://go.googlesource.com/telemetry/+/refs/heads/master/internal/configgen/config.txt)
+   to include the new chart.
+4. An automatic process regenerates the upload config to allow uploading of the
+   counters required for the new chart. This process will also regularly add
+   new versions of the relevant programs to the upload config as they are
+   released.
+
+In order to be approved, new charts cannot carry sensitive user information,
+and additionally must be both useful and feasible. In order to be _useful_,
+charts must serve a specific purpose, with actionable outcomes, that can't be
+served by other means. For example, in order to collect a counter that measures
+the latency of a given operation, it must be shown that this latency can't
+reasonably be measured via benchmarking, and that knowing the latency
+distribution will help meaningfully improve future versions of the program in
+question. In order to be _feasible_, it must be possible to reliably collect
+the requisite data, and the resulting measurements must be statistically
+significant. To demonstrate feasibility, the proposer may be asked to instrument
+the target program with counters and collect them locally first.
+
+The full set of such proposals is available at the
+[proposal project](https://github.com/orgs/golang/projects/29) on GitHub.
+
+## Charts {#charts}
+
+In addition to accepting uploads, the [telemetry.go.dev] website makes uploaded
+data publicly available. Each day, uploaded reports are processed into two
+outputs, which are available on the [telemetry.go.dev] homepage.
+
+- _merged_ reports merged counters from all uploads received on the given day.
+- _charts_ plot uploaded data as specified in the [chart config], which was
+  produced as part of the proposal process. Recall from the discussion of
+  [counters](#counters) that counter names such as `foo:bar` are decomposed
+  into the chart name `foo` and bucket name `bar`. Each chart aggregates
+  counters with the same chart name into the corresponding buckets.
+
+Charts are specified in the [chart config] format. For example, here's the
+chart config for the `gopls/client` chart.
+
+    title: Editor Distribution
+    counter: gopls/client:{vscode,vscodium,vscode-insiders,code-server,eglot,govim,neovim,coc.nvim,sublimetext,other}
+    description: measure editor distribution for gopls users.
+    type: partition
+    issue: https://go.dev/issue/61038
+    issue: https://go.dev/issue/62214 # add vscode-insiders
+    program: golang.org/x/tools/gopls
+    version: v0.13.0 # temporarily back-version to demonstrate config generation.
+
+This configuration describes the chart to be produced, enumerates the set of
+counters to be aggregated, and specifies the program versions to which the
+chart applies. Additionally, the [proposal process](#proposals) requires that
+an accepted proposal be associated with the chart. Here's the chart resulting
+from that config:
+
+<div class="image">
+  <center>
+    <img src="/doc/telemetry/gopls-clients.png" />
+  </center>
+</div>
+
+## IDE Prompting {#ide}
+
+For telemetry to answer the types of questions we want to ask of it, the set of
+users opting in to uploading need not be large--approximately 16,000
+participants would allow for statistically significant measurements at the
+desired level of granularity. However, there is still a cost to assembling this
+healthy sample: we need to ask a large number of Go developers if they want to
+opt in.
+
+Furthermore, even if a large number of users choose to opt in _now_ (perhaps
+after reading a Go blog post), those users may be skewed toward experienced Go
+developers, and over time that initial sample will grow even more skewed.
+Also, as people replace their computers, they must actively choose to opt in
+again. In the telemetry blog post series, this is referred to as the
+["campaign cost"](https://research.swtch.com/telemetry-opt-in#campaign) of
+the opt-in model.
+
+To help keep the sample of participating users fresh, the Go language server
+[`gopls`] supports a prompt that asks users to opt in to Go telemetry.
+Here's what that looks like from VS Code:
+
+<div class="image">
+  <center>
+    <img width="600px" src="/doc/telemetry/prompt.png" />
+  </center>
+</div>
+
+If users choose "Yes", their telemetry [mode](#config) will be set to `on`,
+just as if they had run
+[`gotelemetry on`](/pkg/golang.org/x/telemetry/cmd/gotelemetry). In this way,
+opting in is as easy as possible, and we can continually reach a large and
+stratified sample of Go developers.
+
+## Frequently Asked Question {#faq}
+
+**Q: How do I enable or disable Go telemetry?**
+
+A: Use the `gotelemetry` command, which can be installed with `go install
+golang.org/x/telemetry/cmd/gotelemetry@latest`. Run `gotelemetry off` to
+disable everything, even local collection. Run `gotelemetry on` to enable
+everything, including uploading approved counters to [telemetry.go.dev]. See
+the [Configuration](#config) section for more info.
+
+**Q: Where does local data get stored?**
+
+A: In the <code>[os.UserConfigDir()](/pkg/os#UserConfigDir)/go/telemetry</code> directory.
+
+**Q: How often does data get uploaded, if I opt in?**
+
+A: Approximately once a week.
+
+**Q: What data gets uploaded, if I opt in?**
+
+A: Only counters that are listed in the
+[upload config](https://telemetry.go.dev/config) may be uploaded. 
+This is generated from the [chart config], which may be more readable.
+
+**Q: How do counters get added to the upload config?**
+
+A: Through the [public proposal process](#proposals).
+
+**Q: Where can I see telemetry data that has been uploaded?**
+
+A: Uploaded data is available as charts or merged summaries at [telemetry.go.dev].
+
+**Q: Where is the source code for Go telemetry?**
+
+A: At [golang.org/x/telemetry](/pkg/golang.org/x/telemetry).
+
+[`gopls`]: /pkg/golang.org/x/tools/gopls
+[debug.BuildInfo]: /pkg/runtime/debug#BuildInfo
+[proposal]: /issue/new?assignees=&labels=Telemetry-Proposal&projects=golang%2F29&template=12-telemetry.yml&title=x%2Ftelemetry%2Fconfig%3A+proposal+title
+[telemetry.go.dev]: https://telemetry.go.dev
+[chart config]: /pkg/golang.org/x/telemetry/internal/graphconfig