Merge #146990 #147683

146990: roachtest: unconditionally save clusters that show raft fatal errors r=tbg a=tbg

When a cluster's logs contain a raft panic, it will be extended (by a week),
volume snapshots will be taken, and the cluster will not be destroyed. This
gives us the artifacts for a thorough investigation.

Verified manually via:

```
run --local acceptance/invariant-check-detection/failed=true
```

Here is the (editorialized) output:

```
test-teardown: 2025/05/20 08:15:15 cluster.go:2559: running cmd `([ -d logs ] && grep -RE '^...` on nodes [:1-4]; details in run_081515.744363000_n1-4_d-logs-grep-RE-Fraft.log
test-teardown: 2025/05/20 08:15:16 cluster.go:2995: extending cluster by 168h0m0s
test-teardown: 2025/05/20 08:15:16 cluster.go:1104: saving cluster local [tag:] (4 nodes) for debugging (--debug specified)
test-teardown: 2025/05/20 08:15:16 test_impl.go:478: test failure #2: full stack retained in failure_2.log: (test_runner.go:1705).maybeSaveClusterDueToInvariantProblems: invariant problem - snap name invariant-problem-local-8897676895823393049:
logs/foo.log:F250502 11:37:20.387424 1036 raft/raft.go:2411 ⋮ [T1,Vsystem,n1,s1,r155/1:?/Table/113/1/{43/578…-51/201…}?] 80 match(30115) is out of range [lastIndex(30114)]. Was the raft log corrupted, truncated, or lost?
```

Closes #145953.
Informs #146617.
Informs #138028.

Fixes #146355.

Epic: none

147683: pkg/util/log: parse otlp sink from yaml config r=TheComputerM a=TheComputerM

OpenTelemetry is now an industry standard for o11y and is
more efficient than other log sinks currently available.

This commit only defines basic configuration options for
the OTLP sink, like address, insecure, and compression,
and adds logic to parse them from the YAML config. The
actual sink implementation will follow in a future commit.

Informs: #143049

Release note: None

Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Mudit Somani <[email protected]>

Author: 3 people

docs/generated/logsinks.md

@@ -8,6 +8,8 @@ The supported log output sink types are documented below.
 
 - [Output to HTTP servers.](#output-to-http-servers.)
 
+- [Output to OpenTelemetry compatible collectors.](#output-to-opentelemetry-compatible-collectors.)
+
 - [Standard error stream](#standard-error-stream)
 
 
@@ -242,6 +244,70 @@ Configuration options shared across all sink types:
 
 
 
+<a name="output-to-opentelemetry-compatible-collectors.">
+
+## Sink type: Output to OpenTelemetry compatible collectors.
+
+
+This sink type causes logging data to be sent over the network through gRPC to
+a collector that can ingest log data in an [OTLP](https://opentelemetry.io) format.
+
+The configuration key under the `sinks` key in the YAML
+configuration is `otlp-servers`. Example configuration:
+
+//	sinks:
+//	   otlp-servers:
+//	      health:
+//	         channels: HEALTH
+//	         address: 127.0.0.1:4317
+
+Every new server sink configured automatically inherits the configuration set in the `otlp-defaults` section.
+
+For example:
+
+//	otlp-defaults:
+//	    redactable: false # default: disable redaction markers
+//	sinks:
+//	  otlp-servers:
+//	    health:
+//	       channels: HEALTH
+//	       # This sink has redactable set to false,
+//	       # as the setting is inherited from fluent-defaults
+//	       # unless overridden here.
+
+The default output format for OTLP sinks is
+`json`. [Other supported formats.](log-formats.html)
+
+{{site.data.alerts.callout_info}}
+Run `cockroach debug check-log-config` to verify the effect of defaults inheritance.
+{{site.data.alerts.end}}
+
+
+Type-specific configuration options:
+
+| Field | Description |
+|--|--|
+| `channels` | the list of logging channels that use this sink. See the [channel selection configuration](#channel-format) section for details.  |
+| `address` | the network address of the gRPC endpoint for ingestion of logs on your OpenTelemetry Collector/Platform. The host/address and port parts are separated with a colon. |
+| `insecure` | Disables transport security for the underlying gRPC connection.  Inherited from `otlp-defaults.insecure` if not specified. |
+| `compression` | can be "none" or "gzip" to enable gzip compression. Set to "gzip" by default. Inherited from `otlp-defaults.compression` if not specified. |
+
+
+Configuration options shared across all sink types:
+
+| Field | Description |
+|--|--|
+| `filter` | specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. |
+| `format` | the entry format to use. |
+| `format-options` | additional options for the format. |
+| `redact` | whether to strip sensitive information before log events are emitted to this sink. |
+| `redactable` | whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. |
+| `exit-on-error` | whether the logging system should terminate the process if an error is encountered while writing to this sink. |
+| `auditable` | translated to tweaks to the other settings for this sink during validation. For example, it enables `exit-on-error` and changes the format of files from `crdb-v1` to `crdb-v1-count`. |
+| `buffering` | configures buffering for this log sink, or NONE to explicitly disable. See the [common buffering configuration](#buffering-config) section for details.  |
+
+
+
 <a name="standard-error-stream">
 
 ## Sink type: Standard error stream

pkg/cli/log_flags_test.go

@@ -57,6 +57,17 @@ func TestSetupLogging(t *testing.T) {
 		`flush-trigger-size: 1.0MiB, ` +
 		`max-buffer-size: 50MiB, ` +
 		`format: newline}}`
+	const defaultOtlpConfig = `otlp-defaults: {` +
+		`insecure: false, ` +
+		`compression: gzip, ` +
+		`filter: INFO, ` +
+		`format: json, ` +
+		`redactable: true, ` +
+		`exit-on-error: false, ` +
+		`buffering: {max-staleness: 5s, ` +
+		`flush-trigger-size: 1.0MiB, ` +
+		`max-buffer-size: 50MiB, ` +
+		`format: newline}}`
 	stdFileDefaultsRe := regexp.MustCompile(
 		`file-defaults: \{` +
 			`dir: (?P<path>[^,]+), ` +
@@ -189,6 +200,7 @@ func TestSetupLogging(t *testing.T) {
 		// Shorten the configuration for legibility during reviews of test changes.
 		actual = strings.ReplaceAll(actual, defaultFluentConfig, "<fluentDefaults>")
 		actual = strings.ReplaceAll(actual, defaultHTTPConfig, "<httpDefaults>")
+		actual = strings.ReplaceAll(actual, defaultOtlpConfig, "<otlpDefaults>")
 		actual = stdFileDefaultsRe.ReplaceAllString(actual, "<stdFileDefaults($path)>")
 		actual = fileDefaultsNoMaxSizeRe.ReplaceAllString(actual, "<fileDefaultsNoMaxSize($path)>")
 		actual = strings.ReplaceAll(actual, fileDefaultsNoDir, "<fileDefaultsNoDir>")

pkg/cli/testdata/logflags

@@ -15,6 +15,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -51,6 +52,7 @@ start-single-node
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -91,6 +93,7 @@ sql
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledWarningNoRedaction>}}
 
 run
@@ -99,6 +102,7 @@ init
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledWarningNoRedaction>}}
 
 
@@ -112,6 +116,7 @@ bank
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledInfoNoRedaction>}}
 
 
@@ -123,6 +128,7 @@ demo
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrCfg(FATAL,false)>}}
 
 
@@ -139,6 +145,7 @@ start
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledInfoNoRedaction>}}
 
 
@@ -152,6 +159,7 @@ start
 config: {<stdFileDefaults(/pathA/logs)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -190,6 +198,7 @@ start
 config: {<stdFileDefaults(/mypath)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -229,6 +238,7 @@ start
 config: {<stdFileDefaults(/pathA/logs)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -273,6 +283,7 @@ start
 config: {<stdFileDefaults(/mypath)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -310,6 +321,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -348,6 +360,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -396,6 +409,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -455,6 +469,7 @@ start
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledInfoNoRedaction>}}
 
 
@@ -466,6 +481,7 @@ start
 config: {<stdFileDefaults(/mypath)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -505,6 +521,7 @@ start
 config: {<stdFileDefaults(/pathA)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -544,6 +561,7 @@ init
 config: {<fileDefaultsNoMaxSize(/mypath)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: {channels: {INFO: all},
 dir: /mypath,
 file-permissions: "0640",
@@ -563,6 +581,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -600,6 +619,7 @@ start
 config: {<stdFileDefaults(<defaultLogDir>)>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {file-groups: {default: <fileCfg(INFO: [DEV,
 OPS],
 WARNING: [HEALTH,
@@ -637,6 +657,7 @@ init
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledInfoNoRedaction>}}
 
 # Default when no severity is specified is WARNING.
@@ -647,6 +668,7 @@ init
 config: {<fileDefaultsNoDir>,
 <fluentDefaults>,
 <httpDefaults>,
+<otlpDefaults>,
 sinks: {<stderrEnabledWarningNoRedaction>}}
 
 

pkg/cmd/roachtest/cluster.go

@@ -1109,6 +1109,12 @@ func (c *clusterImpl) Save(ctx context.Context, msg string, l *logger.Logger) {
 	c.destroyState.mu.Unlock()
 }
 
+func (c *clusterImpl) saved() bool {
+	c.destroyState.mu.Lock()
+	defer c.destroyState.mu.Unlock()
+	return c.destroyState.mu.saved
+}
+
 var errClusterNotFound = errors.New("cluster not found")
 
 // validateCluster takes a cluster and checks that the reality corresponds to

pkg/cmd/roachtest/run.go

@@ -152,12 +152,20 @@ func runTests(register func(registry.Registry), filter *registry.TestFilter) err
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 	CtrlC(ctx, l, cancel, cr)
-	// Install goroutine leak checker and run it at the end of the entire test
-	// run. If a test is leaking a goroutine, then it will likely be still around.
-	// We could diff goroutine snapshots before/after each executed test, but that
-	// could yield false positives; e.g., user-specified test teardown goroutines
-	// may still be running long after the test has completed.
-	defer leaktest.AfterTest(l)()
+	if false {
+		// Install goroutine leak checker and run it at the end of the entire test
+		// run. If a test is leaking a goroutine, then it will likely be still around.
+		// We could diff goroutine snapshots before/after each executed test, but that
+		// could yield false positives; e.g., user-specified test teardown goroutines
+		// may still be running long after the test has completed.
+		//
+		// NB: we currently don't do this since it's been firing for a long time and
+		// nobody has cleaned up the leaks. While there are leaks, the leaktest
+		// output pollutes stdout and makes roachtest annoying to use.
+		//
+		// Tracking issue: https://github.com/cockroachdb/cockroach/issues/148196
+		defer leaktest.AfterTest(l)()
+	}
 
 	// We allow roachprod users to set a default auth-mode through the
 	// ROACHPROD_DEFAULT_AUTH_MODE env var. However, roachtests shouldn't