test(o11y): End-to-end HTTP to Cloud Trace integration test #3822

westarle · 2025-11-17T16:33:42Z

No description provided.

codecov · 2025-11-17T16:42:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.92%. Comparing base (5f97cb1) to head (566fd1d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3822   +/-   ##
=======================================
  Coverage   95.92%   95.92%           
=======================================
  Files         156      156           
  Lines        5991     5991           
=======================================
  Hits         5747     5747           
  Misses        244      244

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

src/integration-tests/src/observability/cloud_trace.rs

src/integration-tests/tests/telemetry.rs

westarle · 2025-11-21T16:53:47Z

This passes now (needed retry on 500's and maybe less wonky retry policy.)

dbolduc

Looks mostly good, but some things are strange... mainly the initial sleep and the multiple flushes.

src/integration-tests/tests/telemetry.rs

dbolduc · 2025-11-21T19:54:06Z

src/integration-tests/tests/telemetry.rs

+        let root_span = tracing::info_span!("e2e_root", "otel.name" = span_name);
+        let trace_id = {
+            let _enter = root_span.enter();
+            let otel_ctx = root_span.context();


TIL: opentelemetry::trace::TraceContextExt

src/integration-tests/tests/telemetry.rs

dbolduc · 2025-11-21T20:03:23Z

src/integration-tests/tests/telemetry.rs

+        let mut trace = None;
+
+        for delay in backoff_delays {
+            println!("Waiting {}s before polling...", delay);


nit: add a comment saying why we are doing this. There is actually a good reason.

// Because we are limited by quota, start with a backoff.

src/integration-tests/tests/telemetry.rs

dbolduc

I think the initial headers for the interceptor needs more thought, but maybe not for this PR.

dbolduc · 2025-11-21T22:13:32Z

src/integration-tests/src/observability/auth.rs

+        // Wait for the first refresh to complete.
+        // We ignore the result because if the sender is dropped (unlikely),
+        // the interceptor will just fail requests, which is the correct behavior.
+        let _ = rx.changed().await;


We only send headers in the case of a success, so this hangs on bad credentials, right?

google-cloud-rust/src/integration-tests/src/observability/auth.rs

Lines 129 to 132 in 79bc61a

Err(e) => {

tracing::warn!("Failed to refresh GCP credentials: {e:?}");

sleep(ERROR_RETRY_DELAY).await;

}

So should we:

match headers { // omitted... Err(e) if e.is_transient() => { tracing::warn!("Failed to refresh GCP credentials: {e:?}"); sleep(ERROR_RETRY_DELAY).await; }, Err(e) => break, }

Also, seems like this can't get hit anymore:

google-cloud-rust/src/integration-tests/src/observability/auth.rs

Lines 65 to 68 in 4d8249a

// If the first refresh hasn't completed yet, fail the request.

// The OTLP exporter is expected to handle this transient failure

// with its built-in retry mechanism.

Status::unauthenticated("GCP credentials not yet available")

This is getting complicated enough that we should probably defer it in a separate PR.

dbolduc · 2025-11-21T22:30:25Z

src/integration-tests/tests/telemetry.rs

+        );
+
+        // 4. Force flush to ensure spans are sent.
+        let _ = provider.force_flush();


Q: can we use ?. Are you trying to avoid workaround Timeout errors? https://docs.rs/opentelemetry_sdk/latest/opentelemetry_sdk/error/enum.OTelSdkError.html#variant.Timeout

…ice` and add GAX retry logic in telemetry test.

…loud Trace API verification.

…tFound` and improve error logging.

…remove manual delays in tests.

westarle marked this pull request as ready for review November 17, 2025 17:01

westarle requested a review from a team as a code owner November 17, 2025 17:01

dbolduc reviewed Nov 17, 2025

View reviewed changes

src/integration-tests/src/observability/cloud_trace.rs Outdated Show resolved Hide resolved

dbolduc reviewed Nov 17, 2025

View reviewed changes

src/integration-tests/src/observability/cloud_trace.rs Outdated Show resolved Hide resolved

westarle force-pushed the feat/telemetry-e2e-test branch 3 times, most recently from 5bc4d2d to 8dbfcbd Compare November 19, 2025 20:27

dbolduc reviewed Nov 20, 2025

View reviewed changes

src/integration-tests/tests/telemetry.rs Outdated Show resolved Hide resolved

dbolduc reviewed Nov 20, 2025

View reviewed changes

src/integration-tests/tests/telemetry.rs Outdated Show resolved Hide resolved

westarle requested a review from dbolduc November 21, 2025 16:53

dbolduc reviewed Nov 21, 2025

View reviewed changes

westarle force-pushed the feat/telemetry-e2e-test branch from 207fce3 to 79bc61a Compare November 21, 2025 21:41

dbolduc approved these changes Nov 21, 2025

View reviewed changes

westarle added 8 commits November 24, 2025 13:54

feat(integration-tests): implement CloudTraceClient

f934c25

test(integration-tests): finalize end-to-end telemetry test

f04782d

refactor: replace custom Cloud Trace client with generated `TraceServ…

2e204d7

…ice` and add GAX retry logic in telemetry test.

Deleted prototype cloud trace client.

2fb663b

fmt

923cb2a

refactor: Replace gax retry policy with a manual polling loop for C…

bee39a0

…loud Trace API verification.

fix: Retry trace fetching on Internal RPC errors in addition to `No…

13f13ed

…tFound` and improve error logging.

refactor: make auth interceptor await initial credential refresh and …

566fd1d

…remove manual delays in tests.

westarle force-pushed the feat/telemetry-e2e-test branch from 79bc61a to 566fd1d Compare November 24, 2025 18:54

dbolduc approved these changes Nov 24, 2025

View reviewed changes

westarle enabled auto-merge (squash) November 24, 2025 18:59

	Err(e) => {
	tracing::warn!("Failed to refresh GCP credentials: {e:?}");
	sleep(ERROR_RETRY_DELAY).await;
	}

	// If the first refresh hasn't completed yet, fail the request.
	// The OTLP exporter is expected to handle this transient failure
	// with its built-in retry mechanism.
	Status::unauthenticated("GCP credentials not yet available")

test(o11y): End-to-end HTTP to Cloud Trace integration test #3822

Are you sure you want to change the base?

test(o11y): End-to-end HTTP to Cloud Trace integration test #3822

Uh oh!

Conversation

westarle commented Nov 17, 2025

Uh oh!

codecov bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

westarle commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dbolduc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dbolduc left a comment

Choose a reason for hiding this comment

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

dbolduc Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Nov 17, 2025 •

edited

Loading

westarle commented Nov 21, 2025 •

edited

Loading