Open questions around `url.template` and `url.full`

[antonfirsov]

Area(s)

area:http

Is your change request related to a problem? Please describe.

#675 introduced a new http attribute url.template, which is supposed to be a low-cardinality variant of url.full, however I see several open questions around the expected behavior of uri.template in regards of implementation challanges and it's relation to url.full.

From .NET's HttpClient instrumentation POV we would need to have these questions resolved and the specification adjusted before considering implementing the attribute.

Relationship with url.full

According to the uri.full spec

Sensitive content provided in url.full SHOULD be scrubbed when instrumentations can identify it.

In my understanding an ideal scrubbing implementation would make url.full == url.template. If this is the case, do we still need both properties? If this is not the case, how exactly do we imagine url.full to look like after scrubbing?

url.template has been introduced to have a low cardinality url attribute, but does it mean that we are ok keeping url.full high-cardinality? Moreover, if an instrumented library is unable to scrub sensitive data is it the right choice to define url.full as Required from privacy POV?

Wouldn't it be wiser to rather merge the two properites and state that if an implementation is enlightened to templatize/sanitize url.full it should be emitted in a template form?

Redirects

According to the http client span spec

Instrumentations SHOULD create an HTTP span for each attempt to send an HTTP request over the wire.

in case a user provides a url.template and a redirect happens, what value do we expect to be logged for url.template for the redirect span? If the scrubbing mechanism for url.full is based on the template, what do we expect to be logged in url.full?

Describe the solution you'd like

One possible resolution of the listed problems is to define a standard templating syntax, allowing instrumented libraries to pattern-match a list of pre-defined templates to the actual uri, and log the template in url.full if a match is found. Do not log url.full, or log a placeholder value otherwise.

Another, simpler and cheaper solution is to:

• Log the template in url.full for the first request.
• Do not log url.full for redirects or log a placeholder value.

I don't see value keeping url.full and url.template separate with neither of my proposals. Moreover, regardless of the solution we pick, I'd recommend to make url.full Conditionally Required, allowing implementations to omit it if there is no scrubbing mechanism implemented.

/cc @klauco @trask @lmolkova @joaopgrassi

[lmolkova]

Relationship with url.full

From OTel side, we're more concerned with sensitive data in the query params - #860.

The generic scrubbing is not expected to templatize the URL, it's only expected to scrub sensitive params. We don't yet know how it'll look like (blocklist, allowlist, configurable, etc). Sanitized URL may look like this https://host:port/documents/42?api-version=1.2.3,foo=bar,secret=REDACTED.

Scrubbing process is not expected to remove all dynamic components from the url.full. The result of scrubbing is not intended to have low cardinality - it's not used on metrics.

Users benefit from having both - template and full url - full provides path and safe-to-record properties that are necessary for observability. It's great to also have url.template to group metrics.

TL;DR: url.template is a result of some transformation on url.full, but both are necessary for different reasons.

[lmolkova]

Redirects

In the perfect world we should have span/metric measurement per each request, i.e. each request would have different template.

If it's not always the case, then I'd say url.template should should be consistent with the url.full reported on the span.

[antonfirsov]

The result of scrubbing is not intended to have low cardinality - it's not used on metrics.

Does this mean that the addition of url.full to client metrics like http.client.request.duration will be never considered? Isn't it a problem to have high-cardinality attributes in distributed tracing?

Regarding url.template: considering #675 mentions metrics and the server metric has http.route, shouldn't url.template be added to the Http client metrics spec?

If it's not always the case, then I'd say url.template should should be consistent with the url.full reported on the span.

The implementation we are planning for .NET will omit both on redirects. I still think that url.full should be Conditionally Required given that it's not always possible to sanitize it.

[lmolkova]

Does this mean that the addition of url.full to client metrics like http.client.request.duration will be never considered? Isn't it a problem to have high-cardinality attributes in distributed tracing?

Correct, url.full cannot be added to metrics due to cardinality issues.
url.template is added to client metrics (in a separate table since it's experimental)

semantic-conventions/docs/http/http-metrics.md

Line 666 in c80cd1f

| [`url.template`](/docs/attributes-registry/url.md) | string | The low-cardinality template of an [absolute path reference](https://www.rfc-editor.org/rfc/rfc3986#section-4.2). [1] | `/users/{id}`; `/users/:id`; `/users?id={id}` | `Opt-In` |  |

[lmolkova]

The implementation we are planning for .NET will omit both on redirects. I still think that url.full should be Conditionally Required given that it's not always possible to sanitize it.

So there would be no information about the URL for redirects? Also, which span do you refer to when talking about redirects:

• first request(s) resulting in 30X and location header?
• the final request to the location?
• one span describing the whole sequence?

[antonfirsov]

one span describing the whole sequence

There is no such span in SocketsHttpHandler, only separate spans for each request.

which span do you refer to when talking about redirects

I meant the auto-redirect span(s) following the initial 30x response.

So there would be no information about the URL for redirects?

Actually, we are considering multiple sanitization mechanisms for .NET 9. One of them is to replace url.full with url.template.
This means that when redirect happens, the original url.template becomes invalid so logging that would be problematic. There will be other options available which will allow logging the url on redirects: log the uriginal one (for debugging), without query string, do not log.

Note that currently no attributes are emitted for the http spans, adding them is part of dotnet/runtime#93019. I would only consider adding url.full if we can do it in a safe and future-proof manner.

[lmolkova]

Actually, we are considering multiple sanitization mechanisms for .NET 9. One of them is to replace url.full with url.template.
This means that when redirect happens, the original url.template becomes invalid so logging that would be problematic. There will be other options available which will allow logging the url on redirects: log the uriginal one (for debugging), without query string, do not log.

Note that currently no attributes are emitted for the http spans, adding them is part of dotnet/runtime#93019. I would only consider adding url.full if we can do it in a safe and future-proof manner.

So If I understand correctly:

• redirects
  • span1: url1, status_code: 30X
  • span2: url2, status_code: e.g. 200

there will be multiple modes allowing to:

1. use template1 (instead of url) on span1, but not possible on span2
2. use original url1 on span1 and original url2 on span2
   a. remove query string from URLs
   b. don't include URLs at all

I'd like to suggest the following:

• if you can add a template, always add it as a url.template. If you can't add it - don't add it.
• always add url.full on spans - it's required, without both - url and template, HTTP client spans are kinda useless?
  • if recording URL is not safe, record it as http(s)://host:port/REDACTED/REDACTED/REDACTED?a=REDACTED (or http(s)://host:port/*/*/*?a=*, etc
  • let users record full urls without sanitization - they can do sanitization later on in the telemetry pipeline

Essentially, don't consider url and template as mutually exclusive - they should be treated independently.

[antonfirsov]

So If I understand correctly

Yes, you understood it correctly.

http(s)://host:port/REDACTED/REDACTED/REDACTED?a=REDACTED

Building that string does not feel to be worthy of the cost, it would be simpler just to emit http(s)://host:port/REDACTED. In some cases even http(s)://REDACTED might be desirable. But honestly, I don't see how does it provide more information to the user over just omitting url.full.

if you can add a template, always add it as a url.template

Do you mean by this that if a template is provided by the user, we should only add it as url.template and not substitute it into url.full? Should url.full be the http(s)://REDACTED thingie then? I don't see how/why is this better.

without both - url and template, HTTP client spans are kinda useless?

With proper usage, url and/or template would be present in most cases. Redirections are very rare in server to server communication. If they happen during a strict logging setup, the spans would still indicate their presence. Users can switch to less restrictive logging modes to debug them.

[noahfalk]

if you can add a template, always add it as a url.template. If you can't add it - don't add it.

By 'template' I assume this means only strings that we have a specific expectation are low cardinality right? For example if we had some hypothetical API like this:

HttpRequestMessage.SetLoggedUrl(string urlToLog)

Users could use that API to give any string they wanted, not necessarily a low cardinality one. I assume that string would not count as a template and we should log it as the sanitized url.full, not url.template?

if you can add a template ... If you can't

I'm not sure what determines whether we can or can't? It feels like the spec implies url.template should relate to the URL the request was sent to, but exactly what is allowable and who is being proposed to enforce that relationship wasn't clear to me. A couple examples:

1. The app sends a request to "contuso.com/userId=50" and the app code specifies that url.template="USER_PAGE". Did the dev mess up? Was the http client expected to do verification on it? Perhaps this is perfectly legitimate use of url.template despite not being so close to the spec examples.
2. The app sends a request to "contuso.com/userId=180" which is redirected to "new.contuso.org/billingapp/users/id=180". The app developer's code specified url.template = "/userId={id}" and it was invisible to that code that a redirect even happened. Is it legal for the 2nd span tracking the redirected request to also include the same url.template value as the first span?

[lmolkova]

Building that string does not feel to be worthy of the cost, it would be simpler just to emit http(s)://host:port/REDACTED. In some cases even http(s)://REDACTED might be desirable. But honestly, I don't see how does it provide more information to the user over just omitting url.full.

having url.full=https://host:port/REDACTED is still better than dropping it. It tells users:

1. It's a valid HTTP span (along with other attributes), without url.full it's not a valid HTTP client span.
2. There was url.full populated, it's not a bug of the instrumentation which forgot/failed to stamp it.
3. There are some sanitization settings that are applied that resulted in redaction, users can explore how to change them