chore: introduce metrics change by josecorella · Pull Request #307 · awslabs/aws-encryption-sdk-specification

josecorella · 2025-09-15T16:05:33Z

For best reading and commenting experience, I suggest splitting your window in two; the review page and the rendered page.
Here are the rendered files:

Goals for 9-15-2025 Spec Review:

Agreement on optional metrics agents and how they will impact existing APIs
Agreement that Metrics Agent Interface and Implementation will be implemented in Dafny but only as wrappers and provide extern implementations to make moving off of Dafny easier.
Agreement on interface supported operations.

NOTE: name change from metrics-agent -> metrics-worker

Goals for 2-4-2025 Spec Review:

Agreement on interface supported operations.
Agreement that Metrics Agent Interface and Implementation will be implemented in Dafny but only as wrappers and provide extern implementations to make moving off of Dafny easier.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Check any applicable:

Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

kessplas · 2025-09-15T20:07:40Z

changes/2025-09-12_introduce-metrics-interface/background.md

+metrics. Customers can then ask for updates to the implementations
+CT provides or customers can go an implement their own interfaces that are fine-tuned
+to their use cases.
+


Is there a middle ground? For example, logging information locally without uploading it anywhere.

changes/2025-09-12_introduce-metrics-interface/background.md

changes/2025-09-12_introduce-metrics-interface/change.md

framework/metrics-agent.md

changes/2025-09-12_introduce-metrics-interface/background.md

changes/2025-09-12_introduce-metrics-interface/change.md

framework/metrics-agent.md

texastony · 2025-09-15T20:13:07Z

@josecorella I have a meta/macro question.
Is there a proposal doc that accompanies these change docs?

I appreciate that the Background doc highlights issues and alternatives, but I feel like we a missing a "User Stories" document, that can be used to measure success criteria and what are the table stakes of this work.

It is also possible I just missed such a proposal doc; but without it, it is difficult to work backwards.

changes/2025-09-12_introduce-metrics-interface/background.md

seebees · 2025-09-15T20:14:08Z

framework/metrics-agent.md

+operation AddDate {
+  input: AddDateInput,
+  output: AddOutput,
+  errors: [MetricsPutError]


Nit: You say that the interface should not error, but you have errors here.

seebees · 2025-09-15T20:14:43Z

framework/metrics-worker.md

+// Common output structure
+structure AddOutput {}


I get why you might optimize this. But is this really the best choice? why not have a output per operation?

revisit this in discussion on 2026-02-04

changes/2025-09-12_introduce-metrics-interface/background.md

changes/2025-09-12_introduce-metrics-interface/change.md

texastony · 2025-09-16T00:36:50Z

Potential Issue/Alternative/User Story:

Users of the MPL/ESDK/DB-ESDK are also users of the AWS SDKs. The AWS SDKs have established logging and metric interfaces.
AWS Crypto Tools products are AWS Products, just like the AWS SDKs.

There likely is an implicit customer expectation that Crypto Tools products behave and appear to be consistent with the AWS SDKs.

Therefore, I suggest we carefully evaluate if we can utilize the SDKs metric and logging tooling, and offer a customer experience that closely mimics the SDKs experience.

The current collection of docs does not state this as a goal, but it does leave it open as an opportunity.

i.e: the proposed metric interface could wrap an SDK metric class.

texastony · 2026-01-31T03:26:46Z

changes/2025-09-12_introduce-metrics-interface/change.md

+        Client->>Client: Content Encryption
+    end
+    Client<<->>CMM: GetEncryption/Decryption Materials
+    CMM<<->>Keyring: OnEncrypt/OnDecrypt


I think the doc implies this, but we can go deeper down the stack.

i.e: Keyring <<->> KMS or Keyring <<->> Branch Key Store.

Technically, the H-Keyring is:
Keyring <<->> (Cache | Branch Key Store <<->> (DDB & KMS))

I agree that we could go dive deeper; however, in the interest of this change I don't want to go there besides painting the most basic model for the reader.

But we need the CMC here.

kessplas · 2026-02-04T19:04:33Z

changes/2025-09-12_introduce-metrics-interface/background.md

+- Metrics: Throughout this document and other related documents the word, "metrics" is used extensively.
+  For Crypto Tools' libraries metrics means two things.
+
+      1. Measuring application performance, (e.g. api requests, cache performance, latency).


nit: this doesn't render as a list, it renders as a code block

vscode render preview lied to me....

texastony · 2026-02-04T19:06:00Z

changes/2025-09-12_introduce-metrics-interface/background.md

I disagree about implementing this in Dafny.
I am concerned that putting additional synchronization blocks into our code base will negatively impact performance, and I do not see how a metric interface could exist in Dafny without a synchronization block.

What do you mean by "synchronization blocks"?

I don't think we'd need sync blocks - why can't externs fire (a new thread) and forget?

texastony · 2026-02-04T19:07:06Z

changes/2025-09-12_introduce-metrics-interface/background.md

+versions of these libraries have no logging or metrics publishing
+to either a local application or to an observability service like AWS CloudWatch.
+
+As client side encryption libraries emitting metrics must be done carefully as


Suggested change

As client side encryption libraries emitting metrics must be done carefully as

As client side encryption libraries, emitting metrics must be done carefully as

seebees · 2026-02-04T19:08:40Z

changes/2025-09-12_introduce-metrics-interface/background.md

I wonder about errors. How should or to what extent should errors show up? Can we log the stacktrace or is that an anti-pattern?

kessplas · 2026-02-04T19:09:22Z

changes/2025-09-12_introduce-metrics-interface/change.md

+| ESDK    | T.B.D              | [ESDK.smithy](https://github.com/aws/aws-encryption-sdk/blob/mainline/AwsEncryptionSDK/dafny/AwsEncryptionSdk/Model/esdk.smithy)                                                                                   |
+| MPL     | T.B.D              | [material-provider.smithy](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/dafny/AwsCryptographicMaterialProviders/Model/material-provider.smithy) |
+| DB-ESDK | T.B.D              | [DynamoDbEncryption.smithy](https://github.com/aws/aws-database-encryption-sdk-dynamodb/blob/main/DynamoDbEncryption/dafny/DynamoDbEncryption/Model/DynamoDbEncryption.smithy)                                     |
+


i forgot s3ec

texastony · 2026-02-04T19:09:49Z

changes/2025-09-12_introduce-metrics-interface/background.md

+A popular feature request has been for in depth insights into CT libraries. Many customers
+ask for suggestions on how to reduce network calls to AWS Key Management Service (AWS KMS) and
+followup questions around cache performance.


Issue: "has been for" is rough phrase.
Ideally,
we would quantify the customer demand (x internal services, y external customers) but that takes time and is not worth it;
we all know that the demand is there.

Suggested change

A popular feature request has been for in depth insights into CT libraries. Many customers

ask for suggestions on how to reduce network calls to AWS Key Management Service (AWS KMS) and

followup questions around cache performance.

There is customer demand for in depth insights into CT libraries. Many customers

ask for suggestions on how to reduce network calls to AWS Key Management Service (AWS KMS) and

followup questions around cache performance.

rishav-karanjit · 2026-02-04T19:10:28Z

changes/2025-09-12_introduce-metrics-interface/background.md

+
+### Issue 2: Should Data Plane APIs fail if metrics fail to publish?
+
+#### No (recommended)


I might want to at-least throw a warning?

kessplas · 2026-02-04T19:10:28Z

changes/2025-09-12_introduce-metrics-interface/change.md

+
+## Requirements
+
+The interface should have three requirements.


there's 2 listed?

texastony · 2026-02-04T19:10:53Z

changes/2025-09-12_introduce-metrics-interface/background.md

+
+This allows customers to test how their applications behave when they start to emit
+metrics. Customers can then ask for updates to the implementations
+CT provides or customers can go an implement their own interfaces that are fine-tuned


Suggested change

CT provides or customers can go an implement their own interfaces that are fine-tuned

CT provides or customers can go and implement their own interfaces that are fine-tuned

ShubhamChaturvedi7 · 2026-02-04T19:13:00Z

changes/2025-09-12_introduce-metrics-interface/change.md

+metrics to this one worker and to only sometimes capture metrics to this
+other worker.
+
+#### No (recommended)


I'd say yes to allow metrics on client construction. It's painful (and easy to forget) to supply the agent on every call. We should keep it optional, but should provide a way to set once and forget.

kessplas · 2026-02-04T19:13:02Z

changes/2025-09-12_introduce-metrics-interface/change.md

+        this._bufferSize = builder._bufferSize;
+        this._instructionFileConfig = builder._instructionFileConfig;
+        this._commitmentPolicy = builder._commitmentPolicy;
+       this._metricsWorker = builder._metricsWorkerl


Suggested change

+ this._metricsWorker = builder._metricsWorkerl

+ this._metricsWorker = builder._metricsWorker;

josecorella · 2026-02-04T19:13:14Z

changes/2025-09-12_introduce-metrics-interface/change.md

+
+## Requirements
+
+The interface should have three requirements.


Suggested change

The interface should have three requirements.

The interface should have two requirements.

kessplas · 2026-02-04T19:13:33Z

framework/metrics-worker.md

+| ESDK    | T.B.D              | [ESDK.smithy](https://github.com/aws/aws-encryption-sdk/blob/mainline/AwsEncryptionSDK/dafny/AwsEncryptionSdk/Model/esdk.smithy)                                                                                   |
+| MPL     | T.B.D              | [material-provider.smithy](https://github.com/aws/aws-cryptographic-material-providers-library/blob/main/AwsCryptographicMaterialProviders/dafny/AwsCryptographicMaterialProviders/Model/material-provider.smithy) |
+| DB-ESDK | T.B.D              | [DynamoDbEncryption.smithy](https://github.com/aws/aws-database-encryption-sdk-dynamodb/blob/main/DynamoDbEncryption/dafny/DynamoDbEncryption/Model/DynamoDbEncryption.smithy)                                     |
+


kessplas · 2026-02-04T19:14:02Z

framework/metrics-worker.md

+
+### count
+
+A count is a long value


Suggested change

A count is a long value

A count is a long value.

texastony · 2026-02-04T19:14:10Z

changes/2025-09-12_introduce-metrics-interface/background.md

+of the issues that are described above, (e.g. handling failing requests, perform
+blocking requests to CT libraries, use a separate thread/thread pool that handles
+these request). By providing a wrapper around a language's most popular logging


Issue:

Agreement that Metrics Agent Interface and Implementation will be implemented in Dafny but only as wrappers and provide extern implementations to make moving off of Dafny easier.

I am concerned that Dafny will not allow for non-blocking requests as it does not have async syntax, nor does it have concurrent syntax.

texastony · 2026-02-04T19:16:26Z

changes/2025-09-12_introduce-metrics-interface/change.md

+This list is not exhaustive. Any downstream consumer of any API or client configuration SHOULD
+also be updated as part of this proposed changed.
+
+| API/ Configuration                                                                                                                                                                |


Issue: This list is missing the Cryptographic Materials Cache's APIs/Configuration.
I strongly believe that we should offer insights into cache hits/misses, and therefore need to support the CMC's operations.

kessplas · 2026-02-04T19:22:07Z

changes/2025-09-12_introduce-metrics-interface/change.md

+  @required
+  materials: EncryptionMaterials,
+
+ metricsWorker: aws.cryptography.materialProviders#MetricsWorkerReference


I wonder if this should be a Metrics instance rather than the worker?

ShubhamChaturvedi7 · 2026-02-04T19:24:07Z

framework/metrics-worker.md

+@extendable
+resource MetricsWorker {
+  operations: [
+    AddDate,


How do we integrate these with something like CoralMetricsWorker?

ShubhamChaturvedi7 · 2026-02-04T19:24:59Z

framework/metrics-worker.md

+// Common output structure
+structure AddOutput {}
+
+@aws.polymorph#reference(resource: MetricsWorker)


Do we want to use these custom traits given that we are planning to move away from custom smithy-dafny?

kessplas · 2026-02-04T19:43:50Z

framework/metrics-worker.md

+  transactionId: String
+}
+
+// Common output structure


In at least Coral Metrics and SLF4J (not sure about other languages) all of these API operations return void. I figure we're doing this for future extensibility, but it might be a little annoying for implementors, as now they need to return something instead of void. This could setup NPEs way down the line.

josecorella added 7 commits September 5, 2025 15:53

chore: introduce metrics interface in change doc

ccd27f7

finished background doc

402d3cd

more work on the change doc

0fcc10e

add metrics-agent interface file

97380ef

finish first draft of metrics interface

a42a891

prettify

61fe175

update

e95b0f2

josecorella requested a review from a team as a code owner September 15, 2025 16:05

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Outdated Show resolved Hide resolved

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Outdated Show resolved Hide resolved

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Show resolved Hide resolved

seebees reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/change.md Show resolved Hide resolved

seebees reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/change.md Show resolved Hide resolved

seebees reviewed Sep 15, 2025

View reviewed changes

framework/metrics-agent.md Outdated Show resolved Hide resolved

lucasmcdonald3 reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Outdated Show resolved Hide resolved

changes/2025-09-12_introduce-metrics-interface/change.md Outdated Show resolved Hide resolved

framework/metrics-agent.md Outdated Show resolved Hide resolved

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Show resolved Hide resolved

seebees reviewed Sep 15, 2025

View reviewed changes

texastony reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/background.md Show resolved Hide resolved

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/change.md Show resolved Hide resolved

ShubhamChaturvedi7 reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/change.md Show resolved Hide resolved

kessplas reviewed Sep 15, 2025

View reviewed changes

changes/2025-09-12_introduce-metrics-interface/change.md Show resolved Hide resolved

updates for february read:

ca37826

texastony reviewed Jan 31, 2026

View reviewed changes

udpates

2f17445

kessplas reviewed Feb 4, 2026

View reviewed changes

texastony reviewed Feb 4, 2026

View reviewed changes

seebees reviewed Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

texastony reviewed Feb 4, 2026

View reviewed changes

rishav-karanjit reviewed Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

texastony reviewed Feb 4, 2026

View reviewed changes

ShubhamChaturvedi7 reviewed Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

josecorella commented Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

texastony reviewed Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

ShubhamChaturvedi7 reviewed Feb 4, 2026

View reviewed changes

kessplas reviewed Feb 4, 2026

View reviewed changes

	As client side encryption libraries emitting metrics must be done carefully as
	As client side encryption libraries, emitting metrics must be done carefully as


		### Issue 2: Should Data Plane APIs fail if metrics fail to publish?

		#### No (recommended)


		## Requirements

		The interface should have three requirements.

	CT provides or customers can go an implement their own interfaces that are fine-tuned
	CT provides or customers can go and implement their own interfaces that are fine-tuned

	+ this._metricsWorker = builder._metricsWorkerl
	+ this._metricsWorker = builder._metricsWorker;

	The interface should have three requirements.
	The interface should have two requirements.

Conversation

josecorella commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Check any applicable:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

texastony commented Sep 15, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

texastony commented Sep 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

josecorella commented Sep 15, 2025 •

edited

Loading