m

Author: ajewellamz

TestVectors/dafny/DDBEncryption/src/TestVectors.dfy

@@ -761,10 +761,10 @@ module {:options "-functionSyntax:4"} DdbEncryptionTestVectors {
       TestBucketQueries(rClient, 5, GetBucketQuery51(), trans, "bucket query 51");
       TestBucketQueries(rClient, 5, GetBucketQuery23(), trans, "bucket query 23");
 
-      TestBucketQueries(rClient, 1, GetBucketQuery15F(), trans, "bucket query 15F", false);
-      TestBucketQueries(rClient, 2, GetBucketQuery25F(), trans, "bucket query 25F", false);
-      TestBucketQueries(rClient, 3, GetBucketQuery35F(), trans, "bucket query 35F", false);
-      TestBucketQueries(rClient, 4, GetBucketQuery45F(), trans, "bucket query 45F", false);
+      TestBucketQueries(rClient, 1, GetBucketQuery15F(), trans, "bucket query 15F");
+      TestBucketQueries(rClient, 2, GetBucketQuery25F(), trans, "bucket query 25F");
+      TestBucketQueries(rClient, 3, GetBucketQuery35F(), trans, "bucket query 35F");
+      TestBucketQueries(rClient, 4, GetBucketQuery45F(), trans, "bucket query 45F");
 
       var scanCount4 := 0;
       var scanCount5 := 0;

specification/changes/2025-08-25-bucket-beacons/background.md

@@ -4,7 +4,7 @@
 # Bucket Beacons
 
 `Bucket Beacons` refers to a way to add a little bit more randomness to your [beacons](../../searchable-encryption/beacons.md),
-so they will be less likely to leak information when your data distribution is uneven.
+to add anonymity when your data distribution is uneven
 
 Probably read [changes](./change.md) first, as it gives a brief overview of the interface.
 
@@ -23,7 +23,7 @@ In this case, some hashes will have many many occurrences, and some will have on
 
 An external observer can look at census data and make an educated guess that the hashes with many occurrences are probably "Jones" or "Smith". Pretty soon, you've leaked real information.
 
-## Enter Bucket Beacons
+## Introducing Bucket Beacons
 
 One strategy to combat this is to further divide each item's beacons into separate `buckets`,
 so that the same value in different records might produce different hashes.
@@ -41,14 +41,15 @@ The hash for a value in bucket number zero is the same as the hash in an unbucke
 therefore, there is no difference between "one bucket" and "not using buckets".
 
 Unfortunately, when items are distributed across N buckets, retrieving all of them requires N separate queries.
+
 - Only the `Query` operation is affected.
 - `Scan` and `Get` operations continue to work as usual.
 
 The reason Query behaves differently is that it searches against an index, which requires an exact match. This leaves no opportunity to optimize the query with a “this OR that OR other” approach, unlike a Scan which examines all items.
 
 Note: There is no way to determine an item’s bucket just by looking at its encrypted value.
 
-### Performance
+### Performance Penalties
 
 Multiple queries will always take longer than one query, however;
 if the number of "pages" of results returned by DynamoDB for a query is large compared to the number of buckets,
@@ -57,6 +58,18 @@ then the overall query performance is not impacted very much.
 If the query is only expected to return one result, then of course, making five queries will take five time as long,
 with four of the queries returning nothing.
 
+### Performance Advantages
+
+On the other hand, bucket beacons can provide performance enhancements as well.
+
+Sometimes a common value can share the same hash as a rare one.
+In this case, a search for the rare one pays the penalty of retrieving and discarding all the
+matches for the common one.
+Bucket beacon reduces the maximum number occurrences of a single hash, reducing this penalty.
+
+Increasing the number of buckets can allow you to increase the length of a standard beacon,
+further decreasing the number of results that must be retrieved and discarded.
+
 ### Constrained Beacons
 
 Maybe you need a different number of buckets for different standard beacons.
@@ -245,3 +258,64 @@ which tells the interceptor to calculate beacon A for bucket 4 and beacon B for
 If you know exactly what data has been written, a long enough list of these would find all of the items.
 
 We don't want to implement this until we're sure someone would actually use it.
+
+## Migration
+
+A non-bucketed table is the same as a table with `maximumNumberOfBuckets == 1`.
+
+Further, each individual beacon can either be considered to be unconstrained,
+or constrained to one bucket.
+
+Given that, migration from non-bucketed to bucketed follows the same rules as any other change in bucket settings.
+
+You can always increase maximumNumberOfBuckets.
+It won't magically improve the anonymity of you existing data,
+but new data will be properly anonymized and both old and new data will be found when doing bucketed searches.
+
+The individual beacon constraints must remain unchanged.
+This when you first move to multiple bucket,
+each beacon must either be constrained to one bucket, or be unconstrained.
+
+## Test Strategy
+
+### Normal Operation
+
+Create a table with a maximum of 5 buckets, a variety of beacons with different numbers of buckets,
+and with GSIs built on a variety of combinations of those indexes.
+
+Put 100 items in that table, with different PK attributes,
+but the exact same values for all the other attributes.
+We expect around 20 items per bucket, and are pretty guaranteed that no bucket is empty.
+
+Perform a variety of queries against the table.
+For each, assert that
+
+- GetNumberOfQueries returns the expected value
+- When performing `GetNumberOfQueries` queries,
+  - each bucket returns at least one value
+  - every item is returned exactly one
+- When performing queries with the bucket number set to `GetNumberOfQueries` or greater, an error is returned.
+
+Perform a very large number of Scans against the table.
+These are easier to test, as they do not require a different index for each one, as the Query ones do.
+Test that a single Scan returns every item exactly once.
+
+### Test Bucket Selector
+
+Repeat [Normal Operation](#normal-operation), but
+
+- add an attribute to hold a bucket number
+- include a [Bucket Selector](../../searchable-encryption/search-config.md#bucket-selector)
+  that puts each item in the indicated bucket.
+- when searching on a bucket, assert that the item was in the correct bucket.
+
+### Test Filter Expressions
+
+The functionality of the [Filter Expressions for Query](../../dynamodb-encryption-client/ddb-support.md#filter-expressions-for-query) is well tested by the above.
+
+For a few scan operations, assert that the text of the calculated filter expressions are as expected.
+
+### Test Compound Beacons
+
+Create some complex compound beacons, create indexes with them,
+and repeat the same test as in [Normal Operation](#normal-operation).

specification/dynamodb-encryption-client/ddb-get-number-of-queries.md

@@ -0,0 +1,39 @@
+[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved."
+[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0"
+
+# Get Number of Queries
+
+## Overview
+
+When using [Bucket Beacons](../changes/2025-08-25-bucket-beacons/background.md),
+more than one [query](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html)
+can be necessary to retrieve all of the desired results,
+leading to code something like this:
+
+```text
+    for i = 0 to transformClient.GetNumberOfQueries(query)
+       query.ExpressionAttributeValues.Add(":aws_dbe_bucket", N(i.to_string())
+       dynamoClient.query(query)
+```
+
+## Operation
+
+### Input
+
+This operation MUST take as input the QueryInput structure under consideration.
+
+This operation MUST return the number of queries necessary.
+
+### Behavior
+
+Based on the [standard beacons](../searchable-encryption/beacons.md#standard-beacon)
+used in the [KeyConditionExpression](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.KeyConditionExpressions.html) of the query, calculate the required number of queries.
+
+This value is the minimum of the calculated value,
+and the [maximum number of buckets](../searchable-encryption/search-config.md#max-buckets)
+configured for the table.
+
+The calculated value is the least common multiple of the number of buckets for each of the beacons involved.
+
+This is not needed for a [scan](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html)
+operation. Only one Scan is needed, regardless of bucket settings.

specification/dynamodb-encryption-client/ddb-support.md

@@ -78,6 +78,10 @@ and [Encrypt Item Output](./encrypt-item.md#output).
 To obtain [Beacon Key Materials] GetEncryptedBeacons
 MUST call [Get beacon key after encrypt](../searchable-encryption/search-config.md#get-beacon-key-after-encrypt).
 
+A [bucket number](search-config.md#bucketnumber) MUST be generated
+by calling the [bucket selector](search-config.md#bucket-selector).
+This [bucket number](search-config.md#bucketnumber) is to be used for all [standard beacons](./searchable-encryption/beacons.md#standard-beacon) in the item.
+
 GetEncryptedBeacons MUST NOT operate on [compound beacons](../searchable-encryption/beacons.md#compound-beacon)
 that only have [signed parts](../searchable-encryption/beacons.md#compound-beacon-initialization).
 
@@ -154,6 +158,18 @@ from [Get beacon key for query](../searchable-encryption/search-config.md#get-be
 If the [QueryObject does not have encrypted values](#queryobject-has-encrypted-values)
 then QueryInputForBeacons MUST NOT attempt to obtain [Beacon Key Materials](../searchable-encryption/search-config.md#beacon-key-materials).
 
+When querying, a [bucket number](search-config.md#bucketnumber) MUST be determined by examining
+the `:aws_dbe_bucket` value in the `ExpressionAttributeValues`.
+
+If this value is absent, a bucket number of `0` MUST be used.
+
+If this value is not of type `N` or fails to hold an integer value
+greater than or equal to zero and less than the [max buckets](search-config.md#max-buckets),
+an error MUST be returned.
+
+If this value is valid, then this value is used and the `:aws_dbe_bucket` field MUST
+be removed from the `ExpressionAttributeValues`.
+
 For any operand in the KeyConditionExpression or FilterExpression which is a beacon name,
 the name MUST be replaced by the internal beacon name (i.e. NAME replaced by aws_dbe_b_NAME).
 
@@ -174,6 +190,9 @@ MUST be obtained from the [Beacon Key Materials](../searchable-encryption/search
 [HMAC Keys map](../searchable-encryption/search-config.md#hmac-keys) using the beacon name
 as the key.
 
+If [Bucket Beacons](../changes/2025-08-25-bucket-beacons/background.md) are being used,
+then the [FilterExpression](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.FilterExpression.html) must be augmented as described in [Filter Expressions for Query](#filter-expressions-for-query).
+
 For example if the query is
 "MyBeacon = :value" and ExpressionAttributeValues holds (:value = banana),
 then the ExpressionAttributeValues must be changed to (:value = 13fd),
@@ -192,6 +211,31 @@ in this situation, but that risks leaking the connection between the two beacons
 Similarly, if one of the two was not a beacon, then we would be leaking the fact that
 this beacon came from that text.
 
+### Filter Expressions for Query
+
+if [GetNumberOfQueries](./ddb-get-number-of-queries.md) returns a number less than the configured
+[maximum number of buckets](../searchable-encryption/search-config.md#max-buckets)
+then the FilterExpression MUST be augmented to match against buckets greater than
+the limit returned from GetNumberOfQueries.
+
+For each bucket number that would map to the current bucket,
+calculate the Filter Expression as for that bucket.
+The FilterExpression sent to DynamoDB MUST be the `OR` combination of all of these expressions.
+
+The text of the FilterExpression is unlikely to change between buckets.
+What will change is the values in the ExpressionAttributeValues,
+which will be different if they involve standard beacons calculated with different buckets.
+This implies that additional unique values will need to be added to ExpressionAttributeValues.
+
+As an example, if a table is configured with five buckets,
+and GetNumberOfQueries returns two, and `foo[n]` represents the expression as calculated for bucket `n`,
+then when `:aws_dbe_bucket = 0` the filter expression must be `(foo[0]) OR (foo[2]) OR (foo[4])`
+ands when `:aws_dbe_bucket = 1` the filter expression must be `(foo[1]) OR (foo[3])`.
+
+The resulting FilterExpression might look something like this:
+
+`(aws_dbe_b_Attr3 = :attr3 AND aws_dbe_b_Attr4 = :attr4) OR (aws_dbe_b_Attr3 = :attr3AA AND aws_dbe_b_Attr4 = :attr4AA) OR (aws_dbe_b_Attr3 = :attr3AB AND aws_dbe_b_Attr4 = :attr4AB)`
+
 ### QueryObject has encrypted values
 
 Determines if a Query Object has encrypted values (ENCRYPT_AND_SIGN fields)
@@ -260,7 +304,9 @@ any error encountered during filtering MUST result in a failure of the query ope
 
 Transform an unencrypted ScanInput object for searchable encryption.
 
-The ScanInput is transformed in the same way as [QueryInputForBeacons](#queryinputforbeacons).
+The ScanInput is transformed in the same way as [QueryInputForBeacons](#queryinputforbeacons),
+except that [Filter Expressions for Query](#filter-expressions-for-query) is calculated
+as if GetNumberOfQueries returned `1`.
 
 ## ScanOutputForBeacons
 

specification/searchable-encryption/beacons.md

@@ -5,10 +5,18 @@
 
 ## Version
 
-1.0.0
+1.1.0
 
 ### Changelog
 
+- 1.1.0
+  - add bucket beacons
+
+## Conventions used in this document
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
+in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119).
+
 ## Overview
 
 Beacons use stable hashes of plaintext values of encrypted fields
@@ -276,11 +284,6 @@ If `NAME` appears in an record to be written,
 and `NAME` can also be constructed from other parts of the record,
 then the write must fail if the constructed and supplied values are not equal.
 
-### Conventions used in this document
-
-The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
-in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119).
-
 ### Standard Beacon Initialization
 
 On initialization of a Standard Beacon, the caller MUST provide:
@@ -291,13 +294,17 @@ On initialization of a Standard Beacon, the caller MUST provide:
 On initialization of a Standard Beacon, the caller MAY provide:
 
 - a [terminal location](virtual.md#terminal-location) -- a string
-- a [beacon style](beacon-style-initialization)
+- a [beacon style](#beacon-style-initialization)
+- a [number of buckets](#beacon-constraint)
 
 If no [terminal location](virtual.md#terminal-location) is provided,
 the `name` MUST be used as the [terminal location](virtual.md#terminal-location).
 
 Initialization MUST fail if two standard beacons are configured with the same location.
 
+Initialization MUST fail if [number of buckets](#beacon-constraint) is specified, and is greater than or equal to
+the maximum number of buckets specified in the [beacon version](search-config.md#beacon-version-initialization).
+
 ### Beacon Style Initialization
 
 On initialization of a Beacon Style, the caller MUST provide exactly one of
@@ -401,6 +408,19 @@ This name MUST match the name of one of the [encrypted](#encrypted-part-initiali
 These parts may come from these locally defined parts lists, or from the
 [Global Parts List](search-config.md#global-parts-list), in any combination.
 
+#### Beacon Constraint
+
+A beacon may be constrained to fewer buckets than is specified in the [beacon version](search-config.md#beacon-version-initialization).
+
+If an item is being written or queried as bucket `X`, but the [standard beacon](#standard-beacon-initialization) is constrained to only `N` buckets,
+then the bucket used to [encode](#bucket-beacon-encoding) the beacon MUST be `X % N`, where `%` is the modulo or remainder operation.
+
+Examples:
+
+- If a beacon is constrained to one bucket, then it is always encoded as bucket `0`.
+- If a beacon is constrained to three buckets, then the beacon is always encoded as bucket `0`, `1` or `2`.
+  If the current bucket is `7`, then the beacon is encoded as for bucket `1`.
+
 ### Default Construction
 
 - If no constructors are configured, a default constructor MUST be generated.
@@ -471,11 +491,23 @@ Both standard and compound beacons define two operations
 
 - This operation MUST convert the attribute value of the associated field to
   a sequence of bytes, as per [attribute serialization](../dynamodb-encryption-client/ddb-attribute-serialization.md).
+- The serialized form MUST be augmented as per [bucket beacon encoding](#bucket-beacon-encoding).
 - This operation MUST return the [basicHash](#basichash) of the resulting bytes and the configured [beacon length](#beacon-length).
 - The returned
   [AttributeValue](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html)
   MUST be type "S" String.
 
+#### Bucket Beacon Encoding
+
+Bucket Beacon Encoding converts a sequence of bytes into a possibly different sequence of bytes.
+
+To calculate the [bucket number](search-config.md#bucketnumber) for this beacon by calculating `X % N`
+where `X` is the bucket specified and `N` is the [number of buckets](#beacon-constraint) specified for the beacon.
+
+If this number is zero, then the input sequence of bytes MUST be returned unchanged.
+
+Otherwise, a single byte with a value equal to this calculated bucket number, MUST be appended to the input sequence of bytes.
+
 ### value for a set standard beacon
 
 - This operation MUST convert the value of each item in the set to