Question : how to random access or skip during decryption

[mkeskells]

Problem:

I have a need to handle large files and service a random address range for decryption.
Looking throgh the exampes I dont see an effecient way to do this.
It seems that the underlying encryption data is framed, but the CryptoInputStream doesn't skip in an input stream, and there isnt a support for a Readable Channel that I am aware of

Solution:

With the existing API - I could use skip in the InputStream, but this will (as I see it) decrypt and throw away the skipped bytes, with extra I/O and CPU load
With an enhanced implementation we could skip on the input stream to the start of a frame and then decrypt that frame (skipping internally to the offset needed by decodng and thowin a few bytes if needed)

Out of scope:

[texastony]

@mkeskells

Alas,
the AWS Encryption SDK does not support this use case,
at least not with the message format as it is today.

AWS Crypto Tools has considered designing and implementing a new message format with Merkle trees which would allow partial decryption.
But we have not picked that up.

If you would like, I can turn this Question into a feature request; but I cannot promise that we will pick it up any time soon.

Much Obliged,
AWS Crypto Tools

[mkeskells]

Hi @texastony
I would be happy to work on supporting a format that allows for some support for random/streamed access. Considering a format that could cope with playing an encrypted video, or to support a service that could cope with get bytes x to y.

I would likely need someone to guide me. I have implemented a custom master key provider but dont know the library well enough to be let loose on my own without some support :-)

A couple of questions though -

What is the rationalle for supporting the framed format? It seems that this is the preferred format but I dont see what the advantage is if the frames cant be handled individually so I am probably missing some understanding

Its seems form looking at the code that the frame is constant unencrypted size (apart from the last frame), but I presume that the encrypted size is variable and maybe not predictable from just the input size - I dont know behaviour of the encryption cyphers.
If that is the case then the dead reconning of the start of the frame isnt just maths, we would have to either pad the frames (based on the estimated size for the frame), or emit some index into the stream, or as a seperate artifact
To me the padding seems easier but a foramt chnage, but the separate artifact would be no format change and an easier migration path

[kessplas]

Hey @mkeskells,

First to answer your question about frames:

There are numerous advantages to using a framed message format especially for AES-GCM encryption. Primarily, the use of frames is related to the authenticity property of AES-GCM. Each AES-GCM frame contains its own authentication tag which provides assurance that the ciphertext has not been tampered with. For ciphertexts which are too large to fit into memory (such that data is decrypted using a stream instead), the authentication tag can be verified for each frame before being "released" as plaintext. It is possible to decrypt unframed objects via streaming, but the authenticity can not be verified until the end of the stream is reached, meaning that unauthenticated plaintext will be released and a tag mismatch won't be discovered until later.

Furthermore, a framed message format allows for encryption of large objects. In an unframed scheme, i.e. "plain old" AES-GCM encryption, the maximum size of encryptable plaintext is just under 64Gib. Using frames, the theoretical limit is somewhere in the exabytes. This has to do with IV reuse.

With regard to the current format's suitability for random access decryption:

The ESDK uses a digital signature to enforce policy separation of encrypt/decrypt when used with AWS KMS, which is an important security property. The signature is calculated during encryption of the entire message and appended to the end of the ciphertext. Therefore, it is not possible to decrypt a range of data and be able to verify that the message was not forged by a rogue encryptor.

Given the sensitive nature of the ESDK and depth of understanding required to ensure the various security properties involved in encryption, AWS Crypto Tools is unlikely to accept pull requests for features which modify the security properties of our libraries. Such changes require consultation with cryptographers and go through rigorous security review processes. As Tony mentioned, a new format which supports per-frame signatures is on our roadmap but has not yet been prioritized.

Finally, there is a workaround, though it may not be suitable for you:

The Amazon S3 Encryption Client does support "ranged gets" in S3, which allows for random access / partial decryption of a given ciphertext. HOWEVER, this feature DOES NOT provide any authenticity guarantees that would normally be afforded by AES-GCM. This would only be tractable if you're using S3 as your data store, and if you're able to mitigate threats around ciphertext tampering (as well as encrypt/decrypt policy separation) that authenticity would normally provide. Note that the S3EC's message format is NOT compatible with the ESDK's.