Skip to content

Add streaming encrypt decrypt#202

Merged
skeet70 merged 16 commits intomainfrom
add-streaming-encrypt-decrypt
Apr 2, 2026
Merged

Add streaming encrypt decrypt#202
skeet70 merged 16 commits intomainfrom
add-streaming-encrypt-decrypt

Conversation

@skeet70
Copy link
Copy Markdown
Member

@skeet70 skeet70 commented Mar 31, 2026

Fixes #200

There's a lot of boilerplate to thread this through the Shim -> Frame -> Worker and back up, and a lot of test coverage. I added some examples of using it with files in an examples page on the website, I didn't want to bloat the library with the deps (and steps for it) for those to be in-lib helpers.

The main changes are (walking out from the core):

  • src/frame/worker/crypto/aes/StreamingAes.ts
    • The core streaming implementation modeled after what we did in IronNode and IronOxide. The main difference is that since we're committed to truly streaming here (instead of being directly file based), StreamingDecryptor and StreamingEncryptor both produce TransformStreams (which has its own transform and flush that the processChunk and finalize concepts we need map nicely onto).
  • src/frame/worker/DocumentCrypto.ts
    • does the PRE bits on either side, manages the streaming pipe, generate the key on encrypt
    • if there's a failure in the crypto here or lower down, the stream gets aborted with the error value so that problem gets communicated out and the stream stops piping data. The caller still has to do the right thing and invalidate the bytes they did get, but if they're using standard streaming tools that'll happen on its own since it's the standard.
  • src/frame/sdk/DocumentApi.ts & src/frame/sdk/DocumentAdvancedApi.ts
    • does IC-ID bits (fetches metadata, resolves groups)
    • generates the IV for encrypt
  • src/shim/sdk/DocumentSDK.ts
    • decomposes our doc format in a streaming friendly way

Here's a generated flow diagram of decryptStream, hopefully it's helpful figuring out what's going on:

Shim                          Frame                         Worker
────                          ─────                         ──────
parseStreamHeader()
  read bytes until header+IV
  split into { iv, ciphertextStream }

new TransformStream()
  readable = plaintextStream
    (returned to caller)
  writable = plaintextWritable
    (transferred to Frame)
                              |
postMessage ─────────────────>|
  [ciphertextStream,          |
   plaintextWritable]          |
                              callDocumentMetadataGetApi()
                                -> encryptedSymmetricKey
                              |
                              postMessage ─────────────────>|
                                [encSymKey, privKey,         |
                                 iv, streams]                |
                                                            |
                                            Recrypt.decryptPlaintext()
                                              -> AES symmetric key
                                                            |
                                            StreamingDecryptor.create(key, iv)
                                                            |
                                            ciphertextStream
                                              .pipeThrough(decryptor)
                                              .pipeTo(plaintextWritable)
                                                            |
                                            +------ streaming loop ------+
                                            | for each chunk:            |
                                            |   prepend held-back bytes  |
                                            |   reserve last 16 (tag)   |
                                            |   block-align for GHASH   |
                                            |   GHASH(ciphertext)       |
                                            |   AES-CTR decrypt         |
                                            |   enqueue(plaintext) ─────+──> caller reads
                                            |   ...repeats...           |    plaintextStream
                                            |                           |
                                            | on source end (flush):    |
                                            |   decrypt final remainder |
                                            |   verify GHASH tag        |
                                            |   mismatch -> error stream|
                                            |   match -> close stream   |
                                            +---------------------------+

Copy link
Copy Markdown
Member

@coltfred coltfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed some changes I made as well as some tests. A couple thoughts as comments.

skeet70 and others added 4 commits April 1, 2026 17:07
It's where it's needed, and it makes it so our message types make a little
bit more sense and help document what's going on better.
@skeet70 skeet70 requested a review from coltfred April 2, 2026 01:56
@skeet70 skeet70 merged commit 0770e30 into main Apr 2, 2026
1 check passed
@skeet70 skeet70 deleted the add-streaming-encrypt-decrypt branch April 2, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming encrypt/decrypt

3 participants