|
| 1 | +<!-- omit in toc --> |
| 2 | +# Encrypting a Large File |
| 3 | + |
| 4 | +As mentioned in the [Encrypting a File](../encrypt-file/README.md) guide, the Lit SDK has a maximum payload size limit of approximately `1.5MB` for encryption operations. Since the SDK currently doesn't support file chunking, attempting to encrypt files larger than this will result in out of memory errors. |
| 5 | + |
| 6 | +This guide demonstrates how to securely encrypt and decrypt large files using a combination of symmetric encryption (AES-256-CBC) and Lit Protocol's decentralized access control. |
| 7 | + |
| 8 | +- [Prerequisites](#prerequisites) |
| 9 | +- [Running the Code Example](#running-the-code-example) |
| 10 | + - [Requirements](#requirements) |
| 11 | + - [Steps](#steps) |
| 12 | + - [Expected Output](#expected-output) |
| 13 | +- [Understanding the Code](#understanding-the-code) |
| 14 | + - [Generate the Symmetric Encryption Key](#generate-the-symmetric-encryption-key) |
| 15 | + - [Reading and Encrypting the Large File](#reading-and-encrypting-the-large-file) |
| 16 | + - [Encrypting the Key Material with Lit Protocol](#encrypting-the-key-material-with-lit-protocol) |
| 17 | + - [Decrypting the Key Material](#decrypting-the-key-material) |
| 18 | + - [Decrypting the Large File](#decrypting-the-large-file) |
| 19 | +- [Next Steps](#next-steps) |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +- Understanding of Lit core terminology and concepts covered [here](../README.md#core-terminology) |
| 24 | +- Understanding of Lit encryption terminology and concepts covered [here](../README.md#relevant-terminology) |
| 25 | +- Understanding of the [Connecting to the Lit Network](../connecting-to-lit/README.md) guide |
| 26 | +- Understanding of the [Authenticating a Session](../../_getting-started/authenticating-a-session/README.md) guide |
| 27 | +- Understanding of the [Encrypting a File](../encrypt-file/README.md) guide |
| 28 | + |
| 29 | +## Running the Code Example |
| 30 | + |
| 31 | +### Requirements |
| 32 | + |
| 33 | +- [Node.js](https://nodejs.org/en) |
| 34 | +- [Yarn](https://yarnpkg.com/getting-started) |
| 35 | +- `@lit-protocol/constants` |
| 36 | +- `@lit-protocol/lit-node-client` |
| 37 | +- `@lit-protocol/auth-helpers` |
| 38 | +- `@lit-protocol/types` |
| 39 | + |
| 40 | +### Steps |
| 41 | + |
| 42 | +1. `yarn` to install the dependencies |
| 43 | +2. `yarn test` to run the code example |
| 44 | + |
| 45 | +### Expected Output |
| 46 | + |
| 47 | +After running the code example, you should see output in your terminal: |
| 48 | + |
| 49 | +1. An indication that a connection to the Lit Network was successfully established |
| 50 | +2. The symmetric key and initialization vector were successfully encrypted |
| 51 | + - The `ciphertext` and `dataToEncryptHash` are logged to the terminal for demonstration purposes: |
| 52 | + |
| 53 | +```bash |
| 54 | +ℹ️ ciphertext: jD6Qe8tsWHkLcVr8MqSy/eHAyZcq3M+M+ZI8FPFa+PcpXTYZGx4H4lFJ3bhGQJtGjDXiPkylHvVLmD9EeB9y0kzfWMEnlvjESXWp23EqXkQxlszbJxtNrEywBo046QSyz14BAxWHKYgOHFFoWLCMlIjhZQZUeEwbtZ6XayIsvlzdYQI= |
| 55 | +ℹ️ dataToEncryptHash: 55109cb2bd42b0fccfe4824c1f93953d2fd61ba769b4433dccb62eea6fed0df2 |
| 56 | +``` |
| 57 | + |
| 58 | +3. Session Signatures were successfully generated for the requested session |
| 59 | +4. After the JavaScript test passes, you should see the path for the decrypted file logged to the terminal: |
| 60 | + - NOTE: The decrypted file is actually deleted after the test is complete, for cleanup and sanity purposes. You can disable this behavior by commenting out the `afterEach` function in the [./test/index.spec.ts](./test/index.spec.ts) file. |
| 61 | + - The test compares the decrypted file to the original file to ensure they are the same before deleting the decrypted file. |
| 62 | + |
| 63 | +```bash |
| 64 | +ℹ️ Decrypted content saved to: /Users/user/developer-guides-code/hacker-guides/encryption/encrypt-large-file/src/loremIpsum-decrypted.txt |
| 65 | +``` |
| 66 | + |
| 67 | +## Understanding the Code |
| 68 | + |
| 69 | +The following code from [./src/index.ts](./src/index.ts) does the following: |
| 70 | + |
| 71 | +### Generate the Symmetric Encryption Key |
| 72 | + |
| 73 | +```typescript |
| 74 | +// Generate a random symmetric key and initialization vector |
| 75 | +const symmetricKey = randomBytes(32); // 256 bits for AES-256 |
| 76 | +const initializationVector = randomBytes(16); // 16 bytes for AES |
| 77 | +``` |
| 78 | + |
| 79 | +This code generates the two components for AES-256-CBC encryption: |
| 80 | + |
| 81 | +1. `symmetricKey`: |
| 82 | + - A 32-byte (256-bit) random key used for both encryption and decryption |
| 83 | + - Uses AES-256 (Advanced Encryption Standard) which is a widely trusted symmetric encryption algorithm |
| 84 | + |
| 85 | +2. `initializationVector`: |
| 86 | + - A 16-byte random value that adds randomness to the encryption process |
| 87 | + - Ensures that encrypting the same data multiple times produces different ciphertext outputs |
| 88 | + - Helps prevent pattern analysis and makes the encryption more secure |
| 89 | + |
| 90 | +### Reading and Encrypting the Large File |
| 91 | + |
| 92 | +The `encryptFile` helper function implements a memory-efficient streaming approach to encrypt large files: |
| 93 | + |
| 94 | +1. **Setup**: |
| 95 | + - Creates input/output file streams |
| 96 | + - Initializes AES cipher with the provided key and IV |
| 97 | + - Sets up a 64KB buffer for chunked reading |
| 98 | +2. **Streaming Process**: |
| 99 | + - Reads the input file in 64KB chunks |
| 100 | + - Encrypts each chunk using the cipher |
| 101 | + - Writes encrypted chunks to the output file |
| 102 | + - Continues until entire file is processed |
| 103 | +3. **Cleanup**: |
| 104 | + - Finalizes encryption |
| 105 | + - Closes file streams |
| 106 | + - Returns path to encrypted file |
| 107 | + |
| 108 | +The chunk streaming approach is required because it: |
| 109 | + |
| 110 | +- Avoids loading entire file into memory |
| 111 | +- Can handle files of any size |
| 112 | +- Maintains constant memory usage regardless of file size |
| 113 | + |
| 114 | +The output file will have the same name as the input with `.encrypted` appended (in the case of this example, `loremIpsum.txt.encrypted`). |
| 115 | + |
| 116 | +### Encrypting the Key Material with Lit Protocol |
| 117 | + |
| 118 | +After encrypting the large file with AES encryption, we need to securely store the encryption key and initialization vector. To do this, we: |
| 119 | + |
| 120 | +1. Combine the symmetric key and IV into a single buffer |
| 121 | +2. Encrypt this combined key material using Lit's `encrypt` method |
| 122 | + |
| 123 | +```typescript |
| 124 | +// Combine key and initializationVector for Lit encryption |
| 125 | +const keyData = Buffer.concat([symmetricKey, initializationVector]); |
| 126 | +const { ciphertext, dataToEncryptHash } = await litNodeClient.encrypt({ |
| 127 | + dataToEncrypt: keyData, |
| 128 | + accessControlConditions, |
| 129 | +}); |
| 130 | + |
| 131 | +console.log(`ℹ️ ciphertext: ${ciphertext}`); |
| 132 | +console.log(`ℹ️ dataToEncryptHash: ${dataToEncryptHash}`); |
| 133 | +``` |
| 134 | + |
| 135 | +Because we're utilizing Lit for the encryption of the key material, we still benefit from Lit's decryption Access Control Conditions. Only users meeting the specified conditions can decrypt the key material and access the decrypted file. |
| 136 | + |
| 137 | +### Decrypting the Key Material |
| 138 | + |
| 139 | +After generating the required Session Signatures, we make a request to the Lit Network to decrypt the key material: |
| 140 | + |
| 141 | +```typescript |
| 142 | +// Decrypt the symmetric key using Lit |
| 143 | +const decryptionResponse = await litNodeClient.decrypt({ |
| 144 | + chain: "ethereum", |
| 145 | + sessionSigs: sessionSignatures, |
| 146 | + ciphertext, |
| 147 | + dataToEncryptHash, |
| 148 | + accessControlConditions, |
| 149 | +}); |
| 150 | + |
| 151 | +// Split the decrypted data back into key and initializationVector |
| 152 | +const decryptedKeyData = Buffer.from(decryptionResponse.decryptedData); |
| 153 | +const decryptedKey = decryptedKeyData.subarray(0, 32); |
| 154 | +const decryptedIv = decryptedKeyData.subarray(32); |
| 155 | +``` |
| 156 | + |
| 157 | +With the decrypted key and initialization vector, we can now decrypt the large file using AES-256-CBC. |
| 158 | + |
| 159 | +### Decrypting the Large File |
| 160 | + |
| 161 | +The `decryptFile` helper function implements a streaming approach to decrypt large files, mirroring the encryption process: |
| 162 | + |
| 163 | +1. **Setup**: |
| 164 | + - Creates input/output file streams |
| 165 | + - Initializes AES decipher with the decrypted key and IV |
| 166 | + - Sets up a 64KB buffer for chunked reading |
| 167 | + |
| 168 | +```typescript |
| 169 | +const decipher = createDecipheriv( |
| 170 | + ENCRYPTION_ALGORITHM, |
| 171 | + key, |
| 172 | + initializationVector |
| 173 | +); |
| 174 | + |
| 175 | +const chunkSize = 64 * 1024; // 64KB chunks |
| 176 | +const buffer = Buffer.alloc(chunkSize); |
| 177 | +``` |
| 178 | + |
| 179 | +2. **Streaming Process**: |
| 180 | + - Reads the encrypted file in 64KB chunks |
| 181 | + - Decrypts each chunk using the decipher |
| 182 | + - Writes decrypted chunks to the output file |
| 183 | + - Continues until entire file is processed |
| 184 | + |
| 185 | +```typescript |
| 186 | +while ( |
| 187 | + (bytesRead = (await readStream.read(buffer, 0, chunkSize)).bytesRead) > 0 |
| 188 | +) { |
| 189 | + const chunk = buffer.subarray(0, bytesRead); |
| 190 | + const decryptedChunk = decipher.update(chunk); |
| 191 | + await writeStream.write(decryptedChunk); |
| 192 | +} |
| 193 | +``` |
| 194 | + |
| 195 | +3. **Cleanup**: |
| 196 | + - Finalizes decryption |
| 197 | + - Closes file streams |
| 198 | + - Returns path to decrypted file |
| 199 | + |
| 200 | +Finally, the decrypted file is written to the `src` directory with the name `loremIpsum-decrypted.txt`: |
| 201 | + |
| 202 | +```typescript |
| 203 | +// Decrypt the file using the decrypted symmetric key |
| 204 | +const decryptedFilePath = await decryptFile( |
| 205 | + encryptedFilePath, |
| 206 | + decryptedKey, |
| 207 | + decryptedIv |
| 208 | +); |
| 209 | +console.log(`ℹ️ Decrypted content saved to: ${decryptedFilePath}`); |
| 210 | +``` |
| 211 | + |
| 212 | +## Next Steps |
0 commit comments