Skip to content

Commit 810f370

Browse files
spacesailor24spacesailor24
authored
Merge pull request #80 from LIT-Protocol/wyatt/hacker-guide
Init Encrypt Large File guide
2 parents 3aa9154 + a6f8ca8 commit 810f370

File tree

14 files changed

+38515
-0
lines changed

14 files changed

+38515
-0
lines changed

‎.cursorignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Add directories or file patterns to ignore during indexing (e.g. foo/ or *.csv)
2+
loremIpsum*

‎hacker-guides/encryption/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,4 @@ Now that you have an overview of encrypting data with Lit, you can continue on t
3333

3434
- [Encrypting a String](./encrypt-string/README.md)
3535
- [Encrypting a File](./encrypt-file/README.md)
36+
- [Encrypting a Large File](./encrypt-large-file/README.md)

‎hacker-guides/encryption/encrypt-file/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,3 +166,5 @@ console.log(`ℹ️ Decrypted content saved to: ${outputPath}`);
166166
```
167167

168168
## Next Steps
169+
170+
- If the file you want to encrypt is larger than `1.5MB`, checkout the [Encrypting a Large File](./encrypt-large-file/README.md) guide.

‎hacker-guides/encryption/encrypt-large-file/.env.example‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ETHEREUM_PRIVATE_KEY=

‎hacker-guides/encryption/encrypt-large-file/.gitignore‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
yarn-debug.log*
6+
yarn-error.log*
7+
pnpm-debug.log*
8+
lerna-debug.log*
9+
10+
.env
11+
node_modules
12+
dist
13+
dist-ssr
14+
*.local
15+
16+
# Editor directories and files
17+
.vscode/*
18+
!.vscode/extensions.json
19+
.idea
20+
.DS_Store
21+
*.suo
22+
*.ntvs*
23+
*.njsproj
24+
*.sln
25+
*.sw?

‎hacker-guides/encryption/encrypt-large-file/.mocharc.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"$schema": "https://json.schemastore.org/mocharc.json",
3+
"require": "tsx"
4+
}

‎hacker-guides/encryption/encrypt-large-file/README.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
<!-- omit in toc -->
2+
# Encrypting a Large File
3+
4+
As mentioned in the [Encrypting a File](../encrypt-file/README.md) guide, the Lit SDK has a maximum payload size limit of approximately `1.5MB` for encryption operations. Since the SDK currently doesn't support file chunking, attempting to encrypt files larger than this will result in out of memory errors.
5+
6+
This guide demonstrates how to securely encrypt and decrypt large files using a combination of symmetric encryption (AES-256-CBC) and Lit Protocol's decentralized access control.
7+
8+
- [Prerequisites](#prerequisites)
9+
- [Running the Code Example](#running-the-code-example)
10+
- [Requirements](#requirements)
11+
- [Steps](#steps)
12+
- [Expected Output](#expected-output)
13+
- [Understanding the Code](#understanding-the-code)
14+
- [Generate the Symmetric Encryption Key](#generate-the-symmetric-encryption-key)
15+
- [Reading and Encrypting the Large File](#reading-and-encrypting-the-large-file)
16+
- [Encrypting the Key Material with Lit Protocol](#encrypting-the-key-material-with-lit-protocol)
17+
- [Decrypting the Key Material](#decrypting-the-key-material)
18+
- [Decrypting the Large File](#decrypting-the-large-file)
19+
- [Next Steps](#next-steps)
20+
21+
## Prerequisites
22+
23+
- Understanding of Lit core terminology and concepts covered [here](../README.md#core-terminology)
24+
- Understanding of Lit encryption terminology and concepts covered [here](../README.md#relevant-terminology)
25+
- Understanding of the [Connecting to the Lit Network](../connecting-to-lit/README.md) guide
26+
- Understanding of the [Authenticating a Session](../../_getting-started/authenticating-a-session/README.md) guide
27+
- Understanding of the [Encrypting a File](../encrypt-file/README.md) guide
28+
29+
## Running the Code Example
30+
31+
### Requirements
32+
33+
- [Node.js](https://nodejs.org/en)
34+
- [Yarn](https://yarnpkg.com/getting-started)
35+
- `@lit-protocol/constants`
36+
- `@lit-protocol/lit-node-client`
37+
- `@lit-protocol/auth-helpers`
38+
- `@lit-protocol/types`
39+
40+
### Steps
41+
42+
1. `yarn` to install the dependencies
43+
2. `yarn test` to run the code example
44+
45+
### Expected Output
46+
47+
After running the code example, you should see output in your terminal:
48+
49+
1. An indication that a connection to the Lit Network was successfully established
50+
2. The symmetric key and initialization vector were successfully encrypted
51+
- The `ciphertext` and `dataToEncryptHash` are logged to the terminal for demonstration purposes:
52+
53+
```bash
54+
ℹ️ ciphertext: jD6Qe8tsWHkLcVr8MqSy/eHAyZcq3M+M+ZI8FPFa+PcpXTYZGx4H4lFJ3bhGQJtGjDXiPkylHvVLmD9EeB9y0kzfWMEnlvjESXWp23EqXkQxlszbJxtNrEywBo046QSyz14BAxWHKYgOHFFoWLCMlIjhZQZUeEwbtZ6XayIsvlzdYQI=
55+
ℹ️ dataToEncryptHash: 55109cb2bd42b0fccfe4824c1f93953d2fd61ba769b4433dccb62eea6fed0df2
56+
```
57+
58+
3. Session Signatures were successfully generated for the requested session
59+
4. After the JavaScript test passes, you should see the path for the decrypted file logged to the terminal:
60+
- NOTE: The decrypted file is actually deleted after the test is complete, for cleanup and sanity purposes. You can disable this behavior by commenting out the `afterEach` function in the [./test/index.spec.ts](./test/index.spec.ts) file.
61+
- The test compares the decrypted file to the original file to ensure they are the same before deleting the decrypted file.
62+
63+
```bash
64+
ℹ️ Decrypted content saved to: /Users/user/developer-guides-code/hacker-guides/encryption/encrypt-large-file/src/loremIpsum-decrypted.txt
65+
```
66+
67+
## Understanding the Code
68+
69+
The following code from [./src/index.ts](./src/index.ts) does the following:
70+
71+
### Generate the Symmetric Encryption Key
72+
73+
```typescript
74+
// Generate a random symmetric key and initialization vector
75+
const symmetricKey = randomBytes(32); // 256 bits for AES-256
76+
const initializationVector = randomBytes(16); // 16 bytes for AES
77+
```
78+
79+
This code generates the two components for AES-256-CBC encryption:
80+
81+
1. `symmetricKey`:
82+
- A 32-byte (256-bit) random key used for both encryption and decryption
83+
- Uses AES-256 (Advanced Encryption Standard) which is a widely trusted symmetric encryption algorithm
84+
85+
2. `initializationVector`:
86+
- A 16-byte random value that adds randomness to the encryption process
87+
- Ensures that encrypting the same data multiple times produces different ciphertext outputs
88+
- Helps prevent pattern analysis and makes the encryption more secure
89+
90+
### Reading and Encrypting the Large File
91+
92+
The `encryptFile` helper function implements a memory-efficient streaming approach to encrypt large files:
93+
94+
1. **Setup**:
95+
- Creates input/output file streams
96+
- Initializes AES cipher with the provided key and IV
97+
- Sets up a 64KB buffer for chunked reading
98+
2. **Streaming Process**:
99+
- Reads the input file in 64KB chunks
100+
- Encrypts each chunk using the cipher
101+
- Writes encrypted chunks to the output file
102+
- Continues until entire file is processed
103+
3. **Cleanup**:
104+
- Finalizes encryption
105+
- Closes file streams
106+
- Returns path to encrypted file
107+
108+
The chunk streaming approach is required because it:
109+
110+
- Avoids loading entire file into memory
111+
- Can handle files of any size
112+
- Maintains constant memory usage regardless of file size
113+
114+
The output file will have the same name as the input with `.encrypted` appended (in the case of this example, `loremIpsum.txt.encrypted`).
115+
116+
### Encrypting the Key Material with Lit Protocol
117+
118+
After encrypting the large file with AES encryption, we need to securely store the encryption key and initialization vector. To do this, we:
119+
120+
1. Combine the symmetric key and IV into a single buffer
121+
2. Encrypt this combined key material using Lit's `encrypt` method
122+
123+
```typescript
124+
// Combine key and initializationVector for Lit encryption
125+
const keyData = Buffer.concat([symmetricKey, initializationVector]);
126+
const { ciphertext, dataToEncryptHash } = await litNodeClient.encrypt({
127+
dataToEncrypt: keyData,
128+
accessControlConditions,
129+
});
130+
131+
console.log(`ℹ️ ciphertext: ${ciphertext}`);
132+
console.log(`ℹ️ dataToEncryptHash: ${dataToEncryptHash}`);
133+
```
134+
135+
Because we're utilizing Lit for the encryption of the key material, we still benefit from Lit's decryption Access Control Conditions. Only users meeting the specified conditions can decrypt the key material and access the decrypted file.
136+
137+
### Decrypting the Key Material
138+
139+
After generating the required Session Signatures, we make a request to the Lit Network to decrypt the key material:
140+
141+
```typescript
142+
// Decrypt the symmetric key using Lit
143+
const decryptionResponse = await litNodeClient.decrypt({
144+
chain: "ethereum",
145+
sessionSigs: sessionSignatures,
146+
ciphertext,
147+
dataToEncryptHash,
148+
accessControlConditions,
149+
});
150+
151+
// Split the decrypted data back into key and initializationVector
152+
const decryptedKeyData = Buffer.from(decryptionResponse.decryptedData);
153+
const decryptedKey = decryptedKeyData.subarray(0, 32);
154+
const decryptedIv = decryptedKeyData.subarray(32);
155+
```
156+
157+
With the decrypted key and initialization vector, we can now decrypt the large file using AES-256-CBC.
158+
159+
### Decrypting the Large File
160+
161+
The `decryptFile` helper function implements a streaming approach to decrypt large files, mirroring the encryption process:
162+
163+
1. **Setup**:
164+
- Creates input/output file streams
165+
- Initializes AES decipher with the decrypted key and IV
166+
- Sets up a 64KB buffer for chunked reading
167+
168+
```typescript
169+
const decipher = createDecipheriv(
170+
ENCRYPTION_ALGORITHM,
171+
key,
172+
initializationVector
173+
);
174+
175+
const chunkSize = 64 * 1024; // 64KB chunks
176+
const buffer = Buffer.alloc(chunkSize);
177+
```
178+
179+
2. **Streaming Process**:
180+
- Reads the encrypted file in 64KB chunks
181+
- Decrypts each chunk using the decipher
182+
- Writes decrypted chunks to the output file
183+
- Continues until entire file is processed
184+
185+
```typescript
186+
while (
187+
(bytesRead = (await readStream.read(buffer, 0, chunkSize)).bytesRead) > 0
188+
) {
189+
const chunk = buffer.subarray(0, bytesRead);
190+
const decryptedChunk = decipher.update(chunk);
191+
await writeStream.write(decryptedChunk);
192+
}
193+
```
194+
195+
3. **Cleanup**:
196+
- Finalizes decryption
197+
- Closes file streams
198+
- Returns path to decrypted file
199+
200+
Finally, the decrypted file is written to the `src` directory with the name `loremIpsum-decrypted.txt`:
201+
202+
```typescript
203+
// Decrypt the file using the decrypted symmetric key
204+
const decryptedFilePath = await decryptFile(
205+
encryptedFilePath,
206+
decryptedKey,
207+
decryptedIv
208+
);
209+
console.log(`ℹ️ Decrypted content saved to: ${decryptedFilePath}`);
210+
```
211+
212+
## Next Steps

‎hacker-guides/encryption/encrypt-large-file/package.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"name": "nodejs",
3+
"version": "1.0.0",
4+
"main": "index.js",
5+
"license": "MIT",
6+
"type": "module",
7+
"scripts": {
8+
"test": "npx @dotenvx/dotenvx run -- mocha test/**/*.spec.ts"
9+
},
10+
"dependencies": {
11+
"@dotenvx/dotenvx": "^0.44.1",
12+
"@lit-protocol/auth-helpers": "^6.11.0",
13+
"@lit-protocol/constants": "^6.11.0",
14+
"@lit-protocol/lit-node-client": "^6.11.0",
15+
"@lit-protocol/types": "^6.11.0",
16+
"ethers": "v5"
17+
},
18+
"devDependencies": {
19+
"@types/chai": "^4.3.16",
20+
"@types/chai-json-schema": "^1.4.10",
21+
"@types/mocha": "^10.0.6",
22+
"chai": "^5.1.1",
23+
"chai-json-schema": "^1.5.1",
24+
"mocha": "^10.4.0",
25+
"tsc": "^2.0.4",
26+
"tsx": "^4.12.0",
27+
"typescript": "^5.4.5"
28+
}
29+
}

0 commit comments

Comments
 (0)