Skip to content

Commit 21dffe4

Browse files
authored
Merge pull request #5 from lukachad/testing
Testing
2 parents a77718f + 927c394 commit 21dffe4

File tree

11 files changed

+1013
-353
lines changed

11 files changed

+1013
-353
lines changed

lib/lib-storage/package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@
1515
"clean": "rimraf ./dist-* && rimraf *.tsbuildinfo",
1616
"extract:docs": "api-extractor run --local",
1717
"test": "yarn g:vitest run",
18-
"test:e2e": "yarn g:vitest run -c vitest.config.e2e.ts --mode development",
1918
"test:watch": "yarn g:vitest watch",
19+
"test:browser": "yarn g:vitest run -c vitest.config.browser.ts",
20+
"test:browser:watch": "yarn g:vitest watch -c vitest.config.browser.ts",
21+
"test:e2e": "yarn g:vitest run -c vitest.config.e2e.ts --mode development",
2022
"test:e2e:watch": "yarn g:vitest watch -c vitest.config.e2e.ts"
2123
},
2224
"engines": {
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# @aws-sdk/lib-storage/s3-transfer-manager
2+
3+
## Overview
4+
5+
S3TransferManager is a high level library that helps customer interact with S3
6+
for their most common use cases that involve multiple API operations through SDK JS V3.
7+
S3TransferManager provides the following features:
8+
9+
- automatic multipart upload to S3
10+
- automatic multipart download from S3
11+
- upload all files in a directory to an S3 bucket recursively or non-recursively
12+
- download all objects in a bucket to a local directory recursively or non-recursively
13+
- transfer progress listener
14+
15+
## Installation
16+
17+
## Getting Started
18+
19+
## Configuration
20+
21+
When creating an instance of the S3TransferManager, users can configure some of it's client options
22+
to best fit their use case.
23+
24+
- s3ClientInstance - specify the low level S3 client that will be used to send reqeusts to S3
25+
- targetPartSizeBytes - specify the target part size to use in mulitpart transfer. Does not
26+
apply to the last part and downloads if multipartDownloadType is PART
27+
- multipartUploadThresholdBytes - specify the size threshold in bytes for multipart upload.
28+
- checksumValidationEnabled - option to disable checksum validation for donwload.
29+
- multipartDownloadType - specify how the SDK should perform multipart download. Either RANGE or PART.
30+
- eventListeners - transfer progress listeners to receive event-driven updates on transfer
31+
progress throughout the lifecycle of a request at client level. Supported callbacks:
32+
- transferInitiated: A new transfer has been initiated. This method is invoked exactly once per
33+
transfer, right after the operation has started. It allows users to retrieve the request and ProgressSnapshot.
34+
- bytesTransferred: Additional bytes have been submitted or received. This method may be called
35+
many times per transfer, depending on the transfer size and I/O buffer sizes. It must be called
36+
at least once for a successful transfer. It allows users to retrieve the the request and the ProgressSnapshot.
37+
- transferComplete: The transfer has completed successfully. This method is called exactly once for
38+
a successful transfer. It allows users to retrieve the request, the response and the ProgressSnapshot.
39+
- transferFailed: The transfer has failed. This method is called exactly once for a failed transfer.
40+
It allows users to retrieve the request and a progress snapshot.
41+
42+
### Example
43+
44+
```js
45+
import { S3Client } from "@aws-sdk/client-s3";
46+
import { S3TransferManager } from "@aws-sdk/lib-storage";
47+
48+
const tm = new S3TransferManager ({
49+
s3ClientInstance: new S3Client({}),
50+
multipartDownloadType: "RANGE",
51+
targetPartSizeBytes: 8 * 1024 * 1024
52+
multipartThresholdBytes: 16 * 1024 * 1024,
53+
checksumValidationEnabled: true,
54+
checksumAlgorithm: CRC32,
55+
multipartDownloadType: PART,
56+
eventListeners: {
57+
transferInitiated: [transferStarted],
58+
bytesTrnasferred: [progressBar],
59+
transferComplete: [{
60+
handleEvent: console.log({
61+
request, snapshot, response
62+
})
63+
}],
64+
trasnferFailed: [transferFailed]
65+
}
66+
})
67+
```
68+
69+
### Constructor Options
70+
71+
## API Reference
72+
73+
## Methods
74+
75+
### upload()
76+
77+
### download()
78+
79+
The download() function in S3TransferManager is a wrapper function for the S3 GetObjectCommand
80+
allowing users to download objects from an S3 bucket using multipart download of two types
81+
which are specified in the configuration of the S3TransferManager instance: Part GET and Ranged GET.
82+
Both of which download the object using GetObjectCommand in separate streams then join them into
83+
one single stream. The S3TransferManager download() supports Readable and ReadableStream for node and browser.
84+
85+
- Part GET
86+
- Use case: Optimizes downloads for objects that were uploaded using the S3 multipart upload
87+
- How it works: Uses the S3 native download feature with the PartNumber parameter. It fetches part 1 of the object to get the metadata then downloads the remaining parts concurrently.
88+
- Range GET
89+
- Use case: Allows for multipart download for any S3 object regardless of whether it was
90+
uploaded using multipart upload or not
91+
- How it works: Uses the HTTP Range request with the bytes=start-end headers to split objects into
92+
chunks based on the user-provided byte range header, or if not included the MIN_PART_SIZE to make concurrent range requests.
93+
94+
Users can also include an abortController allowing for cancellation mid download along
95+
with eventListeners for the callbacks: 'transferInitiated', 'bytesTransferred', 'transferComplete',
96+
and 'transferFailed' at client level and request level. 'bytesTransferred' provides progress updates per byte chunk during streaming.
97+
98+
#### Validation
99+
100+
Both multipartDownloadTypes have methods that validates the bytes and ranges of the multipart download requests. In multipartDownloadType PART, bytes of the part boundaries in each concurrent request are checked for whether they match the expected byte boundaries. In multipartDownloadType RANGE, the byte ranges are checked for whether they match the expected ranges. An error is thrown on mismatches and all requests for download is cancelled.
101+
102+
Both both PART and RANGE GET uses the S3 standard IfMatch header with the initial ETag for subsequent parts to ensure object version consistency during a download.
103+
104+
#### uploadAll()
105+
106+
#### downloadAll()
107+
108+
### Event Handling
109+
110+
#### addEventListener()
111+
112+
#### removeEventListener()
113+
114+
#### dispatchEvent()
115+
116+
## Transfer Options
117+
118+
### AbortSignal
119+
120+
### Event Listeners
121+
122+
## Examples
123+
124+
### Basic Upload
125+
126+
### Basic Download
127+
128+
### Multipart Download
129+
130+
### Event Handling
131+
132+
### Abort Operations
133+
134+
## Performance Considerations
135+
136+
## Error Handling
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
import { StreamingBlobPayloadOutputTypes } from "@smithy/types";
2+
import { sdkStreamMixin } from "@smithy/util-stream";
3+
import { describe, expect, it, vi } from "vitest";
4+
5+
import { joinStreams } from "./join-streams.browser";
6+
7+
describe("join-streams tests", () => {
8+
const createReadableStreamWithContent = (content: Uint8Array) =>
9+
new ReadableStream({
10+
start(controller) {
11+
controller.enqueue(content);
12+
controller.close();
13+
},
14+
});
15+
16+
const createEmptyReadableStream = () =>
17+
new ReadableStream({
18+
start(controller) {
19+
controller.close();
20+
},
21+
});
22+
23+
const consumeReadableStream = async (stream: any): Promise<Uint8Array[]> => {
24+
const reader = stream.getReader();
25+
const chunks: Uint8Array[] = [];
26+
27+
while (true) {
28+
const { done, value } = await reader.read();
29+
if (done) break;
30+
if (value) chunks.push(value);
31+
}
32+
33+
return chunks;
34+
};
35+
36+
const testCases = [
37+
{
38+
name: "ReadableStream",
39+
createStream: () => new ReadableStream(),
40+
createWithContent: createReadableStreamWithContent,
41+
createEmpty: createEmptyReadableStream,
42+
consume: consumeReadableStream,
43+
isInstance: (stream: any) => typeof stream.getReader === "function",
44+
},
45+
];
46+
47+
testCases.forEach(({ name, createStream, createWithContent, createEmpty, consume, isInstance }) => {
48+
describe(`joinStreams() with ${name}`, () => {
49+
it("should return single stream when only one stream is provided", async () => {
50+
const stream = createStream();
51+
const mixedStream = sdkStreamMixin(stream);
52+
const result = await joinStreams([Promise.resolve(mixedStream as StreamingBlobPayloadOutputTypes)]);
53+
54+
expect(result).toBeDefined();
55+
expect(result).not.toBe(stream);
56+
expect(isInstance(result)).toBe(true);
57+
});
58+
59+
it("should join multiple streams into a single stream", async () => {
60+
const contents = [
61+
new Uint8Array([67, 104, 117, 110, 107, 32, 49]), // "Chunk 1"
62+
new Uint8Array([67, 104, 117, 110, 107, 32, 50]), // "Chunk 2"
63+
new Uint8Array([67, 104, 117, 110, 107, 32, 51]), // "Chunk 3"
64+
];
65+
66+
const streams = contents.map((content) =>
67+
Promise.resolve(sdkStreamMixin(createWithContent(content)) as StreamingBlobPayloadOutputTypes)
68+
);
69+
70+
const joinedStream = await joinStreams(streams);
71+
72+
const chunks = await consume(joinedStream);
73+
74+
expect(chunks.length).toBe(contents.length);
75+
chunks.forEach((chunk, i) => {
76+
expect(chunk).toEqual(contents[i]);
77+
});
78+
});
79+
80+
it("should handle consecutive calls of joining multiple streams into a single stream", async () => {
81+
for (let i = 0; i <= 3; i++) {
82+
const contents = [
83+
new Uint8Array([67, 104, 117, 110, 107, 32, 49]), // "Chunk 1"
84+
new Uint8Array([67, 104, 117, 110, 107, 32, 50]), // "Chunk 2"
85+
new Uint8Array([67, 104, 117, 110, 107, 32, 51]), // "Chunk 3"
86+
];
87+
88+
const streams = contents.map((content) =>
89+
Promise.resolve(sdkStreamMixin(createWithContent(content)) as StreamingBlobPayloadOutputTypes)
90+
);
91+
92+
const joinedStream = await joinStreams(streams);
93+
94+
const chunks = await consume(joinedStream);
95+
96+
expect(chunks.length).toBe(contents.length);
97+
chunks.forEach((chunk, i) => {
98+
expect(chunk).toEqual(contents[i]);
99+
});
100+
}
101+
});
102+
103+
it("should handle streams with no data", async () => {
104+
const streams = [
105+
Promise.resolve(sdkStreamMixin(createEmpty()) as StreamingBlobPayloadOutputTypes),
106+
Promise.resolve(sdkStreamMixin(createEmpty()) as StreamingBlobPayloadOutputTypes),
107+
];
108+
109+
const joinedStream = await joinStreams(streams);
110+
111+
const chunks = await consume(joinedStream);
112+
113+
expect(chunks.length).toBe(0);
114+
});
115+
116+
it("should report progress via eventListeners", async () => {
117+
const streams = [
118+
Promise.resolve(
119+
sdkStreamMixin(createWithContent(new Uint8Array([100, 97, 116, 97]))) as StreamingBlobPayloadOutputTypes
120+
), // "data"
121+
Promise.resolve(
122+
sdkStreamMixin(createWithContent(new Uint8Array([109, 111, 114, 101]))) as StreamingBlobPayloadOutputTypes
123+
), // "more"
124+
];
125+
126+
const onBytesSpy = vi.fn();
127+
const onCompletionSpy = vi.fn();
128+
129+
const joinedStream = await joinStreams(streams, {
130+
onBytes: onBytesSpy,
131+
onCompletion: onCompletionSpy,
132+
});
133+
134+
await consume(joinedStream);
135+
136+
expect(onBytesSpy).toHaveBeenCalled();
137+
expect(onCompletionSpy).toHaveBeenCalledWith(expect.any(Number), 1);
138+
});
139+
140+
it("should throw error for unsupported stream types", async () => {
141+
const blob = new Blob(["test"]);
142+
await expect(
143+
joinStreams([Promise.resolve(blob as unknown as StreamingBlobPayloadOutputTypes)])
144+
).rejects.toThrow("Unsupported Stream Type");
145+
});
146+
});
147+
});
148+
});

0 commit comments

Comments
 (0)