|
| 1 | +--- |
| 2 | +title: AmazonS3 |
| 3 | +toc_max_heading_level: 4 |
| 4 | +description: CocoIndex AmazonS3 Built-in Sources |
| 5 | +--- |
| 6 | + |
| 7 | +### Setup for Amazon S3 |
| 8 | + |
| 9 | +#### Setup AWS accounts |
| 10 | + |
| 11 | +You need to setup AWS accounts to own and access Amazon S3. In particular, |
| 12 | + |
| 13 | +* Setup an AWS account from [AWS homepage](https://aws.amazon.com/) or login with an existing account. |
| 14 | +* AWS recommends all programming access to AWS should be done using [IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) instead of root account. You can create an IAM user at [AWS IAM Console](https://console.aws.amazon.com/iam/home). |
| 15 | +* Make sure your IAM user at least have the following permissions in the IAM console: |
| 16 | + * Attach permission policy `AmazonS3ReadOnlyAccess` for read-only access to Amazon S3. |
| 17 | + * (optional) Attach permission policy `AmazonSQSFullAccess` to receive notifications from Amazon SQS, if you want to enable change event notifications. |
| 18 | + Note that `AmazonSQSReadOnlyAccess` is not enough, as we need to be able to delete messages from the queue after they're processed. |
| 19 | + |
| 20 | + |
| 21 | +#### Setup Credentials for AWS SDK |
| 22 | + |
| 23 | +AWS SDK needs to access credentials to access Amazon S3. |
| 24 | +The easiest way to setup credentials is to run: |
| 25 | + |
| 26 | +```sh |
| 27 | +aws configure |
| 28 | +``` |
| 29 | + |
| 30 | +It will create a credentials file at `~/.aws/credentials` and config at `~/.aws/config`. |
| 31 | + |
| 32 | +See the following documents if you need more control: |
| 33 | + |
| 34 | +* [`aws configure`](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) |
| 35 | +* [Globally configuring AWS SDKs and tools](https://docs.aws.amazon.com/sdkref/latest/guide/creds-config-files.html) |
| 36 | + |
| 37 | + |
| 38 | +#### Create Amazon S3 buckets |
| 39 | + |
| 40 | +You can create a Amazon S3 bucket in the [Amazon S3 Console](https://s3.console.aws.amazon.com/s3/home), and upload your files to it. |
| 41 | + |
| 42 | +It's also doable by using the AWS CLI `aws s3 mb` (to create buckets) and `aws s3 cp` (to upload files). |
| 43 | +When doing so, make sure your current user also has permission policy `AmazonS3FullAccess`. |
| 44 | + |
| 45 | +#### (Optional) Setup SQS queue for event notifications |
| 46 | + |
| 47 | +You can setup an Amazon Simple Queue Service (Amazon SQS) queue to receive change event notifications from Amazon S3. |
| 48 | +It provides a change capture mechanism for your AmazonS3 data source, to trigger reprocessing of your AWS S3 files on any creation, update or deletion. Please use a dedicated SQS queue for each of your S3 data source. |
| 49 | + |
| 50 | +This is how to setup: |
| 51 | + |
| 52 | +* Create a SQS queue with proper access policy. |
| 53 | + * In the [Amazon SQS Console](https://console.aws.amazon.com/sqs/home), create a queue. |
| 54 | + * Add access policy statements, to make sure Amazon S3 can send messages to the queue. |
| 55 | + ```json |
| 56 | + { |
| 57 | + ... |
| 58 | + "Statement": [ |
| 59 | + ... |
| 60 | + { |
| 61 | + "Sid": "__publish_statement", |
| 62 | + "Effect": "Allow", |
| 63 | + "Principal": { |
| 64 | + "Service": "s3.amazonaws.com" |
| 65 | + }, |
| 66 | + "Resource": "${SQS_QUEUE_ARN}", |
| 67 | + "Action": "SQS:SendMessage", |
| 68 | + "Condition": { |
| 69 | + "ArnLike": { |
| 70 | + "aws:SourceArn": "${S3_BUCKET_ARN}" |
| 71 | + } |
| 72 | + } |
| 73 | + } |
| 74 | + ] |
| 75 | + } |
| 76 | + ``` |
| 77 | + |
| 78 | + Here, you need to replace `${SQS_QUEUE_ARN}` and `${S3_BUCKET_ARN}` with the actual ARN of your SQS queue and S3 bucket. |
| 79 | + You can find the ARN of your SQS queue in the existing policy statement (it starts with `arn:aws:sqs:`), and the ARN of your S3 bucket in the S3 console (it starts with `arn:aws:s3:`). |
| 80 | + |
| 81 | +* In the [Amazon S3 Console](https://s3.console.aws.amazon.com/s3/home), open your S3 bucket. Under *Properties* tab, click *Create event notification*. |
| 82 | + * Fill in an arbitrary event name, e.g. `S3ChangeNotifications`. |
| 83 | + * If you want your AmazonS3 data source to expose a subset of files sharing a prefix, set the same prefix here. Otherwise, leave it empty. |
| 84 | + * Select the following event types: *All object create events*, *All object removal events*. |
| 85 | + * Select *SQS queue* as the destination, and specify the SQS queue you created above. |
| 86 | + |
| 87 | +AWS's [Guide of Configuring a Bucket for Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification) provides more details. |
| 88 | + |
| 89 | +### Spec |
| 90 | + |
| 91 | +The spec takes the following fields: |
| 92 | +* `bucket_name` (`str`): Amazon S3 bucket name. |
| 93 | +* `prefix` (`str`, optional): if provided, only files with path starting with this prefix will be imported. |
| 94 | +* `binary` (`bool`, optional): whether reading files as binary (instead of text). |
| 95 | +* `included_patterns` (`list[str]`, optional): a list of glob patterns to include files, e.g. `["*.txt", "docs/**/*.md"]`. |
| 96 | + If not specified, all files will be included. |
| 97 | +* `excluded_patterns` (`list[str]`, optional): a list of glob patterns to exclude files, e.g. `["*.tmp", "**/*.log"]`. |
| 98 | + Any file or directory matching these patterns will be excluded even if they match `included_patterns`. |
| 99 | + If not specified, no files will be excluded. |
| 100 | + |
| 101 | + :::info |
| 102 | + |
| 103 | + `included_patterns` and `excluded_patterns` are using Unix-style glob syntax. See [globset syntax](https://docs.rs/globset/latest/globset/index.html#syntax) for the details. |
| 104 | + |
| 105 | + ::: |
| 106 | + |
| 107 | +* `sqs_queue_url` (`str`, optional): if provided, the source will receive change event notifications from Amazon S3 via this SQS queue. |
| 108 | + |
| 109 | + :::info |
| 110 | + |
| 111 | + We will delete messages from the queue after they're processed. |
| 112 | + If there are unrelated messages in the queue (e.g. test messages that SQS will send automatically on queue creation, messages for a different bucket, for non-included files, etc.), we will delete the message upon receiving it, to avoid repeatedly receiving irrelevant messages after they're redelivered. |
| 113 | + |
| 114 | + ::: |
| 115 | + |
| 116 | +### Schema |
| 117 | + |
| 118 | +The output is a [*KTable*](/docs/core/data_types#ktable) with the following sub fields: |
| 119 | + |
| 120 | +* `filename` (*Str*, key): the filename of the file, including the path, relative to the root directory, e.g. `"dir1/file1.md"`. |
| 121 | +* `content` (*Str* if `binary` is `False`, otherwise *Bytes*): the content of the file. |
0 commit comments