-
Need Help Getting Started?
-
- The S3 connector is fully operational and ready for use. While we're finalizing the documentation,
- feel free to reach out to our support team for setup assistance.
+
+## Overview
+
+Amazon S3 (Simple Storage Service) is a scalable object storage service designed to store and retrieve any amount of data from anywhere on the web. It provides industry-leading scalability, data availability, security, and performance.
+
+The S3 connector enables you to sync files and folders from your S3 buckets into PipesHub, making your cloud storage content searchable through AI-powered search and accessible across your organization.
+
+### S3 Data Structure
+
+The connector understands S3's hierarchical structure:
+
+**Buckets → Folders → Files**
+
+| Entity | Description |
+|--------|-------------|
+| **Buckets** | Top-level containers for organizing objects (similar to root directories) |
+| **Folders** | Logical groupings of objects using prefixes (e.g., `documents/reports/`) |
+| **Files** | Individual objects stored in buckets with unique keys (paths) |
+
+### What Gets Synced
+
+The connector indexes the following content for AI-powered search:
+
+- **Files**: All file types stored in S3 buckets (documents, images, videos, archives, etc.)
+- **Folders**: Directory structure and organization
+- **Metadata**: File names, sizes, modification dates, MIME types
+- **Permissions**: Access control based on bucket and object permissions
+
+---
+
+## Configuration Guide
+
+
+
+
+
+
+
+### Setup Overview
+
+The S3 connector provides access to your AWS S3 buckets through Access Key authentication. It syncs files, folders, and metadata, enabling comprehensive search and access across your S3 content.
+
+### Authentication
+
+The S3 connector uses **AWS Access Key ID and Secret Access Key** for authentication. This method allows secure programmatic access to your S3 buckets without requiring OAuth setup.
+
+
+ Access Key authentication is simpler to set up than OAuth but requires you to manage AWS IAM credentials. Ensure you follow AWS security best practices when creating and storing access keys.
+
+
+## How to configure and enable the S3 Connector
+
+### Step 1: Create an IAM User in AWS
+
+1. **Sign in to AWS Management Console:**
+ Navigate to [console.aws.amazon.com](https://console.aws.amazon.com) and sign in with your AWS account credentials.
+
+2. **Access IAM Service:**
+ - In the AWS Management Console, search for **"IAM"** in the top search bar
+ - Click on **"IAM"** from the results to open the Identity and Access Management console
+
+
+
+
+
+3. **Navigate to Users:**
+ - In the left sidebar, click **"Users"**
+ - Click the **"Create user"** button (orange button at the top right)
+
+
+
+
+
+### Step 2: Configure User Details
+
+1. **Enter User Name:**
+ - Enter a descriptive name for the user (e.g., "PipesHub-S3-Connector")
+ - Click **"Next"** button
+
+
+
+
+
+2. **Set Permissions:**
+ - Select **"Attach policies directly"**
+ - Search for and select **"AmazonS3ReadOnlyAccess"** policy
+ - This policy provides read-only access to all S3 buckets
+
+
+ For more granular control, you can create a custom IAM policy that restricts access to specific buckets. See the "Advanced: Custom IAM Policy" section below for details.
+
+
+
+
+
+
+3. **Review and Create:**
+ - Review the user configuration
+ - Click **"Create user"** button at the bottom
+
+### Step 3: Create Access Keys
+
+1. **Access Security Credentials:**
+ - After creating the user, you'll be redirected to the user details page
+ - Click on the **"Security credentials"** tab
+ - Scroll down to the **"Access keys"** section
+
+
+
+
+
+2. **Create Access Key:**
+ - Click **"Create access key"** button
+ - Select **"Application running outside AWS"** as the use case
+ - Check the confirmation box
+ - Click **"Next"** button
+
+
+
+
+
+3. **Download or Copy Credentials:**
+ - **Access Key ID**: Copy this value immediately
+ - **Secret Access Key**: Click **"Show"** button and copy this value immediately
+
+
+
+ **Important:** The Secret Access Key is shown only once. If you don't save it now, you'll need to create a new access key pair. Store these credentials securely.
+
+
+
+
+
+
+
+4. **Download Credentials (Optional):**
+ - Click **"Download .csv file"** button to save credentials to a secure location
+ - Click **"Done"** button to complete the process
+
+### Step 4: Configure Connector in PipesHub
+
+1. **Navigate to Connectors:**
+ - In PipesHub, go to **Settings** → **Connectors**
+ - Find the **S3** connector in the list
+ - Click **"Configure"** button or link
+
+
+
+
+
+2. **Enter Access Credentials:**
+ - **Access Key**: Paste your AWS Access Key ID
+ - **Secret Key**: Paste your AWS Secret Access Key
+ - Click **"Save Auth Settings"** or **"Next"** button
+
+
+
+
+
+
+ Ensure you copy the credentials exactly as shown. Extra spaces or line breaks will cause authentication to fail.
+
+
+### Step 5: Configure Sync Settings
+
+1. **Choose Sync Strategy:**
+ - **Scheduled**: Automatically sync at regular intervals (default: every 60 minutes)
+ - **Manual**: Sync only when you trigger it manually
+
+
+
+
+
+2. **Set Sync Interval (if Scheduled):**
+ - Choose how often to sync (e.g., 15 min, 30 min, 1 hour, 4 hours, 24 hours)
+ - Default is 60 minutes
+
+
+ More frequent syncs keep your data up-to-date but consume more API resources. For large buckets, consider longer intervals to reduce AWS API usage.
+
+
+### Step 6: Configure Filters (Optional)
+
+Filters allow you to control what data is synced and indexed from your S3 buckets.
+
+
+
+
+
+#### Sync Filters
+
+Sync filters determine which content is downloaded from S3. Data excluded by sync filters is never downloaded.
+
+**Available Sync Filters:**
+
+1. **Bucket Names** - Filter by specific S3 buckets
+ - **Operator**: `In` (include only) or `Not In` (exclude)
+ - **Selection**: Choose from a **searchable dropdown list** of all your S3 buckets
+ - The dropdown displays all accessible buckets
+ - Type to search and filter buckets by name
+ - Select multiple buckets as needed
+
+
+ The bucket list is dynamically fetched from your AWS account. You don't need to manually enter bucket names - simply search and select from the dropdown.
+
+
+2. **File Extensions** - Filter files by extension
+ - **Operator**: `In` (include) or `Not In` (exclude)
+ - **Value**: Enter file extensions separated by commas (e.g., `pdf, docx, txt`)
+ - **Note**: Extensions are case-insensitive and don't require the leading dot
+ - **Use case**: Sync only specific file types (e.g., documents only, exclude images)
+
+3. **Modified Date** - Filter by last modification date
+ - **Operators**: `Is After`, `Is Before`, or `Is Between`
+ - **Use case**: Sync only recently updated files (e.g., files modified in the last 6 months)
+
+4. **Created Date** - Filter by creation date
+ - **Operators**: `Is After`, `Is Before`, or `Is Between`
+ - **Use case**: Sync only newly created files or exclude legacy files
+
+#### Indexing Filters
+
+Indexing filters control what synced data gets processed for AI search. All data is synced, but only enabled content types are indexed.
+
+**Available Indexing Filters:**
+
+- **Index Files** (default: enabled) - Include file content in search
+- **Index Folders** (default: enabled) - Include folder structure in search
+
+**Example Configurations:**
+
+- **Sync only specific buckets**: Bucket Names → Operator: `In`, select buckets from dropdown
+- **Sync only documents**: File Extensions → Operator: `In`, Value: `pdf, docx, txt, pptx`
+- **Exclude images**: File Extensions → Operator: `Not In`, Value: `jpg, png, gif, svg`
+- **Recent files only**: Modified Date → Operator: `Is After`, Date: `YYYY-MM-DD` (e.g., files modified in the last 6 months)
+- **Sync all but index only documents**: Enable all sync filters, disable indexing for non-document files
+
+
+ Filters can be configured during initial setup or modified later. Changes to filters will take effect on the next sync.
+
+
+### Step 7: Enable the Connector
+
+1. After configuration is complete, toggle the connector status to **"Enable"**
+2. You can do this by:
+ - Using the toggle switch next to **"Connector Status"**
+ - Or clicking the **"Enable"** button in the Quick Actions panel
+
+
+
+
+
+3. The connector will verify credentials and begin initial synchronization
+4. Wait for the status to show **"Active"** or **"Syncing"**
+5. Monitor the **Indexing Progress** to track sync completion
+
+## Advanced: Custom IAM Policy
+
+For enhanced security, you can create a custom IAM policy that restricts access to specific buckets or prefixes.
+
+### Example Custom Policy
+
+Here's a basic example policy for a single bucket:
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:ListAllMyBuckets",
+ "s3:GetBucketLocation"
+ ],
+ "Resource": "*"
+ },
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:ListBucket",
+ "s3:GetObject"
+ ],
+ "Resource": [
+ "arn:aws:s3:::your-bucket-name",
+ "arn:aws:s3:::your-bucket-name/*"
+ ]
+ }
+ ]
+}
+```
+
+### Enhanced Custom Policy (Multiple Buckets)
+
+For production use with multiple buckets and additional permissions, use this comprehensive policy:
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "VisualEditor0",
+ "Effect": "Allow",
+ "Action": [
+ "s3:GetObject",
+ "s3:ListBucket",
+ "s3:GetBucketLocation",
+ "s3:GetObjectVersion",
+ "s3:GetObjectAttributes",
+ "s3:GetObjectTagging"
+ ],
+ "Resource": [
+ "arn:aws:s3:::*/*",
+ "arn:aws:s3:::your-bucket-name-1",
+ "arn:aws:s3:::your-bucket-name-2",
+ "arn:aws:s3:::your-bucket-name-3"
+ ]
+ },
+ {
+ "Sid": "VisualEditor1",
+ "Effect": "Allow",
+ "Action": "s3:ListAllMyBuckets",
+ "Resource": "*"
+ }
+ ]
+}
+```
+
+
+ Replace the bucket names in the Resource array (`your-bucket-name-1`, `your-bucket-name-2`, `your-bucket-name-3`) with your actual bucket names. The `arn:aws:s3:::*/*` allows access to all objects in the specified buckets.
+
+
+**Steps to apply custom policy:**
+
+1. In IAM, go to **"Policies"** → **"Create policy"**
+2. Use the JSON editor to paste the policy above (replace `your-bucket-name` with your actual bucket name)
+3. Name the policy (e.g., "PipesHub-S3-ReadOnly-SpecificBuckets")
+4. Attach the policy to your IAM user instead of the managed policy
+
+## Supported Features
+
+The S3 connector syncs the following data from your AWS S3 buckets:
+
+- **Buckets**: All accessible buckets in your AWS account
+- **Files**: All file types with their metadata (name, size, modification date, MIME type)
+- **Folders**: Directory structure and organization using S3 prefixes
+- **Permissions**: Access control based on bucket and object-level permissions
+
+## Data Sync Behavior
+
+### Initial Sync
+
+The first sync performs a complete scan of your configured S3 buckets:
+
+- **Bucket Discovery**: Lists all accessible buckets in your AWS account (or uses configured bucket filters)
+- **Region Detection**: Automatically detects and caches the region for each bucket
+- **Full Object Listing**: Fetches all objects from selected buckets using pagination
+- **Metadata Extraction**: Extracts file names, sizes, modification dates, MIME types, and paths
+- **Folder Structure**: Creates logical folder structure from S3 object prefixes
+- **Indexing**: Indexes file content for AI-powered search (based on indexing filter settings)
+- **Permission Assignment**: Assigns permissions based on connector scope (Personal or Team)
+
+**Sync Duration Estimates:**
+
+| Bucket Size | Objects | Estimated Time |
+|-------------|---------|----------------|
+| Small | < 1,000 | 5-15 minutes |
+| Medium | 1,000 - 10,000 | 15-45 minutes |
+| Large | 10,000 - 100,000 | 1-3 hours |
+| Very Large | 100,000 - 1,000,000 | 3-8 hours |
+| Enterprise | 1,000,000+ | 8+ hours |
+
+
+ Use bucket and file extension filters to significantly reduce initial sync time. For example, syncing only PDF files from a specific bucket can reduce sync time by 80-90%.
+
+
+### Incremental Sync
+
+After the initial sync, subsequent syncs are much faster:
+
+- **Timestamp-Based Detection**: Uses `LastModified` timestamps to identify changed objects
+- **Sync Point Tracking**: Stores the last sync timestamp per bucket in sync points
+- **Change Detection**: Only fetches objects modified since the last sync timestamp
+- **Pagination Resume**: Uses continuation tokens to resume interrupted syncs
+- **Efficient Updates**: Processes only new, modified, or deleted objects
+- **API Optimization**: Reduces API calls by 90%+ compared to full syncs
+
+**How It Works:**
+
+1. Connector reads the sync point for each bucket (stored in our database)
+2. Retrieves the `last_sync_time` timestamp
+3. Queries S3 for objects with `LastModified >= last_sync_time`
+4. Processes only the changed objects
+5. Updates the sync point with the new maximum timestamp
+
+
+ Incremental syncs typically take 5-15 minutes for most buckets, regardless of total bucket size, as they only process changes.
+
+
+### Permission Handling
+
+The connector respects AWS IAM permissions and PipesHub access controls:
+
+- **IAM-Based Access**: Only syncs buckets and objects the IAM user has permission to access
+- **Bucket Policies**: Respects S3 bucket policies and access control lists (ACLs)
+- **PipesHub Scope**:
+ - **Personal Scope**: Only the connector creator can access synced content
+ - **Team Scope**: All organization users have read access to synced content
+- **Permission Inheritance**: Files inherit permissions from their parent bucket (Record Group)
+- **Access Control**: Users see only content they have permission to view in both AWS and PipesHub
+
+### Sync Frequency
+
+**Scheduled Sync:**
+- Default interval: 60 minutes
+- Configurable: 15 min, 30 min, 1 hour, 4 hours, 24 hours
+- Runs automatically in the background
+- Best for: Keeping data up-to-date with minimal manual intervention
+
+**Manual Sync:**
+- Triggered on-demand from the connector settings
+- Best for: Testing, troubleshooting, or syncing after bulk changes
+- Useful when: You need immediate sync after uploading many files
+
+
+ For large buckets with infrequent changes, consider longer sync intervals (4-24 hours) to reduce AWS API usage and costs.
+
+
+
+
Ready to Get Started?
+
+ Connect your S3 buckets to PipesHub in just a few minutes. Follow the step-by-step guide above to enable organization-wide file search and access across all your S3 content.
-
\ No newline at end of file
+
diff --git a/images/connectors/s3/access_key_credentials.png b/images/connectors/s3/access_key_credentials.png
new file mode 100644
index 0000000..888d64b
Binary files /dev/null and b/images/connectors/s3/access_key_credentials.png differ
diff --git a/images/connectors/s3/attach_policy.png b/images/connectors/s3/attach_policy.png
new file mode 100644
index 0000000..ad6ab34
Binary files /dev/null and b/images/connectors/s3/attach_policy.png differ
diff --git a/images/connectors/s3/create_access_key.png b/images/connectors/s3/create_access_key.png
new file mode 100644
index 0000000..186b820
Binary files /dev/null and b/images/connectors/s3/create_access_key.png differ
diff --git a/images/connectors/s3/create_user.png b/images/connectors/s3/create_user.png
new file mode 100644
index 0000000..2e33ef0
Binary files /dev/null and b/images/connectors/s3/create_user.png differ
diff --git a/images/connectors/s3/enable_connector.png b/images/connectors/s3/enable_connector.png
new file mode 100644
index 0000000..f8126a4
Binary files /dev/null and b/images/connectors/s3/enable_connector.png differ
diff --git a/images/connectors/s3/enter_credentials.png b/images/connectors/s3/enter_credentials.png
new file mode 100644
index 0000000..4645c55
Binary files /dev/null and b/images/connectors/s3/enter_credentials.png differ
diff --git a/images/connectors/s3/iam_console.png b/images/connectors/s3/iam_console.png
new file mode 100644
index 0000000..6f2cdea
Binary files /dev/null and b/images/connectors/s3/iam_console.png differ
diff --git a/images/connectors/s3/s3-filters.png b/images/connectors/s3/s3-filters.png
new file mode 100644
index 0000000..2f43e88
Binary files /dev/null and b/images/connectors/s3/s3-filters.png differ
diff --git a/images/connectors/s3/s3_config.png b/images/connectors/s3/s3_config.png
new file mode 100644
index 0000000..ed498cd
Binary files /dev/null and b/images/connectors/s3/s3_config.png differ
diff --git a/images/connectors/s3/security_credentials.png b/images/connectors/s3/security_credentials.png
new file mode 100644
index 0000000..d65035c
Binary files /dev/null and b/images/connectors/s3/security_credentials.png differ
diff --git a/images/connectors/s3/sync_strategy.png b/images/connectors/s3/sync_strategy.png
new file mode 100644
index 0000000..4cdf2a8
Binary files /dev/null and b/images/connectors/s3/sync_strategy.png differ
diff --git a/images/connectors/s3/user_name.png b/images/connectors/s3/user_name.png
new file mode 100644
index 0000000..58590e8
Binary files /dev/null and b/images/connectors/s3/user_name.png differ
-