-
Notifications
You must be signed in to change notification settings - Fork 15
docs: add cloud metadata guide and map it to iam-role-assume + PrivateLink setup #1073
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Copilot
wants to merge
6
commits into
main
Choose a base branch
from
copilot/add-cloud-metadata-guide
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+250
−16
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
215f200
docs: add cloud metadata guide and map it to iam-role-assume + Privat…
Copilot 634e61f
docs: address review comments on cloud-metadata page
Copilot 2d3ae40
docs: fix console navigation to Connection tab and add Cloud Meta/Pri…
Copilot bcc5deb
docs: add missing screenshots for cloud-metadata and iam-role-assume …
cyberchen98 ef301b9
docs: align Console field names, fix nav paths, and remove stale exte…
Copilot 594d0c4
docs: remove enable_config_load Cloud/self-hosted restriction — now u…
Copilot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| title: "Cloud metadata" | ||
| sidebarTitle: "Cloud metadata" | ||
| description: "View and use the cloud metadata values exposed in the RisingWave Cloud Console, including the IAM role ARN, PrivateLink Principal, and egress public IPs for your project." | ||
| --- | ||
|
|
||
| RisingWave Cloud exposes a set of environment-specific metadata values for each project. These values are required when setting up cross-account access (IAM role assume) and PrivateLink connections. All values are read-only and are generated automatically when a project is created. | ||
|
|
||
| ## Where to find cloud metadata | ||
|
|
||
| In the [RisingWave Cloud Console](https://cloud.risingwave.com), go to your project and click **Connection** in the left sidebar. Then select the **Cloud Meta** tab. | ||
|
|
||
| <Frame> | ||
| <img src="/images/cloud/cloud-metadata/connection-cloud-meta.png" alt="Cloud Meta tab in the Connection page of the RisingWave Cloud Console"/> | ||
| </Frame> | ||
|
|
||
| <Note> | ||
| Cloud metadata is only available for projects on the **Standard** plan or above. | ||
| </Note> | ||
|
|
||
| ## Metadata fields | ||
|
|
||
| ### Workload Identity (IAM Role ARN) | ||
|
|
||
| | Field | Example value | | ||
| |:------|:--------------| | ||
| | **Workload Identity (IAM Role ARN)** | `arn:aws:iam::023339134545:role/g1jk13sq56ejdref71h1cvokbh-role` | | ||
|
|
||
| The AWS IAM role ARN that RisingWave Cloud uses to access AWS resources on behalf of this project. When you configure [IAM role assume](/cloud/iam-role-assume) (cross-account S3 access), you add this ARN as a trusted principal in your IAM role's trust policy. | ||
|
|
||
| ### PrivateLink Principal | ||
|
|
||
| | Field | Example value | | ||
| |:------|:--------------| | ||
| | **PrivateLink Principal** | `arn:aws:iam::023339134545:role/test-useast1-eks-a-cloudagent-role` | | ||
|
|
||
| The AWS principal associated with the RisingWave Cloud deployment that hosts your project. When you create a PrivateLink endpoint service in your AWS account, add this principal to the list of **Allowed principals** so that RisingWave Cloud can connect to your service. | ||
|
|
||
| ### Egress public IPs | ||
|
|
||
| | Field | Example value | | ||
| |:------|:--------------| | ||
| | **Egress public IPs** | `203.0.113.10, 203.0.113.11` | | ||
|
|
||
| The public IP addresses from which outbound traffic from this project originates. Add these IPs to the allowlist of any firewall rules or security groups that restrict inbound access to your services (for example, a database or Kafka cluster that RisingWave connects to). | ||
|
|
||
| ## Cloud metadata by platform | ||
|
|
||
| | Metadata field | AWS | GCP | Azure | | ||
| |:---------------|:----|:----|:------| | ||
| | Workload Identity (IAM Role ARN) | ✅ | ✅ (service account email) | ✅ (managed identity resource ID) | | ||
| | PrivateLink Principal | ✅ (AWS account ARN) | ✅ (GCP project number) | ✅ (Azure subscription ID) | | ||
| | Egress public IPs | ✅ | ✅ | ✅ | | ||
|
|
||
| <Note> | ||
| GCP and Azure metadata field names differ from the AWS equivalents. The Console labels each field for the platform of your project. | ||
| </Note> | ||
|
|
||
| ## Next steps | ||
|
|
||
| - [Set up IAM role assume](/cloud/iam-role-assume) — use the IAM role ARN to grant RisingWave Cloud cross-account S3 access. | ||
| - [Configure PrivateLink](/cloud/create-a-connection) — use the PrivateLink Principal when setting up your endpoint service's allowed principals. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| --- | ||
| title: "Set up IAM role assume" | ||
| sidebarTitle: "IAM role assume" | ||
| description: "Configure cross-account S3 access so that RisingWave Cloud can assume an IAM role in your AWS account without sharing long-term credentials." | ||
| --- | ||
|
|
||
| IAM role assume (also called cross-account role delegation) lets RisingWave Cloud call `sts:AssumeRole` to obtain temporary credentials for your AWS IAM role. This avoids storing long-term access keys and follows AWS security best practices. | ||
|
|
||
| Use this method when you want RisingWave Cloud to read from or write to an S3 bucket in **your** AWS account using a role you control. | ||
|
|
||
| ## How it works | ||
|
|
||
| ``` | ||
| RisingWave Cloud (AWS account: <rw-account-id>) | ||
| → calls sts:AssumeRole | ||
| → on: arn:aws:iam::<your-account-id>:role/<your-role> | ||
| → receives temporary credentials | ||
| → accesses your S3 bucket | ||
| ``` | ||
|
|
||
| 1. You create an IAM role in your AWS account with the required S3 permissions. | ||
| 2. You configure the role's trust policy to allow RisingWave Cloud's IAM role ARN to assume it. | ||
| 3. You set `s3.assume_role` (or `s3.iam_role_arn`) in your RisingWave SQL statement to the ARN of your role. | ||
|
|
||
| ## Required cloud metadata values | ||
|
|
||
| Before you begin, retrieve the following values from your project's [Cloud metadata](/cloud/cloud-metadata) page in the RisingWave Cloud Console (**Connection** → **Cloud Meta** tab): | ||
|
|
||
| | Metadata field | Where it's used | | ||
| |:---------------|:----------------| | ||
| | **Workload Identity (IAM Role ARN)** | The `Principal.AWS` value in your IAM trust policy | | ||
|
|
||
| ## Step-by-step setup | ||
|
|
||
| ### Step 1 — Create an IAM policy for S3 access | ||
|
|
||
| Create a policy that grants RisingWave the necessary permissions on your S3 bucket. Replace `<bucket>` and `<prefix>` with your values. | ||
|
|
||
| ```json Example: S3 read/write policy | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "s3:PutObject", | ||
| "s3:GetObject", | ||
| "s3:GetObjectVersion", | ||
| "s3:DeleteObject", | ||
| "s3:DeleteObjectVersion" | ||
| ], | ||
| "Resource": "arn:aws:s3:::<bucket>/<prefix>/*" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "s3:ListBucket", | ||
| "s3:GetBucketLocation" | ||
| ], | ||
| "Resource": "arn:aws:s3:::<bucket>", | ||
| "Condition": { | ||
| "StringLike": { | ||
| "s3:prefix": ["<prefix>/*"] | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 2 — Create an IAM role with the policy attached | ||
|
|
||
| 1. In the [AWS IAM console](https://console.aws.amazon.com/iam/), create a new role. | ||
| 2. Select **Another AWS account** as the trusted entity type and enter the AWS account ID extracted from the **Workload Identity (IAM Role ARN)** (the 12-digit number after `iam::`). | ||
| 3. Attach the S3 policy created in Step 1. | ||
|
|
||
| ### Step 3 — Configure the trust policy | ||
|
|
||
| After the role is created, update its trust policy to allow the specific RisingWave Cloud IAM role to assume it. | ||
|
|
||
| Replace `<iam-role-arn>` with the **Workload Identity (IAM Role ARN)** from Cloud metadata. | ||
|
|
||
| ```json Example: Trust policy | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Principal": { | ||
| "AWS": "<iam-role-arn>" | ||
| }, | ||
| "Action": "sts:AssumeRole" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 4 — Use the role in RisingWave | ||
|
|
||
| Set `s3.assume_role` to the ARN of the role you created in Step 2 and set `enable_config_load = 'true'`. | ||
|
|
||
| ```sql Example: Sink using IAM role assume | ||
| CREATE SINK my_sink FROM my_source WITH ( | ||
| connector = 'redshift', -- or 'snowflake_v2', 'iceberg', etc. | ||
|
|
||
| -- S3 staging | ||
| s3.bucket_name = '<your-bucket>', | ||
| s3.region_name = '<your-region>', | ||
| s3.path = '<your-prefix>', | ||
|
|
||
| -- IAM role assume (no long-term credentials needed) | ||
| s3.assume_role = 'arn:aws:iam::<your-account-id>:role/<your-role>', | ||
| enable_config_load = 'true', | ||
|
|
||
| -- other connector-specific parameters... | ||
| ); | ||
| ``` | ||
|
|
||
| For connectors that use `s3.iam_role_arn` instead of `s3.assume_role` (for example, the Iceberg connector), use: | ||
|
|
||
| ```sql Example: Iceberg connector with IAM role assume | ||
| CREATE SINK my_iceberg_sink FROM my_mv WITH ( | ||
| connector = 'iceberg', | ||
| type = 'append-only', | ||
| -- ... | ||
| s3.iam_role_arn = 'arn:aws:iam::<your-account-id>:role/<your-role>', | ||
| enable_config_load = 'true', | ||
| ); | ||
| ``` | ||
|
|
||
| ## Verify the setup | ||
|
|
||
| After creating the sink or source, run a quick sanity check: | ||
|
|
||
| 1. **Check that the role is assumable** — in the AWS console, use **IAM → Roles → \<your-role\> → Trust relationships** to confirm the correct principal and condition are present. | ||
| 2. **Check S3 permissions** — run an `aws s3 ls s3://<bucket>/<prefix>/` command using the role (via `aws sts assume-role`) to verify read access. | ||
| 3. **Check RisingWave logs** — if the sink or source fails, look for `AccessDenied` or `InvalidClientTokenId` errors. These indicate a misconfigured trust policy or incorrect ARN. | ||
|
|
||
| ## Common errors | ||
|
|
||
| | Error | Likely cause | Fix | | ||
| |:------|:-------------|:----| | ||
| | `AccessDenied` when calling `AssumeRole` | The trust policy principal doesn't match the RisingWave Cloud IAM role ARN | Copy the exact **Workload Identity (IAM Role ARN)** from Cloud metadata and update the trust policy | | ||
| | `AccessDenied` on S3 operations after assuming | The attached S3 policy is missing required actions | Add the missing S3 actions (see Step 1) | | ||
| | `InvalidClientTokenId` | Incorrect IAM role ARN in the trust policy | Use the **Workload Identity (IAM Role ARN)** from Cloud metadata | | ||
|
|
||
| ## Managing allowed principals in RisingWave Cloud | ||
|
|
||
| To view and manage the IAM role ARNs that are authorized to access your project's resources via IAM Assume Role, go to your project in the [RisingWave Cloud Console](https://cloud.risingwave.com), click **Connection** in the left sidebar, and select the **Allowed Principals** tab. | ||
|
|
||
| <Frame> | ||
| <img src="/images/cloud/iam-role-assume/connection-allowed-principals.png" alt="Allowed Principals tab in the Connection page of the RisingWave Cloud Console"/> | ||
| </Frame> | ||
|
|
||
| Click **+ Add Principal** to add an IAM Role ARN that is allowed to access this project's resources. | ||
|
|
||
| ## Related pages | ||
|
|
||
| - [Cloud metadata](/cloud/cloud-metadata) — retrieve the IAM role ARN for your project | ||
| - [PrivateLink connection configuration](/cloud/create-a-connection) — set up private network connectivity | ||
| - [Sink to Amazon Redshift](/integrations/destinations/redshift) — end-to-end example with S3 staging | ||
| - [Sink to Snowflake](/integrations/destinations/snowflake-v2) — end-to-end example with S3 staging |
cyberchen98 marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.