diff --git a/src/assets/images/changelog/r2/compaction.png b/src/assets/images/changelog/r2/compaction.png new file mode 100644 index 000000000000000..a593507c0349f0c Binary files /dev/null and b/src/assets/images/changelog/r2/compaction.png differ diff --git a/src/content/changelog/r2/2025-09-25-data-catalog-compaction.mdx b/src/content/changelog/r2/2025-09-25-data-catalog-compaction.mdx new file mode 100644 index 000000000000000..dc6ba4dcaae00f9 --- /dev/null +++ b/src/content/changelog/r2/2025-09-25-data-catalog-compaction.mdx @@ -0,0 +1,27 @@ +--- +title: R2 Data Catalog now supports compaction +description: Users can now enable compaction on R2 Data Catalog +products: + - r2 +date: 2025-09-25T13:00:00 +hidden: true +--- +import { + LinkCard, +} from "~/components"; + +You can now enable automatic compaction for [Apache Iceberg](https://iceberg.apache.org/) tables in [R2 Data Catalog](/r2/data-catalog/) to improve query performance. + +Compaction is the process of taking a group of small files and combining them into fewer larger files. This is an important maintenance operation as it helps ensure that query performance remains consistent by reducing the number of files that needs to be scanned. + +To enable automatic compaction in R2 Data Catalog, find it under **R2 Data Catalog** in your R2 bucket settings in the dashboard. + +![compaction-dash](~/assets/images/changelog/r2/compaction.png) + +Or with [Wrangler](/workers/wrangler/), run: + +```bash +npx wrangler r2 bucket catalog compaction enable --target-size 128 --token +``` + +To get started with compaction, check out [manage catalogs](/r2/data-catalog/manage-catalogs/). For best practices and limitations, refer to [about compaction](/r2/data-catalog/about-compaction/). \ No newline at end of file diff --git a/src/content/docs/r2/data-catalog/about-compaction.mdx b/src/content/docs/r2/data-catalog/about-compaction.mdx new file mode 100644 index 000000000000000..32ff9d45c73af24 --- /dev/null +++ b/src/content/docs/r2/data-catalog/about-compaction.mdx @@ -0,0 +1,41 @@ +--- +pcx_content_type: configuration +title: About compaction +description: Learn about R2 Data Catalog compaction +sidebar: + order: 4 +--- + +## What is compaction? + +Compaction is the process of taking a group of small files and combining them into fewer larger files. This is an important maintenance operation as it helps ensure that query performance remains consistent by reducing the number of files that needs to be scanned. + +## Why do I need compaction? + +Every write operation in [Apache Iceberg](https://iceberg.apache.org/), no matter how small or large, results in a series of new files being generated. As time goes on, the number of files can grow unbounded. This can lead to: +- Slower queries and increased I/O operations: Without compaction, query engines will have to open and read each individual file, resulting in longer query times and increased costs. +- Increased metadata overhead: Query engines must scan metadata files to determine which ones to read. With thousands of small files, query planning takes longer even before data is accessed. +- Reduced compression efficiency: Smaller files compress less efficiently than larger files, leading to higher storage costs and more data to transfer during queries. + +## R2 Data Catalog automatic compaction + +R2 Data Catalog can now [manage compaction](/r2/data-catalog/manage-catalogs) for Apache Iceberg tables stored in R2. When enabled, compaction runs automatically and combines new files that have not been compacted yet. + +Compacted files are prefixed with `compacted-` in the `/data/` directory. + +### Choosing the right target file size + +You can configure the target file size for compaction. Currently, the minimum is 64 MB and the maximum is 512 MB. + +Different compute engines have different optimal file sizes, so check their documentation. + +Performance tradeoffs depend on your use case. For example, queries that return small amounts of data may perform better with smaller files, as larger files could result in reading unnecessary data. +- For workloads that are more latency sensitive, consider a smaller target file size (for example, 64 MB - 128 MB) +- For streaming ingest workloads, consider medium file sizes (for example, 128 MB - 256 MB) +- For OLAP style queries that need to scan a lot of data, consider larger file sizes (for example, 256 MB - 512 MB) + +## Current limitations +- During open beta, compaction will compact up to 2 GB worth of files once per hour for each table. +- Only data files stored in parquet format are currently supported with compaction. +- Snapshot expiration and orphan file cleanup is not supported yet. +- Minimum target file size is 64 MB and maximum is 512 MB. diff --git a/src/content/docs/r2/data-catalog/config-examples/index.mdx b/src/content/docs/r2/data-catalog/config-examples/index.mdx index d0d4e7ea80b71ca..0010b7a970d64c8 100644 --- a/src/content/docs/r2/data-catalog/config-examples/index.mdx +++ b/src/content/docs/r2/data-catalog/config-examples/index.mdx @@ -3,7 +3,7 @@ pcx_content_type: navigation title: Connect to Iceberg engines head: [] sidebar: - order: 4 + order: 5 group: hideIndex: true description: Find detailed setup instructions for Apache Spark and other common query engines. diff --git a/src/content/docs/r2/data-catalog/manage-catalogs.mdx b/src/content/docs/r2/data-catalog/manage-catalogs.mdx index 1f9c4c576dbbbea..ec8642967252688 100644 --- a/src/content/docs/r2/data-catalog/manage-catalogs.mdx +++ b/src/content/docs/r2/data-catalog/manage-catalogs.mdx @@ -28,7 +28,8 @@ Learn how to: Enabling the catalog on a bucket turns on the REST catalog interface and provides a **Catalog URI** and **Warehouse name** required by Iceberg clients. Once enabled, you can create and manage Iceberg tables in that bucket. -### Dashboard + + 1. In the Cloudflare dashboard, go to the **R2 object storage** page. @@ -39,8 +40,8 @@ Enabling the catalog on a bucket turns on the REST catalog interface and provide 4. Once enabled, note the **Catalog URI** and **Warehouse name**. -### Wrangler CLI - + + To enable the catalog on your bucket, run the [`r2 bucket catalog enable command`](/workers/wrangler/commands/#r2-bucket-catalog-enable): ```bash @@ -48,13 +49,16 @@ npx wrangler r2 bucket catalog enable ``` After enabling, Wrangler will return your catalog URI and warehouse name. + + + ## Disable R2 Data Catalog on a bucket When you disable the catalog on a bucket, it immediately stops serving requests from the catalog interface. Any Iceberg table references stored in that catalog become inaccessible until you re-enable it. -### Dashboard - + + 1. In the Cloudflare dashboard, go to the **R2 object storage** page. @@ -63,13 +67,76 @@ When you disable the catalog on a bucket, it immediately stops serving requests 3. Switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and select **Disable**. -### Wrangler CLI - + + To disable the catalog on your bucket, run the [`r2 bucket catalog disable command`](/workers/wrangler/commands/#r2-bucket-catalog-disable): ```bash npx wrangler r2 bucket catalog disable ``` + + + +## Enable compaction +Compaction improves query performance by combining the many small files created during data ingestion into fewer, larger files according to the set `target file size`. For more information about compaction and why it's valuable, refer to [About compaction](/r2/data-catalog/about-compaction/). + + + + +1. In the Cloudflare dashboard, go to the **R2 object storage** page. + + +2. Select the bucket you want to enable compaction on. +3. Switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and click on the **Edit** icon next to the compaction card. +4. Enable compaction and optionally set a target file size. The default is 128 MB. +5. (Optional) Provide a Cloudflare API token for compaction to access and rewrite files in your bucket. +6. Select **Save**. + + + + + +To enable the compaction on your catalog, run the [`r2 bucket catalog enable command`](/workers/wrangler/commands/#r2-bucket-catalog-compaction-enable): + +```bash +npx wrangler r2 bucket catalog compaction enable --target-size 128 --token +``` + + + +:::note[API token permission requirements] +Compaction requires a Cloudflare API token with both R2 storage and R2 Data Catalog read/write permissions to act as a service credential. The compaction process uses this token to read files, combine them, and update table metadata. + +Refer to [Authenticate your Iceberg engine](#authenticate-your-iceberg-engine) for details on creating a token with the required permissions. + +Once enabled, compaction applies retroactively to all existing tables and automatically to newly created tables. During open beta, we currently compact up to 2 GB worth of files once per hour for each table. + +## Disable compaction +Disabling compaction will prevent the process from running for all tables managed by the catalog. You can re-enable it at any time. + + + + + +1. In the Cloudflare dashboard, go to the **R2 object storage** page. + + +2. Select the bucket you want to enable compaction on. +3. Switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and click on the **edit** icon next to the compaction card. +4. Disable compaction. +5. Select **Save**. + + + + + +To disable the compaction on your catalog, run the [`r2 bucket catalog disable command`](/workers/wrangler/commands/#r2-bucket-catalog-compaction-disable): + +```bash +npx wrangler r2 bucket catalog compaction disable +``` + + ## Authenticate your Iceberg engine diff --git a/src/content/partials/workers/wrangler-commands/r2.mdx b/src/content/partials/workers/wrangler-commands/r2.mdx index 9ceb5b01bf7dad7..63b9661e3db0115 100644 --- a/src/content/partials/workers/wrangler-commands/r2.mdx +++ b/src/content/partials/workers/wrangler-commands/r2.mdx @@ -106,6 +106,40 @@ wrangler r2 bucket catalog get [OPTIONS] - `NAME` - The name of the R2 bucket whose data catalog status to retrieve. + + +Enable compaction on a [R2 Data Catalog](/r2/data-catalog/). + +```txt +wrangler r2 bucket catalog compaction enable [OPTIONS] +``` + +- `BUCKET` + - The name of the bucket to enable R2 Data Catalog compaction for. +- `--token` + - The R2 API token with R2 Data Catalog edit permissions +- `--target-size` + - The target file size (in MB) compaction will attempt to generate. Default: 128. + + + +Disable compaction on a [R2 Data Catalog](/r2/data-catalog/). + +```txt +wrangler r2 bucket catalog compaction disable [OPTIONS] +``` + +- `BUCKET` + - The name of the bucket to enable R2 Data Catalog compaction for. + Set the [CORS configuration](/r2/buckets/cors/) for an R2 bucket from a JSON file.