Skip to content

Commit 371c4fc

Browse files
soyeric128soyeric128
authored
added (#1702)
1 parent 80759af commit 371c4fc

File tree

3 files changed

+108
-0
lines changed

3 files changed

+108
-0
lines changed

‎docs/en/guides/51-access-data-lake/02-iceberg.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,13 @@ Switches the current session to the specified catalog.
124124
USE CATALOG <catalog_name>
125125
```
126126

127+
## Iceberg Table Functions
128+
129+
Databend provides the following table functions for querying Iceberg metadata, allowing users to inspect snapshots and manifests efficiently:
130+
131+
- [ICEBERG_MANIFEST](/sql/sql-functions/table-functions/iceberg-manifest)
132+
- [ICEBERG_SNAPSHOT](/sql/sql-functions/table-functions/iceberg-snapshot)
133+
127134
## Usage Examples
128135

129136
This example shows how to create an Iceberg catalog using a REST-based connection, specifying the service address, warehouse location (S3), and optional parameters like AWS region and custom endpoint:

‎docs/en/sql-reference/20-sql-functions/17-table-functions/iceberg-manifest.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
title: ICEBERG_MANIFEST
3+
---
4+
import FunctionDescription from '@site/src/components/FunctionDescription';
5+
6+
<FunctionDescription description="Introduced or updated: v1.2.709"/>
7+
8+
Returns metadata about manifest files of an Iceberg table, including file paths, partitioning details, and snapshot associations.
9+
10+
## Syntax
11+
12+
```sql
13+
ICEBERG_MANIFEST('<database_name>', '<table_name>');
14+
```
15+
16+
## Output
17+
18+
The function returns a table with the following columns:
19+
20+
- `content` (`INT`): The content type (0 for data files, 1 for delete files).
21+
- `path` (`STRING`): The file path of the data or delete file.
22+
- `length` (`BIGINT`): The file size in bytes.
23+
- `partition_spec_id` (`INT`): The partition specification ID associated with the file.
24+
- `added_snapshot_id` (`BIGINT`): The snapshot ID that added this file.
25+
- `added_data_files_count` (`INT`): The number of new data files added.
26+
- `existing_data_files_count` (`INT`): The number of existing data files referenced.
27+
- `deleted_data_files_count` (`INT`): The number of data files deleted.
28+
- `added_delete_files_count` (`INT`): The number of delete files added.
29+
- `partition_summaries` (`MAP<STRING, STRING>`): Summary of partition values related to the file.
30+
31+
## Examples
32+
33+
```sql
34+
SELECT * FROM ICEBERG_MANIFEST('tpcds', 'catalog_returns');
35+
36+
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
37+
│ content │ path │ length │ partition_spec │ added_snapshot │ added_data_fil │ existing_data_ │ deleted_data_ │ added_delete_ │ existing_dele │ deleted_delet │ partition_sum │
38+
│ Int32 │ String │ Int64 │ _id │ _id │ es_count │ files_count │ files_count │ files_count │ te_files_coun │ e_files_count │ maries │
39+
│ │ │ │ Int32 │ Nullable(Int64 │ Nullable(Int32 │ Nullable(Int32 │ Nullable(Int3 │ Nullable(Int3 │ t │ Nullable(Int3 │ Array(Nullabl │
40+
│ │ │ │ │ ) │ ) │ ) │ 2) │ 2) │ Nullable(Int3 │ 2) │ e(Tuple(Nulla │
41+
│ │ │ │ │ │ │ │ │ │ 2) │ │ ble(Boolean), │
42+
│ │ │ │ │ │ │ │ │ │ │ │ Nullable(Bool │
43+
│ │ │ │ │ │ │ │ │ │ │ │ ean), String, │
44+
│ │ │ │ │ │ │ │ │ │ │ │ String))) │
45+
├─────────┼────────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼───────────────┼───────────────┼───────────────┼───────────────┼───────────────┤
46+
│ 0 │ s3://warehouse │ 9241 │ 0 │ 75657674165904 │ 2 │ 0 │ 0 │ 2 │ 0 │ 0 │ [] │
47+
│ │ /catalog_retur │ │ │ 11866 │ │ │ │ │ │ │ │
48+
│ │ ns/metadata/fa │ │ │ │ │ │ │ │ │ │ │
49+
│ │ 1ea4d5-a382-49 │ │ │ │ │ │ │ │ │ │ │
50+
│ │ 7a-9f22-1acb9a │ │ │ │ │ │ │ │ │ │ │
51+
│ │ 74a346-m0.avro │ │ │ │ │ │ │ │ │ │ │
52+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
53+
```

‎docs/en/sql-reference/20-sql-functions/17-table-functions/iceberg-snapshot.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
title: ICEBERG_SNAPSHOT
3+
---
4+
import FunctionDescription from '@site/src/components/FunctionDescription';
5+
6+
<FunctionDescription description="Introduced or updated: v1.2.709"/>
7+
8+
Returns metadata about snapshots of an Iceberg table, including information about data changes, operations, and summary statistics.
9+
10+
## Syntax
11+
12+
```sql
13+
ICEBERG_SNAPSHOT('<database_name>', '<table_name>');
14+
```
15+
16+
## Output
17+
18+
The function returns a table with the following columns:
19+
20+
- `committed_at` (`TIMESTAMP`): The timestamp when the snapshot was committed.
21+
- `snapshot_id` (`BIGINT`): The unique identifier of the snapshot.
22+
- `parent_id` (`BIGINT`): The parent snapshot ID, if applicable.
23+
- `operation` (`STRING`): The type of operation performed (e.g., append, overwrite, delete).
24+
- `manifest_list` (`STRING`): The file path of the manifest list associated with the snapshot.
25+
- `summary` (`MAP<STRING, STRING>`): A JSON-like structure containing additional metadata, such as:
26+
- `added-data-files`: Number of newly added data files.
27+
- `added-records`: Number of new records added.
28+
- `total-records`: Total number of records in the snapshot.
29+
- `total-files-size`: Total size of all data files (in bytes).
30+
- `total-data-files`: Total number of data files in the snapshot.
31+
- `total-delete-files`: Total number of delete files in the snapshot.
32+
33+
## Examples
34+
35+
```sql
36+
SELECT * FROM ICEBERG_SNAPSHOT('tpcds', 'catalog_returns');
37+
38+
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
39+
│ committed_at │ snapshot_id │ parent_id │ operation │ manifest_list │ summary │
40+
├────────────────────────────┼─────────────────────┼───────────┼───────────┼──────────────────────────────────────────────────────┼─────────────────────────────────────────────────────┤
41+
│ 2025-03-12 23:18:26.626000 │ 7565767416590411866 │ 0 │ append │ s3://warehouse/catalog_returns/metadata/snap-7565767 │ {'spark.app.id':'local-1741821433430','added-data-f │
42+
│ │ │ │ │ 416590411866-1-fa1ea4d5-a382-497a-9f22-1acb9a74a346. │ iles':'2','added-records':'144067','total-equality- │
43+
│ │ │ │ │ avro │ deletes':'0','changed-partition-count':'1','total-r │
44+
│ │ │ │ │ │ ecords':'144067','total-files-size':'7679811','tota │
45+
│ │ │ │ │ │ l-data-files':'2','added-files-size':'7679811','tot │
46+
│ │ │ │ │ │ al-delete-files':'0','total-position-deletes':'0'} │
47+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
48+
```

0 commit comments

Comments
 (0)