Skip to content

Commit 5fddc50

Browse files
ianton-ruianton-ru
committed
Swarm documentation
1 parent 1a190de commit 5fddc50

File tree

9 files changed

+257
-0
lines changed

9 files changed

+257
-0
lines changed

‎docs/en/engines/table-engines/integrations/iceberg.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,64 @@ CREATE TABLE example_table ENGINE = Iceberg(
296296

297297
`Iceberg` table engine and table function support metadata cache storing the information of manifest files, manifest list and metadata json. The cache is stored in memory. This feature is controlled by setting `use_iceberg_metadata_files_cache`, which is enabled by default.
298298

299+
## Altinity Antalya branch
300+
301+
### Specify storage type in arguments
302+
303+
Only in Altinity Antalya branch table engine `Iceberg` can work with all storages.
304+
In this case storage may be specified with named argument `storage_type`.
305+
Possible values are `s3`, `azure`, `hdfs`, `local`:
306+
307+
```sql
308+
CREATE TABLE iceberg_table_s3
309+
ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression])
310+
311+
CREATE TABLE iceberg_table_azure
312+
ENGINE = Iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [account_name, account_key, format, compression])
313+
314+
CREATE TABLE iceberg_table_hdfs
315+
ENGINE = Iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method])
316+
317+
CREATE TABLE iceberg_table_local
318+
ENGINE = Iceberg(storage_type='local', path_to_table, [,format] [,compression_method])
319+
```
320+
321+
### Specify storage type in named collection
322+
323+
Only in Altinity Antalya branch `storage_type` may be part of named collection.
324+
325+
```xml
326+
<clickhouse>
327+
<named_collections>
328+
<iceberg_conf>
329+
<url>http://test.s3.amazonaws.com/clickhouse-bucket/</url>
330+
<access_key_id>test<access_key_id>
331+
<secret_access_key>test</secret_access_key>
332+
<format>auto</format>
333+
<structure>auto</structure>
334+
<storage_type>s3</storage_type>
335+
</iceberg_conf>
336+
</named_collections>
337+
</clickhouse>
338+
```
339+
340+
```sql
341+
CREATE TABLE iceberg_table ENGINE=Iceberg(iceberg_conf, filename = 'test_table')
342+
```
343+
344+
Default value for `storage_type` is `s3`.
345+
346+
### `object_storage_cluster` setting.
347+
348+
Only in Altinity Antalya branch alternative syntax for `Iceberg` table engine can be executed on cluster with non-empty setting `object_storage_cluster` with cluster name.
349+
350+
```sql
351+
CREATE TABLE iceberg_table_s3
352+
ENGINE = Iceberg(storage_type='s3', url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression]);
353+
354+
SELECT * FROM iceberg_table_s3 SETTINGS object_storage_cluster='cluster_simple';
355+
```
356+
299357
## See also {#see-also}
300358

301359
- [iceberg table function](/sql-reference/table-functions/iceberg.md)

‎docs/en/sql-reference/distribution-on-cluster.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Task distribution in *Cluster family functions
2+
3+
Table functions `s3Cluster`, `azureBlobStorageCluster`, `hdsfCluster`, `icebergCluster`, etc., as well as table engines `S3`, `Azure`, `HDFS`, `Iceberg` with setting `object_storage_cluster` distribute tasks between all cluster nodes, or only between `object_storage_max_nodes` cluster nodes. Setting `object_storage_max_nodes` limits number of nodes to process distributed query. In this case random nodes are choosed for each query.
4+
5+
Single task is a single source file.
6+
7+
For each file one of cluster nodes selected as primary node. Primary node selected with consistence Rendezvous Hashing algorythm, this algorythm guarantees that for each file sthe same node selected as primary when cluster not changed, and when cluster is changed only files from deleted nodes or with new nodes as primary are affected. This incerasy cache efficiency.
8+
Each node starts to process files for which this node is primary. When node processes all those files it can take some files from other nodes. Node gets file immediately or only when primary node does not ask for new files in `lock_object_storage_task_distribution_ms` milliseconds. Settiing `lock_object_storage_task_distribution_ms` has default value in 500 milliseconds, and can be used to achieve balance between caching and processing files when some nodes are overloaded.
9+
10+
If node must be shutdowned in some time, command `SYSTEM STOP SWARM MODE` can be used to stop getting new tasks for `*Cluster`-family queries. In this case node stop getting new files, but processes alrready started files. When it processes all files, node can be shutdowned withou any error on initiator.
11+
12+
Getting new tasks can be reenabled with `SYSTEM START SWARM MODE` command.

‎docs/en/sql-reference/table-functions/azureBlobStorageCluster.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,20 @@ SELECT count(*) FROM azureBlobStorageCluster(
5353

5454
See [azureBlobStorage](/sql-reference/table-functions/azureBlobStorage#using-shared-access-signatures-sas-sas-tokens) for examples.
5555

56+
## Altinity Antalya branch
57+
58+
### `object_storage_cluster` setting.
59+
60+
Only in Altinity Antalya branch alternative syntax for `azureBlobStorageCluster` table function is `azureBlobStorage` function with non-empty setting `object_storage_cluster` with cluster name.
61+
62+
```sql
63+
SELECT count(*) FROM azureBlobStorage(
64+
'http://azurite1:10000/devstoreaccount1', 'testcontainer', 'test_cluster_count.csv', 'devstoreaccount1',
65+
'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV',
66+
'auto', 'key UInt64')
67+
SETTINGS object_storage_cluster='cluster_simple'
68+
```
69+
5670
## Related {#related}
5771

5872
- [AzureBlobStorage engine](../../engines/table-engines/integrations/azureBlobStorage.md)

‎docs/en/sql-reference/table-functions/deltalakeCluster.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,17 @@ A table with the specified structure for reading data from cluster in the specif
3636
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
3737
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.
3838

39+
## Altinity Antalya branch
40+
41+
### `object_storage_cluster` setting.
42+
43+
Only in Altinity Antalya branch alternative syntax for `deltaLakeCluster` table function is `deltaLake` function with non-empty setting `object_storage_cluster` with cluster name.
44+
45+
```sql
46+
SELECT count(*) FROM deltaLake(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
47+
SETTINGS object_storage_cluster='cluster_simple'
48+
```
49+
3950
## Related {#related}
4051

4152
- [deltaLake engine](engines/table-engines/integrations/deltalake.md)

‎docs/en/sql-reference/table-functions/hdfsCluster.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,18 @@ FROM hdfsCluster('cluster_simple', 'hdfs://hdfs1:9000/{some,another}_dir/*', 'TS
5959
If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`.
6060
:::
6161

62+
## Altinity Antalya branch
63+
64+
### `object_storage_cluster` setting.
65+
66+
Only in Altinity Antalya branch alternative syntax for `hdfsCluster` table function is `hdfs` function with non-empty setting `object_storage_cluster` with cluster name.
67+
68+
```sql
69+
SELECT count(*)
70+
FROM hdfs('hdfs://hdfs1:9000/{some,another}_dir/*', 'TSV', 'name String, value UInt32')
71+
SETTINGS object_storage_cluster='cluster_simple'
72+
```
73+
6274
## Related {#related}
6375

6476
- [HDFS engine](../../engines/table-engines/integrations/hdfs.md)

‎docs/en/sql-reference/table-functions/hudiCluster.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,18 @@ A table with the specified structure for reading data from cluster in the specif
4242
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
4343
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.
4444

45+
## Altinity Antalya branch
46+
47+
### `object_storage_cluster` setting.
48+
49+
Only in Altinity Antalya branch alternative syntax for `hudiCluster` table function is `hudi` function with non-empty setting `object_storage_cluster` with cluster name.
50+
51+
```sql
52+
SELECT *
53+
FROM hudi(url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
54+
SETTINGS object_storage_cluster='cluster_simple'
55+
```
56+
4557
## Related {#related}
4658

4759
- [Hudi engine](engines/table-engines/integrations/hudi.md)

‎docs/en/sql-reference/table-functions/iceberg.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,49 @@ Table function `iceberg` is an alias to `icebergS3` now.
302302
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
303303
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.
304304

305+
## Altinity Antalya branch
306+
307+
### Specify storage type in arguments
308+
309+
Only in Altinity Antalya branch table function `iceberg` can work with all storages.
310+
In this case storage may be specified with named argument `storage_type`.
311+
Possible values are `s3`, `azure`, `hdfs`, `local`:
312+
313+
```sql
314+
iceberg(storage_type='s3', url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method])
315+
316+
iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method])
317+
318+
iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method])
319+
320+
iceberg(storage_type='local', path_to_table, [,format] [,compression_method])
321+
```
322+
323+
### Specify storage type in named collection
324+
325+
Only in Altinity Antalya branch `storage_type` may be part of named collection.
326+
327+
```xml
328+
<clickhouse>
329+
<named_collections>
330+
<iceberg_conf>
331+
<url>http://test.s3.amazonaws.com/clickhouse-bucket/</url>
332+
<access_key_id>test<access_key_id>
333+
<secret_access_key>test</secret_access_key>
334+
<format>auto</format>
335+
<structure>auto</structure>
336+
<storage_type>s3</storage_type>
337+
</iceberg_conf>
338+
</named_collections>
339+
</clickhouse>
340+
```
341+
342+
```sql
343+
iceberg(named_collection[, option=value [,..]])
344+
```
345+
346+
Default value for `storage_type` is `s3`.
347+
305348
## See Also {#see-also}
306349

307350
* [Iceberg engine](/engines/table-engines/integrations/iceberg.md)

‎docs/en/sql-reference/table-functions/icebergCluster.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,84 @@ SELECT * FROM icebergS3Cluster('cluster_simple', 'http://test.s3.amazonaws.com/c
4949
- `_time` — Last modified time of the file. Type: `Nullable(DateTime)`. If the time is unknown, the value is `NULL`.
5050
- `_etag` — The etag of the file. Type: `LowCardinality(String)`. If the etag is unknown, the value is `NULL`.
5151

52+
## Altinity Antalya branch
53+
54+
### `icebergLocalCluster` table function
55+
56+
Only in Altinity Antalya branch `icebergLocalCluster` is a function to make distributed cluster request when iceberg data is storend on shared network storage, mounted with local path.
57+
Path must be the same on all replicas.
58+
59+
```sql
60+
icebergLocalCluster(cluster_name, path_to_table, [,format] [,compression_method])
61+
```
62+
63+
### Specify storage type in arguments
64+
65+
Only in Altinity Antalya branch table function `iceberg` can work with all storages.
66+
In this case storage may be specified with named argument `storage_type`.
67+
Possible values are `s3`, `azure`, `hdfs`, `local`:
68+
69+
```sql
70+
icebergCluster(storage_type='s3', cluster_name, url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method])
71+
72+
icebergCluster(storage_type='azure', cluster_name, connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method])
73+
74+
icebergCluster(storage_type='hdfs', cluster_name, path_to_table, [,format] [,compression_method])
75+
76+
icebergCluster(storage_type='local', cluster_name, path_to_table, [,format] [,compression_method])
77+
```
78+
79+
### Specify storage type in named collection
80+
81+
Only in Altinity Antalya branch `storage_type` may be part of named collection.
82+
83+
```xml
84+
<clickhouse>
85+
<named_collections>
86+
<iceberg_conf>
87+
<url>http://test.s3.amazonaws.com/clickhouse-bucket/</url>
88+
<access_key_id>test<access_key_id>
89+
<secret_access_key>test</secret_access_key>
90+
<format>auto</format>
91+
<structure>auto</structure>
92+
<storage_type>s3</storage_type>
93+
</iceberg_conf>
94+
</named_collections>
95+
</clickhouse>
96+
```
97+
98+
```sql
99+
icebergCluster(iceberg_conf[, option=value [,..]])
100+
```
101+
102+
Default value for `storage_type` is `s3`.
103+
104+
### `object_storage_cluster` setting.
105+
106+
Only in Altinity Antalya branch alternative syntax for `icebergCluster` table function is `iceberg` function with non-empty setting `object_storage_cluster` with cluster name.
107+
108+
```sql
109+
icebergS3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
110+
111+
icebergAzure(connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
112+
113+
icebergHDSF(path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
114+
115+
icebergLocal(path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
116+
117+
icebergS3(option=value [,..]]) SETTINGS object_storage_cluster='cluster_name'
118+
119+
iceberg(storage_type='s3', url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
120+
121+
iceberg(storage_type='azure', connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
122+
123+
iceberg(storage_type='hdfs', path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
124+
125+
iceberg(storage_type='local', path_to_table, [,format] [,compression_method]) SETTINGS object_storage_cluster='cluster_name'
126+
127+
iceberg(iceberg_conf[, option=value [,..]]) SETTINGS object_storage_cluster='cluster_name'
128+
```
129+
52130
**See Also**
53131

54132
- [Iceberg engine](/engines/table-engines/integrations/iceberg.md)

‎docs/en/sql-reference/table-functions/s3Cluster.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,23 @@ Users can use the same approaches as document for the s3 function [here](/sql-re
9090

9191
For details on optimizing the performance of the s3 function see [our detailed guide](/integrations/s3/performance).
9292

93+
## Altinity Antalya branch
94+
95+
### `object_storage_cluster` setting.
96+
97+
Only in Altinity Antalya branch alternative syntax for `s3Cluster` table function is `s3` function with non-empty setting `object_storage_cluster` with cluster name.
98+
99+
```sql
100+
SELECT * FROM s3(
101+
'http://minio1:9001/root/data/{clickhouse,database}/*',
102+
'minio',
103+
'ClickHouse_Minio_P@ssw0rd',
104+
'CSV',
105+
'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'
106+
) ORDER BY (name, value, polygon)
107+
SETTINGS object_storage_cluster='cluster_simple'
108+
```
109+
93110
## Related {#related}
94111

95112
- [S3 engine](../../engines/table-engines/integrations/s3.md)

0 commit comments

Comments
 (0)