Skip to content

Commit 22e4c68

Browse files
poojanilangekareric-maynard
poojanilangekar
and
eric-maynard
authored
Add a regression test for Catalog Federation (#2286)
* Add a regression test for Catalog Federation * Install jq dependency * Fix token issues * Update regtests/README.md Co-authored-by: Eric Maynard <[email protected]> * Update README.md --------- Co-authored-by: Eric Maynard <[email protected]>
1 parent efc68e8 commit 22e4c68

File tree

5 files changed

+269
-2
lines changed

5 files changed

+269
-2
lines changed

‎regtests/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ ENV LANGUAGE='en_US:en'
2525

2626
USER root
2727
RUN apt update
28-
RUN apt-get install -y diffutils wget curl python3.10-venv
28+
RUN apt-get install -y diffutils wget curl python3.10-venv jq
2929
RUN mkdir -p /home/spark && \
3030
chown -R spark /home/spark && \
3131
mkdir -p /tmp/polaris-regtests && \

‎regtests/README.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,14 @@ project, just run:
8585
env POLARIS_HOST=localhost ./regtests/run.sh
8686
```
8787

88+
The catalog federation tests rely on the following configurations in `application.properties` to
89+
be set in order to succeed.
90+
91+
```
92+
polaris.features."ENABLE_CATALOG_FEDERATION"=true
93+
polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"=true
94+
```
95+
8896
To run the tests in verbose mode, with test stdout printing to console, set the `VERBOSE`
8997
environment variable to `1`; you can also choose to run only a subset of tests by specifying the
9098
test directories as arguments to `run.sh`. For example, to run only the `t_spark_sql` tests in
@@ -208,4 +216,4 @@ and download all of the test dependencies into it. From here, `run.sh` will be a
208216
To debug, setup IntelliJ to point at your virtual environment to find your test dependencies
209217
(see https://www.jetbrains.com/help/idea/configuring-python-sdk.html). Then run the test in your IDE.
210218

211-
The above is handled automatically when running reg tests from the docker image.
219+
The above is handled automatically when running reg tests from the docker image.

‎regtests/docker-compose.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ services:
3737
polaris.features."DROP_WITH_PURGE_ENABLED": "true"
3838
polaris.features."ALLOW_INSECURE_STORAGE_TYPES": "true"
3939
polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": "[\"FILE\",\"S3\",\"GCS\",\"AZURE\"]"
40+
polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS": "true"
41+
polaris.features."ENABLE_CATALOG_FEDERATION": "true"
4042
polaris.readiness.ignore-severe-issues: "true"
4143
volumes:
4244
- ./credentials:/tmp/credentials/

‎regtests/t_catalog_federation/ref/catalog_federation.sh.ref‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
=== Setting up Catalog Federation Test ===
2+
Creating new principal...
3+
Creating local catalog...
4+
Create local catalog response code: 201
5+
Setting up permissions...
6+
Grant TABLE_WRITE_DATA to catalog_admin response code: 201
7+
Assign catalog_admin to service_admin response code: 201
8+
Assign service_admin to new-user response code: 201
9+
Creating external catalog (passthrough facade)...
10+
Create external catalog response code: 201
11+
Setting up permissions for external catalog...
12+
Grant TABLE_WRITE_DATA to external catalog_admin response code: 201
13+
Assign catalog_admin to service_admin for external catalog response code: 201
14+
Catalogs created successfully
15+
16+
=== Starting federation test ===
17+
=== Creating data via LOCAL catalog ===
18+
spark-sql ()> use polaris;
19+
spark-sql ()> create namespace if not exists ns1;
20+
spark-sql ()> create table if not exists ns1.test_table (id int, name string);
21+
spark-sql ()> insert into ns1.test_table values (1, 'Alice');
22+
spark-sql ()> insert into ns1.test_table values (2, 'Bob');
23+
spark-sql ()> create namespace if not exists ns2;
24+
spark-sql ()> create table if not exists ns2.test_table (id int, name string);
25+
spark-sql ()> insert into ns2.test_table values (1, 'Apache Spark');
26+
spark-sql ()> insert into ns2.test_table values (2, 'Apache Iceberg');
27+
spark-sql ()>
28+
=== Accessing data via EXTERNAL catalog ===
29+
spark-sql ()> use polaris;
30+
spark-sql ()> show namespaces;
31+
ns1
32+
ns2
33+
spark-sql ()> select * from ns1.test_table order by id;
34+
1 Alice
35+
2 Bob
36+
spark-sql ()> insert into ns1.test_table values (3, 'Charlie');
37+
spark-sql ()> select * from ns2.test_table order by id;
38+
1 Apache Spark
39+
2 Apache Iceberg
40+
spark-sql ()> insert into ns2.test_table values (3, 'Apache Polaris');
41+
spark-sql ()>
42+
=== Verifying federation via LOCAL catalog ===
43+
spark-sql ()> use polaris;
44+
spark-sql ()> select * from ns1.test_table order by id;
45+
1 Alice
46+
2 Bob
47+
3 Charlie
48+
spark-sql ()> select * from ns2.test_table order by id;
49+
1 Apache Spark
50+
2 Apache Iceberg
51+
3 Apache Polaris
52+
spark-sql ()> drop table ns1.test_table;
53+
spark-sql ()> drop table ns2.test_table;
54+
spark-sql ()> drop namespace ns1;
55+
spark-sql ()> drop namespace ns2;
56+
spark-sql ()>
57+
=== Cleaning up catalogs and principal ===
58+
Delete external catalog response code: 204
59+
Delete local catalog response code: 204
60+
Delete principal response code: 204
61+
Catalog federation test completed successfully!

‎regtests/t_catalog_federation/src/catalog_federation.sh‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
# This test creates an INTERNAL catalog and an EXTERNAL catalog with passthrough facade
22+
# to demonstrate true catalog federation.
23+
24+
set -e
25+
26+
27+
SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN}"
28+
29+
echo "=== Setting up Catalog Federation Test ==="
30+
31+
# Step 1: Create a new principal
32+
echo "Creating new principal..."
33+
PRINCIPAL_RESPONSE=$(curl -s -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Content-Type: application/json' \
34+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/principals \
35+
-d '{
36+
"principal": {
37+
"name": "new-user"
38+
}
39+
}')
40+
41+
NEW_CLIENT_ID=$(echo "$PRINCIPAL_RESPONSE" | jq -r '.credentials.clientId')
42+
NEW_CLIENT_SECRET=$(echo "$PRINCIPAL_RESPONSE" | jq -r '.credentials.clientSecret')
43+
44+
# Step 2: Create local catalog
45+
echo "Creating local catalog..."
46+
RESPONSE_CODE=$(curl -s -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Content-Type: application/json' \
47+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs \
48+
-d '{
49+
"type": "INTERNAL",
50+
"name": "test-catalog-local",
51+
"properties": {
52+
"default-base-location": "file:///tmp/warehouse"
53+
},
54+
"storageConfigInfo": {
55+
"storageType": "FILE",
56+
"allowedLocations": ["file:///tmp/warehouse"]
57+
}
58+
}' \
59+
--write-out "%{http_code}")
60+
echo "Create local catalog response code: $RESPONSE_CODE"
61+
62+
63+
64+
# Step 3: Grant permissions
65+
echo "Setting up permissions..."
66+
67+
# Grant TABLE_WRITE_DATA privilege to catalog_admin for local catalog
68+
RESPONSE_CODE=$(curl -s -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
69+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs/test-catalog-local/catalog-roles/catalog_admin/grants \
70+
-d '{"type": "catalog", "privilege": "TABLE_WRITE_DATA"}' \
71+
--write-out "%{http_code}")
72+
echo "Grant TABLE_WRITE_DATA to catalog_admin response code: $RESPONSE_CODE"
73+
74+
# Assign catalog_admin to service_admin
75+
RESPONSE_CODE=$(curl -s -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
76+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/principal-roles/service_admin/catalog-roles/test-catalog-local \
77+
-d '{"name": "catalog_admin"}' \
78+
--write-out "%{http_code}")
79+
echo "Assign catalog_admin to service_admin response code: $RESPONSE_CODE"
80+
81+
# Assign service_admin to new-user
82+
RESPONSE_CODE=$(curl -s -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
83+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/principals/new-user/principal-roles \
84+
-d '{"name": "service_admin"}' \
85+
--write-out "%{http_code}")
86+
echo "Assign service_admin to new-user response code: $RESPONSE_CODE"
87+
88+
# Step 4: Create external catalog
89+
echo "Creating external catalog (passthrough facade)..."
90+
RESPONSE_CODE=$(curl -s -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Content-Type: application/json' \
91+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs \
92+
-d "{
93+
\"type\": \"EXTERNAL\",
94+
\"name\": \"test-catalog-external\",
95+
\"connectionConfigInfo\": {
96+
\"connectionType\": \"ICEBERG_REST\",
97+
\"uri\": \"http://${POLARIS_HOST:-localhost}:8181/api/catalog\",
98+
\"remoteCatalogName\": \"test-catalog-local\",
99+
\"authenticationParameters\": {
100+
\"authenticationType\": \"OAUTH\",
101+
\"tokenUri\": \"http://${POLARIS_HOST:-localhost}:8181/api/catalog/v1/oauth/tokens\",
102+
\"clientId\": \"${NEW_CLIENT_ID}\",
103+
\"clientSecret\": \"${NEW_CLIENT_SECRET}\",
104+
\"scopes\": [\"PRINCIPAL_ROLE:ALL\"]
105+
}
106+
},
107+
\"properties\": {
108+
\"default-base-location\": \"file:///tmp/warehouse\"
109+
},
110+
\"storageConfigInfo\": {
111+
\"storageType\": \"FILE\",
112+
\"allowedLocations\": [\"file:///tmp/warehouse\"]
113+
}
114+
}" \
115+
--write-out "%{http_code}")
116+
echo "Create external catalog response code: $RESPONSE_CODE"
117+
118+
# Step 5: Grant permissions for external catalog
119+
echo "Setting up permissions for external catalog..."
120+
121+
# Grant TABLE_WRITE_DATA privilege to catalog_admin role for test-catalog-external
122+
RESPONSE_CODE=$(curl -s -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
123+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs/test-catalog-external/catalog-roles/catalog_admin/grants \
124+
-d '{"type": "catalog", "privilege": "TABLE_WRITE_DATA"}' \
125+
--write-out "%{http_code}")
126+
echo "Grant TABLE_WRITE_DATA to external catalog_admin response code: $RESPONSE_CODE"
127+
128+
# Assign catalog_admin role to service_admin principal-role for test-catalog-external
129+
RESPONSE_CODE=$(curl -s -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
130+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/principal-roles/service_admin/catalog-roles/test-catalog-external \
131+
-d '{"name": "catalog_admin"}' \
132+
--write-out "%{http_code}")
133+
echo "Assign catalog_admin to service_admin for external catalog response code: $RESPONSE_CODE"
134+
135+
echo "Catalogs created successfully"
136+
137+
echo ""
138+
echo "=== Starting federation test ==="
139+
140+
# Test data operations via local catalog
141+
echo "=== Creating data via LOCAL catalog ==="
142+
cat << EOF | ${SPARK_HOME}/bin/spark-sql -S --conf spark.sql.catalog.polaris.token="${SPARK_BEARER_TOKEN}" --conf spark.sql.catalog.polaris.warehouse=test-catalog-local --conf spark.sql.defaultCatalog=polaris --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
143+
use polaris;
144+
create namespace if not exists ns1;
145+
create table if not exists ns1.test_table (id int, name string);
146+
insert into ns1.test_table values (1, 'Alice');
147+
insert into ns1.test_table values (2, 'Bob');
148+
create namespace if not exists ns2;
149+
create table if not exists ns2.test_table (id int, name string);
150+
insert into ns2.test_table values (1, 'Apache Spark');
151+
insert into ns2.test_table values (2, 'Apache Iceberg');
152+
EOF
153+
154+
echo ""
155+
echo "=== Accessing data via EXTERNAL catalog ==="
156+
cat << EOF | ${SPARK_HOME}/bin/spark-sql -S --conf spark.sql.catalog.polaris.token="${SPARK_BEARER_TOKEN}" --conf spark.sql.catalog.polaris.warehouse=test-catalog-external --conf spark.sql.defaultCatalog=polaris --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
157+
use polaris;
158+
show namespaces;
159+
select * from ns1.test_table order by id;
160+
insert into ns1.test_table values (3, 'Charlie');
161+
select * from ns2.test_table order by id;
162+
insert into ns2.test_table values (3, 'Apache Polaris');
163+
EOF
164+
165+
echo ""
166+
echo "=== Verifying federation via LOCAL catalog ==="
167+
cat << EOF | ${SPARK_HOME}/bin/spark-sql -S --conf spark.sql.catalog.polaris.token="${SPARK_BEARER_TOKEN}" --conf spark.sql.catalog.polaris.warehouse=test-catalog-local --conf spark.sql.defaultCatalog=polaris --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
168+
use polaris;
169+
select * from ns1.test_table order by id;
170+
select * from ns2.test_table order by id;
171+
drop table ns1.test_table;
172+
drop table ns2.test_table;
173+
drop namespace ns1;
174+
drop namespace ns2;
175+
EOF
176+
177+
echo ""
178+
echo "=== Cleaning up catalogs and principal ==="
179+
# Clean up catalogs
180+
RESPONSE_CODE=$(curl -X DELETE -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
181+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs/test-catalog-external \
182+
--write-out "%{http_code}")
183+
echo "Delete external catalog response code: $RESPONSE_CODE"
184+
185+
RESPONSE_CODE=$(curl -X DELETE -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
186+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs/test-catalog-local \
187+
--write-out "%{http_code}")
188+
echo "Delete local catalog response code: $RESPONSE_CODE"
189+
190+
# Clean up principal
191+
RESPONSE_CODE=$(curl -X DELETE -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" -H 'Accept: application/json' -H 'Content-Type: application/json' \
192+
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/principals/new-user \
193+
--write-out "%{http_code}")
194+
echo "Delete principal response code: $RESPONSE_CODE"
195+
196+
echo "Catalog federation test completed successfully!"

0 commit comments

Comments
 (0)