Skip to content

Commit da47a4a

Browse files
authored
Run Catalog integration tests against REST Catalog impls (#2482)
<!-- Thanks for opening a pull request! --> <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. --> Closes #2481 # Rationale for this change This allows users to run our test suite against their REST Catalog implementations. ## Are these changes tested? Test-only change ## Are there any user-facing changes? - Adds support for running catalog tests against REST Catalog implementations. <!-- In the case of user-facing changes, please add the changelog label. -->
1 parent e5e7453 commit da47a4a

File tree

2 files changed

+41
-1
lines changed

2 files changed

+41
-1
lines changed

mkdocs/docs/contributing.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,34 @@ make test-integration-rebuild
155155

156156
To rebuild the containers from scratch.
157157

158+
#### Running Integration Tests against REST Catalogs
159+
160+
!!! warning "Do not run against production catalogs"
161+
The integration tests will delete data throughout the entirety of your catalog. Running these integration tests against production catalogs will result in data loss.
162+
163+
PyIceberg supports the ability to run our catalog tests against an arbitrary REST Catalog.
164+
165+
In order to run the test catalog, you will need to specify which REST catalog to run against with the `PYICEBERG_TEST_CATALOG` environment variable
166+
167+
```sh
168+
export PYICEBERG_TEST_CATALOG=test_catalog
169+
```
170+
171+
The catalog in question can be configured either through the ~/.pyiceberg.yaml file or through environment variables.
172+
173+
```yaml
174+
catalog:
175+
test_catalog:
176+
uri: http://rest-catalog/ws/
177+
credential: t-1234:secret
178+
```
179+
180+
```sh
181+
export PYICEBERG_CATALOG__TEST_CATALOG__URI=thrift://localhost:9083
182+
export PYICEBERG_CATALOG__TEST_CATALOG__ACCESS_KEY_ID=username
183+
export PYICEBERG_CATALOG__TEST_CATALOG__SECRET_ACCESS_KEY=password
184+
```
185+
158186
## Code standards
159187

160188
Below are the formalized conventions that we adhere to in the PyIceberg project. The goal of this is to have a common agreement on how to evolve the codebase, but also using it as guidelines for newcomers to the project.

tests/integration/test_catalog.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,13 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18+
import os
1819
from pathlib import Path, PosixPath
1920
from typing import Generator, List
2021

2122
import pytest
2223

23-
from pyiceberg.catalog import Catalog, MetastoreCatalog
24+
from pyiceberg.catalog import Catalog, MetastoreCatalog, load_catalog
2425
from pyiceberg.catalog.hive import HiveCatalog
2526
from pyiceberg.catalog.memory import InMemoryCatalog
2627
from pyiceberg.catalog.rest import RestCatalog
@@ -74,6 +75,16 @@ def rest_catalog() -> Generator[Catalog, None, None]:
7475
clean_up(test_catalog)
7576

7677

78+
@pytest.fixture(scope="function")
79+
def rest_test_catalog() -> Generator[Catalog, None, None]:
80+
if test_catalog_name := os.environ.get("PYICEBERG_TEST_CATALOG"):
81+
test_catalog = load_catalog(test_catalog_name)
82+
yield test_catalog
83+
clean_up(test_catalog)
84+
else:
85+
pytest.skip("PYICEBERG_TEST_CATALOG environment variables not set")
86+
87+
7788
@pytest.fixture(scope="function")
7889
def hive_catalog() -> Generator[Catalog, None, None]:
7990
test_catalog = HiveCatalog(
@@ -95,6 +106,7 @@ def hive_catalog() -> Generator[Catalog, None, None]:
95106
pytest.lazy_fixture("sqlite_catalog_file"),
96107
pytest.lazy_fixture("rest_catalog"),
97108
pytest.lazy_fixture("hive_catalog"),
109+
pytest.lazy_fixture("rest_test_catalog"),
98110
]
99111

100112

0 commit comments

Comments
 (0)