Skip to content

Commit d9fc2b6

Browse files
pieternpietern
authored
Add tests to ensure API compatibility when using positional arguments (#507)
This test requires the environment variable DATABRICKS_CLI_MAIN_CHECKOUT to be set to the path of a reference checkout of this repository. It will dynamically load the argspecs from both the reference repository and the current repository and compare them. The test fails if the current repository contains incompatible modifications of function signatures. In CI we can run this test with DATABRICKS_CLI_MAIN_CHECKOUT pointing to a checkout of the main branch. It will test compatibility of the public API in the PR against the public API in the main branch, and fail on error.
1 parent e8254cc commit d9fc2b6

File tree

3 files changed

+232
-0
lines changed

3 files changed

+232
-0
lines changed

‎.github/workflows/push.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,3 +122,30 @@ jobs:
122122
databricks --version
123123
which dbfs
124124
dbfs --version
125+
126+
compat:
127+
name: Public API compatibility
128+
runs-on: ubuntu-latest
129+
steps:
130+
- name: Checkout
131+
uses: actions/checkout@v2
132+
133+
- name: Checkout main branch
134+
uses: actions/checkout@v2
135+
with:
136+
ref: main
137+
path: databricks-cli-main
138+
139+
- name: Set up Python
140+
uses: actions/setup-python@v1
141+
142+
- name: Install pip
143+
run: pip install --upgrade pip
144+
145+
- name: Install development dependencies (Python 3)
146+
run: pip install --upgrade -r dev-requirements-py3.txt
147+
148+
- name: Run compatibility test
149+
run: |
150+
export DATABRICKS_CLI_MAIN_CHECKOUT=$GITHUB_WORKSPACE/databricks-cli-main
151+
pytest --capture=no tests/test_compat.py

‎scripts/test_compatibility.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env bash
2+
# Databricks CLI
3+
# Copyright 2017 Databricks, Inc.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License"), except
6+
# that the use of services to which certain application programming
7+
# interfaces (each, an "API") connect requires that the user first obtain
8+
# a license for the use of the APIs from Databricks, Inc. ("Databricks"),
9+
# by creating an account at www.databricks.com and agreeing to either (a)
10+
# the Community Edition Terms of Service, (b) the Databricks Terms of
11+
# Service, or (c) another written agreement between Licensee and Databricks
12+
# for the use of the APIs.
13+
#
14+
# You may not use this file except in compliance with the License.
15+
# You may obtain a copy of the License at
16+
#
17+
# http://www.apache.org/licenses/LICENSE-2.0
18+
#
19+
# Unless required by applicable law or agreed to in writing, software
20+
# distributed under the License is distributed on an "AS IS" BASIS,
21+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
22+
# See the License for the specific language governing permissions and
23+
# limitations under the License.
24+
25+
set -e
26+
27+
# Make sure /tmp holds an up to date checkout of this repository.
28+
databricks_cli_repo=https://github.com/databricks/databricks-cli
29+
databricks_cli_path=/tmp/databricks_cli_main
30+
echo "Updating $databricks_cli_repo in $databricks_cli_path..."
31+
if [ ! -d $databricks_cli_path ]; then
32+
git clone $databricks_cli_repo $databricks_cli_path
33+
else
34+
(
35+
cd $databricks_cli_path
36+
git pull --ff origin
37+
)
38+
fi
39+
40+
# Run compatibility test given the reference checkout.
41+
export DATABRICKS_CLI_MAIN_CHECKOUT=$databricks_cli_path
42+
exec python3 -m pytest tests/test_compat.py

‎tests/test_compat.py‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Databricks CLI
2+
# Copyright 2017 Databricks, Inc.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"), except
5+
# that the use of services to which certain application programming
6+
# interfaces (each, an "API") connect requires that the user first obtain
7+
# a license for the use of the APIs from Databricks, Inc. ("Databricks"),
8+
# by creating an account at www.databricks.com and agreeing to either (a)
9+
# the Community Edition Terms of Service, (b) the Databricks Terms of
10+
# Service, or (c) another written agreement between Licensee and Databricks
11+
# for the use of the APIs.
12+
#
13+
# You may not use this file except in compliance with the License.
14+
# You may obtain a copy of the License at
15+
#
16+
# http://www.apache.org/licenses/LICENSE-2.0
17+
#
18+
# Unless required by applicable law or agreed to in writing, software
19+
# distributed under the License is distributed on an "AS IS" BASIS,
20+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
21+
# See the License for the specific language governing permissions and
22+
# limitations under the License.
23+
24+
import importlib
25+
import inspect
26+
import json
27+
import os
28+
import re
29+
import sys
30+
import tempfile
31+
import unittest
32+
33+
import six
34+
35+
36+
def normalize_module_name(name):
37+
"""
38+
Replace `databricks_cli_main` with `databricks_cli` to make objects comparable.
39+
"""
40+
return re.sub(r"^(databricks_cli)_main", r"\1", name)
41+
42+
43+
def func_key(func):
44+
"""
45+
Returns key of the full package path to the specified function.
46+
"""
47+
return normalize_module_name(func.__module__) + "." + func.__qualname__
48+
49+
50+
def collect_argspecs(modules):
51+
return {
52+
func_key(func): inspect.getfullargspec(func).args
53+
for module in modules
54+
for (_, clazz) in inspect.getmembers(module, predicate=inspect.isclass)
55+
for (_, func) in inspect.getmembers(clazz, predicate=inspect.isfunction)
56+
# Ignore functions that are defined outside the specified module.
57+
if module.__name__ == func.__module__
58+
}
59+
60+
61+
# pylint: disable=superfluous-parens
62+
def _test_compatibility(current_argspecs, existing_argspecs):
63+
removed_functions = []
64+
incompatible_functions = []
65+
for existing_name, existing_args in existing_argspecs.items():
66+
if existing_name not in current_argspecs:
67+
removed_functions.append(existing_name)
68+
continue
69+
70+
# New arguments may only be appended to ensure compatibility
71+
# when functions are called with positional parameters.
72+
current_args = current_argspecs[existing_name]
73+
if (
74+
len(current_args) < len(existing_args)
75+
or current_args[0 : len(existing_args)] != existing_args
76+
):
77+
incompatible_functions.append(existing_name)
78+
continue
79+
80+
if removed_functions:
81+
print("Removed functions:")
82+
for fn in removed_functions:
83+
print(" * " + fn)
84+
assert len(removed_functions) == 0
85+
86+
if incompatible_functions:
87+
print("Incompatible functions:")
88+
for fn in incompatible_functions:
89+
current_args = current_argspecs[fn]
90+
existing_args = existing_argspecs[fn]
91+
print(" * " + fn + ": " + str(existing_args) + " -> " + str(current_args))
92+
assert len(incompatible_functions) == 0
93+
94+
95+
def import_databricks_modules(root):
96+
return [
97+
# Generated code.
98+
importlib.import_module(".sdk.service", root),
99+
# Functionality under the API package is used as an SDK by some.
100+
importlib.import_module(".cluster_policies.api", root),
101+
importlib.import_module(".clusters.api", root),
102+
importlib.import_module(".dbfs.api", root),
103+
importlib.import_module(".groups.api", root),
104+
importlib.import_module(".instance_pools.api", root),
105+
importlib.import_module(".jobs.api", root),
106+
importlib.import_module(".libraries.api", root),
107+
importlib.import_module(".pipelines.api", root),
108+
importlib.import_module(".repos.api", root),
109+
importlib.import_module(".runs.api", root),
110+
importlib.import_module(".secrets.api", root),
111+
importlib.import_module(".tokens.api", root),
112+
importlib.import_module(".unity_catalog.api", root),
113+
importlib.import_module(".workspace.api", root),
114+
]
115+
116+
117+
def databricks_cli_main_checkout_path():
118+
return os.getenv('DATABRICKS_CLI_MAIN_CHECKOUT')
119+
120+
121+
def can_perform_compatibility_check():
122+
"""
123+
Checks if the user has specified a path to a parallel checkout
124+
of the main branch of the databricks-cli repository.
125+
126+
The files in this checkout will be imported under an aliased
127+
package name such that we have access to the argument specs
128+
of the functions in both the main branch and this repository.
129+
"""
130+
path = databricks_cli_main_checkout_path()
131+
return path and os.path.isdir(path)
132+
133+
134+
@unittest.skipIf(six.PY2 or not can_perform_compatibility_check(), reason=None)
135+
def test_compatibility():
136+
"""
137+
To run this test, checkout a reference copy of this repository and set
138+
the DATABRICKS_CLI_MAIN_CHECKOUT environment variable to its path.
139+
140+
See `.github/workflows/` for the configuration of the GitHub action that runs it.
141+
"""
142+
143+
# Make the specified path importable by symlinking its `databricks_cli`
144+
# directory
145+
tmpdir = tempfile.mkdtemp()
146+
sys.path.append(tmpdir)
147+
os.symlink(
148+
os.path.join(databricks_cli_main_checkout_path(), "databricks_cli"),
149+
os.path.join(tmpdir, "databricks_cli_main"),
150+
)
151+
152+
current_argspecs = collect_argspecs(import_databricks_modules("databricks_cli"))
153+
existing_argspecs = collect_argspecs(import_databricks_modules("databricks_cli_main"))
154+
_test_compatibility(current_argspecs, existing_argspecs)
155+
156+
157+
if __name__ == '__main__':
158+
# If run directly, write the argspecs in this repository to stdout.
159+
json.dump(
160+
obj=collect_argspecs(import_databricks_modules("databricks_cli")),
161+
fp=sys.stdout,
162+
indent=2,
163+
)

0 commit comments

Comments
 (0)