Update databricks-sdk requirement from <0.42,>=0.40 to >=0.44,<0.45 (#3686)

Updates the requirements on
[databricks-sdk](https://github.com/databricks/databricks-sdk-py) to
permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-py/releases">databricks-sdk's
releases</a>.</em></p>
<blockquote>
<h2>v0.44.0</h2>
<h2>[Release] Release v0.44.0</h2>
<h3>Internal Changes</h3>
<ul>
<li>Fix <code>tests/integration/test_dbutils.py::test_secrets</code> (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/884">#884</a>).</li>
</ul>
<h3>API Changes:</h3>
<ul>
<li>Added <code>get_message_query_result_by_attachment()</code> method
for <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/genie.html">w.genie</a>
workspace-level service.</li>
<li>Added <code>id</code> field for
<code>databricks.sdk.service.apps.App</code>.</li>
<li>Added <code>limit_config</code> field for
<code>databricks.sdk.service.billing.UpdateBudgetPolicyRequest</code>.</li>
<li>Added <code>volumes</code> field for
<code>databricks.sdk.service.compute.ClusterLogConf</code>.</li>
<li>Removed <code>review_state</code>, <code>reviews</code> and
<code>runner_collaborators</code> fields for
<code>databricks.sdk.service.cleanrooms.CleanRoomAssetNotebook</code>.</li>
</ul>
<p>OpenAPI SHA: 99f644e72261ef5ecf8d74db20f4b7a1e09723cc, Date:
2025-02-11</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-py/blob/main/CHANGELOG.md">databricks-sdk's
changelog</a>.</em></p>
<blockquote>
<h2>[Release] Release v0.44.0</h2>
<h3>Internal Changes</h3>
<ul>
<li>Fix <code>tests/integration/test_dbutils.py::test_secrets</code> (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/884">#884</a>).</li>
</ul>
<h3>API Changes:</h3>
<ul>
<li>Added <code>get_message_query_result_by_attachment()</code> method
for <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/genie.html">w.genie</a>
workspace-level service.</li>
<li>Added <code>id</code> field for
<code>databricks.sdk.service.apps.App</code>.</li>
<li>Added <code>limit_config</code> field for
<code>databricks.sdk.service.billing.UpdateBudgetPolicyRequest</code>.</li>
<li>Added <code>volumes</code> field for
<code>databricks.sdk.service.compute.ClusterLogConf</code>.</li>
<li>Removed <code>review_state</code>, <code>reviews</code> and
<code>runner_collaborators</code> fields for
<code>databricks.sdk.service.cleanrooms.CleanRoomAssetNotebook</code>.</li>
</ul>
<p>OpenAPI SHA: 99f644e72261ef5ecf8d74db20f4b7a1e09723cc, Date:
2025-02-11</p>
<h2>[Release] Release v0.43.0</h2>
<h3>API Changes:</h3>
<ul>
<li>Added <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/lakeview_embedded.html">w.lakeview_embedded</a>
workspace-level service and <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/query_execution.html">w.query_execution</a>
workspace-level service.</li>
<li>Added <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/redash_config.html">w.redash_config</a>
workspace-level service.</li>
<li>Added <code>gcp_oauth_token</code> field for
<code>databricks.sdk.service.catalog.TemporaryCredentials</code>.</li>
<li>Added <code>options</code> field for
<code>databricks.sdk.service.catalog.UpdateCatalog</code>.</li>
<li>Added <code>disabled</code> field for
<code>databricks.sdk.service.jobs.RunTask</code>.</li>
</ul>
<p>OpenAPI SHA: c72c58f97b950fcb924a90ef164bcb10cfcd5ece, Date:
2025-02-03</p>
<h3>Bug Fixes</h3>
<ul>
<li>Fix docs generation when two services have the same name (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/872">#872</a>).</li>
</ul>
<h3>Internal Changes</h3>
<ul>
<li>Add CICD environment to the User Agent (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/866">#866</a>).</li>
<li>Add unit tests for retriable requests (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/879">#879</a>).</li>
<li>Extract &quot;before retry&quot; handler, use it to rewind the
stream (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/878">#878</a>).</li>
<li>Update Model Serving <code>http_request</code> mixin to correctly
use the underlying API. (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/pull/876">#876</a>).</li>
</ul>
<h3>Backward Incompatible Changes</h3>
<ul>
<li>Changed <code>create()</code> method for <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html">w.serving_endpoints</a>
workspace-level service with new required argument order.</li>
<li>Changed <code>http_request()</code> method for <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html">w.serving_endpoints</a>
workspace-level service to type <code>http_request()</code> method for
<a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html">w.serving_endpoints</a>
workspace-level service.</li>
<li>Changed <code>http_request()</code> method for <a
href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html">w.serving_endpoints</a>
workspace-level service to return
<code>databricks.sdk.service.serving.HttpRequestResponse</code>
dataclass.</li>
<li>Changed <code>config</code> field for
<code>databricks.sdk.service.serving.CreateServingEndpoint</code> to no
longer be required.</li>
<li>Removed <code>securable_kind</code> field for
<code>databricks.sdk.service.catalog.CatalogInfo</code>.</li>
<li>Removed <code>securable_kind</code> field for
<code>databricks.sdk.service.catalog.ConnectionInfo</code>.</li>
<li>Removed <code>status_code</code> and <code>text</code> fields for
<code>databricks.sdk.service.serving.ExternalFunctionResponse</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/3c391a03d14bffa9a715bd949eed58adafded9e3"><code>3c391a0</code></a>
[Release] Release v0.44.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/885">#885</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/3ab8c3bffc5ef086c8fb8d5fd2923c0020c51f36"><code>3ab8c3b</code></a>
[Internal] Fix
<code>tests/integration/test_dbutils.py::test_secrets</code> (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/884">#884</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/b64ef183e4840a0ead31a7a5a9f10c505a91e66e"><code>b64ef18</code></a>
[Release] Release v0.43.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/883">#883</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/6f094aafffc13fbe5c236e6a7735b1b239b1ab78"><code>6f094aa</code></a>
[Release] Release v0.42.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/880">#880</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/aa4f0f3fb01f208f5ebdcd6570bcb1e728c10aec"><code>aa4f0f3</code></a>
[Fix] Exclude localhost from hitting proxies for metadata service
requests (#...</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/95277c8625cff51099eca4a8ca129c4865b9776e"><code>95277c8</code></a>
[Internal] Add unit tests for retriable requests (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/879">#879</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/533939668b50f66f02a5cc345aabbc086c153e9e"><code>5339396</code></a>
[Internal] Update Model Serving <code>http_request</code> mixin to
correctly use the und...</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/762c57b9bfa14cc30bf5538007f116b676a50172"><code>762c57b</code></a>
[Internal] Extract &quot;before retry&quot; handler, use it to rewind
the stream (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/878">#878</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/4bcfb0ad3c817f6e635bb930ff433f56ef5cbd6f"><code>4bcfb0a</code></a>
[Fix] Fix docs generation when two services have the same name (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/872">#872</a>)</li>
<li><a
href="https://github.com/databricks/databricks-sdk-py/commit/5576d32844cc04c7aeb332a56c40216883df066b"><code>5576d32</code></a>
[Internal] Add CICD environment to the User Agent (<a
href="https://redirect.github.com/databricks/databricks-sdk-py/issues/866">#866</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/databricks/databricks-sdk-py/compare/v0.40.0...v0.44.0">compare
view</a></li>
</ul>
</details>
<br />

<details>
<summary>Most Recent Ignore Conditions Applied to This Pull
Request</summary>

| Dependency Name | Ignore Conditions |
| --- | --- |
| databricks-sdk | [>= 0.25.a, < 0.26] |
</details>


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Cor Zuurmond <[email protected]>
Co-authored-by: Pritish Pai <[email protected]>
Co-authored-by: Guenia Izquierdo Delgado <[email protected]>

Author: 4 people

pyproject.toml

@@ -44,7 +44,8 @@ classifiers = [
     "Topic :: Utilities",
 ]
 
-dependencies = ["databricks-sdk>=0.40,<0.42",
+
+dependencies = ["databricks-sdk>=0.44.0,<0.45.0",
                 "databricks-labs-lsql>=0.16.0,<0.17.0",
                 "databricks-labs-blueprint>=0.10.0,<0.11.0",
                 "PyYAML>=6.0.0,<6.1.0",

src/databricks/labs/ucx/hive_metastore/table_migration_status.py

@@ -7,7 +7,7 @@
 from databricks.labs.lsql.backends import SqlBackend
 from databricks.sdk import WorkspaceClient
 from databricks.sdk.errors import DatabricksError, NotFound
-from databricks.sdk.service.catalog import CatalogInfo, CatalogInfoSecurableKind, SchemaInfo
+from databricks.sdk.service.catalog import CatalogInfo, CatalogType, SchemaInfo
 
 from databricks.labs.ucx.framework.crawlers import CrawlerBase
 from databricks.labs.ucx.framework.utils import escape_sql_identifier
@@ -79,10 +79,9 @@ class TableMigrationStatusRefresher(CrawlerBase[TableMigrationStatus]):
     properties for the presence of the marker.
     """
 
-    _skip_catalog_securable_kinds = [
-        CatalogInfoSecurableKind.CATALOG_INTERNAL,
-        CatalogInfoSecurableKind.CATALOG_SYSTEM,
-    ]
+    _skip_catalog_types = {
+        CatalogType.SYSTEM_CATALOG,
+    }
 
     def __init__(self, ws: WorkspaceClient, sql_backend: SqlBackend, schema, tables_crawler: TablesCrawler):
         super().__init__(sql_backend, "hive_metastore", schema, "migration_status", TableMigrationStatus)
@@ -163,7 +162,7 @@ def _try_fetch(self) -> Iterable[TableMigrationStatus]:
     def _iter_catalogs(self) -> Iterable[CatalogInfo]:
         try:
             for catalog in self._ws.catalogs.list():
-                if catalog.securable_kind in self._skip_catalog_securable_kinds:
+                if catalog.catalog_type in self._skip_catalog_types:
                     continue
                 yield catalog
         except DatabricksError as e:

tests/unit/hive_metastore/test_table_migration_status.py

@@ -4,7 +4,7 @@
 import pytest
 from databricks.sdk import WorkspaceClient
 from databricks.sdk.errors import BadRequest, DatabricksError, NotFound
-from databricks.sdk.service.catalog import CatalogInfoSecurableKind, CatalogInfo, SchemaInfo, TableInfo
+from databricks.sdk.service.catalog import CatalogInfo, CatalogType, SchemaInfo, TableInfo
 
 from databricks.labs.ucx.hive_metastore.tables import TablesCrawler
 from databricks.labs.ucx.hive_metastore.table_migration_status import TableMigrationStatusRefresher
@@ -67,20 +67,12 @@ def test_table_migration_status_refresher_get_seen_tables_handles_errors_on_tabl
     tables_crawler.snapshot.assert_not_called()
 
 
-@pytest.mark.parametrize(
-    "securable_kind",
-    [
-        CatalogInfoSecurableKind.CATALOG_INTERNAL,
-        CatalogInfoSecurableKind.CATALOG_SYSTEM,
-    ],
-)
-def test_table_migration_status_refresher_get_seen_tables_skips_builtin_catalog(
-    mock_backend, securable_kind: CatalogInfoSecurableKind
-) -> None:
+def test_table_migration_status_refresher_get_seen_tables_skips_system_catalog(mock_backend) -> None:
+    """We can skip the system catalog when refreshing the table migration status."""
     ws = create_autospec(WorkspaceClient)
     ws.catalogs.list.return_value = [
         CatalogInfo(name="test"),
-        CatalogInfo(name="system", securable_kind=securable_kind),
+        CatalogInfo(name="system", catalog_type=CatalogType.SYSTEM_CATALOG),
     ]
 
     def schemas_list(catalog_name: str) -> Iterable[SchemaInfo]: