Release v0.35.0 (#2543)

* Added `databricks labs ucx delete-credential` cmd to delete the UC
roles created by UCX
([#2504](#2504)). In this
release, we've added several new commands to the `labs.yml` file for
managing Unity Catalog (UC) roles in Databricks, specifically for AWS.
The new commands include `databricks labs ucx delete-missing-principals`
and `databricks labs ucx delete-credential`. The `databricks labs ucx
delete-missing-principals` command helps manage UC roles created through
the `create-missing-principals` cmd by listing all the UC roles in AWS
and allowing users to select roles to delete. It also checks for unused
roles before deletion. The `databricks labs ucx delete-credential`
command deletes UC roles created by UCX and is equipped with an optional
`aws-profile` flag for authentication purposes. Additionally, we've
added a new method `delete_uc_role` in the `access.py` file for deleting
UC roles and introduced new test cases to ensure correct behavior. These
changes resolve issue
[#2359](#2359), improving
the overall management of UC roles in AWS.
* Added basic documentation for linter message codes
([#2536](#2536)). A new
section, "Linter Message Codes," has been added to the README file,
providing detailed explanations, examples, and resolution instructions
for various linter message codes related to Unity Catalog (UC)
migration. To help users familiarize themselves with the different
message codes that may appear during linting, a new command, `python
tests/integration/source_code/message_codes.py`, has been implemented.
Running this command will display a list of message codes, including
`cannot-autofix-table-reference`, `catalog-api-in-shared-clusters`,
`changed-result-format-in-uc`, `dbfs-read-from-sql-query`, `dbfs-usage`,
`dependency-not-found`, `direct-filesystem-access`,
`implicit-dbfs-usage`, `jvm-access-in-shared-clusters`,
`legacy-context-in-shared-clusters`, `not-supported`,
`notebook-run-cannot-compute-value`, `python-udf-in-shared-clusters`,
`rdd-in-shared-clusters`, `spark-logging-in-shared-clusters`,
`sql-parse-error`, `sys-path-cannot-compute-value`,
`table-migrated-to-uc`, `to-json-in-shared-clusters`, and
`unsupported-magic-line`. Users are encouraged to review these message
codes and their corresponding explanations to ensure a smooth migration
to Unity Catalog.
* Added linters for direct filesystem access in Python and SQL code
([#2519](#2519)). In this
release, linters have been added for detecting direct file system access
(DFSA) in Python and SQL code, specifically addressing Direct File
System Access in the Unity Catalog. Initially, the linters only detect
DBFS, but the plan is to expand detection to all DSFSAs. This change is
part of issue [#2519](#2519)
and includes new unit tests. The linters will flag code that accesses
the file system directly, which is not allowed in Unity Catalog,
including SQL queries that read from DBFS and Python code that reads
from or displays data using DBFS paths. Developers are required to
modify such code to use Unity Catalog tables or volumes instead,
ensuring that their code is compatible with Unity Catalog's deprecation
of direct file system access and DBFS, ultimately resulting in better
project integration and performance.
* Clean up left over uber principal resources for AWS
([#2449](#2449)). In this
release, we have made significant improvements to our open-source
library, particularly in the management of AWS resources and
permissions. We have added new methods to handle the creation and
deletion of Uber instance profiles and external locations in AWS. The
`create_uber_principal` method in the `AWSResourcePermissions` class has
been updated to allow more fine-grained control over the migration
process and ensure proper creation and configuration of all necessary
resources. Additionally, we have introduced a new `AWSResources` class
and `aws_resource_permissions` property in the `aws_cli_ctx` fixture to
improve the management of AWS resources and permissions. We have also
added new unit tests to ensure proper error handling when creating AWS
IAM roles for Unity Catalog Migration (ucx) in specific scenarios. These
changes enhance the functionality, test coverage, and overall quality of
our library.
* Improve log warning about skipped grants
([#2517](#2517)). In this
release, we have implemented improvements to the warning messages
displayed during the verification of Unified Client Context (UCX)
behavior for Access Control List (ACL) migration and User-Defined
Function (UDF) behavior migration. Previously, generic warning messages
were logged when specific Hive metastore grants could not be identified.
This release enhances the warning messages by providing more specific
information about the skipped grants, including the action type of the
Hive metastore grant that failed to be mapped. Additionally,
unittest.mock has been utilized to create a mock object for the
GroupManager class, and a new method called MigrateGrants has been
introduced, which applies a list of grant loaders to a specific table.
These changes improve the logging and error handling, ensuring that
software engineers have a clear understanding of any skipped grants
during UCX and UDF behavior migration.
* Support sql notebooks in functional tests
([#2513](#2513)). This pull
request introduces support for SQL notebooks in the functional testing
framework, expanding its capabilities beyond Python notebooks. The
changes include migrating relevant tests from `test_notebook_linter` to
functional tests, as well as introducing new classes and methods to
support SQL notebook testing. These changes improve the flexibility and
scope of the testing framework, enabling developers to test SQL
notebooks and ensuring that they meet quality standards. The commit also
includes the addition of a new SQL notebook for demonstrating Unity
Catalog table migrations, as well as modifications to various tests and
regular expressions to accommodate SQL notebooks. Note that environment
variables `DATABRICKS_HOST` and `DATABRICKS_TOKEN` are currently
hardcoded as `any`, requiring further updates based on the specific
testing environment.
* Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20
([#2533](#2533)). In this
update, we have updated the `sqlglot` dependency to a version greater
than or equal to 25.5.0 and less than 25.20. The previous requirement
allowed for versions up to 25.19, but we have chosen to update to a
newer version that includes new features and bug fixes. The changelog
and commit history for version 25.19 of `sqlglot` are provided in the
pull request, highlighting the breaking changes, new features, and bug
fixes. By updating to this version, we will benefit from the latest
improvements and bug fixes in the `sqlglot` library. We encourage users
to review the changelog and test their code to ensure compatibility with
the new version.
* [chore] fixed `make fmt` warnings related to sdk upgrade
([#2534](#2534)). In this
change, warnings related to a recent SDK upgrade have been addressed by
modifying the `create` function in the `fixtures.py` file. The `create`
function is responsible for generating a `Wait[ServingEndpointDetailed]`
object, which contains information about the endpoint name and its core
configuration. The `ServedModelInput` instance was previously created
using positional arguments, but has been updated to use named arguments
instead (`model_name`, `model_version`, `workload_size`,
`scale_to_zero_enabled`). This modification enhances code readability
and maintainability, making it easier for software engineers to
understand and modify the codebase.

Dependency updates:

* Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20
([#2533](#2533)).

Author: nfx

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Version changelog
 
+## 0.35.0
+
+* Added `databricks labs ucx delete-credential` cmd to delete the UC roles created by UCX ([#2504](https://github.com/databrickslabs/ucx/issues/2504)). In this release, we've added several new commands to the `labs.yml` file for managing Unity Catalog (UC) roles in Databricks, specifically for AWS. The new commands include `databricks labs ucx delete-missing-principals` and `databricks labs ucx delete-credential`. The `databricks labs ucx delete-missing-principals` command helps manage UC roles created through the `create-missing-principals` cmd by listing all the UC roles in AWS and allowing users to select roles to delete. It also checks for unused roles before deletion. The `databricks labs ucx delete-credential` command deletes UC roles created by UCX and is equipped with an optional `aws-profile` flag for authentication purposes. Additionally, we've added a new method `delete_uc_role` in the `access.py` file for deleting UC roles and introduced new test cases to ensure correct behavior. These changes resolve issue [#2359](https://github.com/databrickslabs/ucx/issues/2359), improving the overall management of UC roles in AWS.
+* Added basic documentation for linter message codes ([#2536](https://github.com/databrickslabs/ucx/issues/2536)). A new section, "Linter Message Codes," has been added to the README file, providing detailed explanations, examples, and resolution instructions for various linter message codes related to Unity Catalog (UC) migration. To help users familiarize themselves with the different message codes that may appear during linting, a new command, `python tests/integration/source_code/message_codes.py`, has been implemented. Running this command will display a list of message codes, including `cannot-autofix-table-reference`, `catalog-api-in-shared-clusters`, `changed-result-format-in-uc`, `dbfs-read-from-sql-query`, `dbfs-usage`, `dependency-not-found`, `direct-filesystem-access`, `implicit-dbfs-usage`, `jvm-access-in-shared-clusters`, `legacy-context-in-shared-clusters`, `not-supported`, `notebook-run-cannot-compute-value`, `python-udf-in-shared-clusters`, `rdd-in-shared-clusters`, `spark-logging-in-shared-clusters`, `sql-parse-error`, `sys-path-cannot-compute-value`, `table-migrated-to-uc`, `to-json-in-shared-clusters`, and `unsupported-magic-line`. Users are encouraged to review these message codes and their corresponding explanations to ensure a smooth migration to Unity Catalog.
+* Added linters for direct filesystem access in Python and SQL code ([#2519](https://github.com/databrickslabs/ucx/issues/2519)). In this release, linters have been added for detecting direct file system access (DFSA) in Python and SQL code, specifically addressing Direct File System Access in the Unity Catalog. Initially, the linters only detect DBFS, but the plan is to expand detection to all DSFSAs. This change is part of issue [#2519](https://github.com/databrickslabs/ucx/issues/2519) and includes new unit tests. The linters will flag code that accesses the file system directly, which is not allowed in Unity Catalog, including SQL queries that read from DBFS and Python code that reads from or displays data using DBFS paths. Developers are required to modify such code to use Unity Catalog tables or volumes instead, ensuring that their code is compatible with Unity Catalog's deprecation of direct file system access and DBFS, ultimately resulting in better project integration and performance.
+* Clean up left over uber principal resources for AWS ([#2449](https://github.com/databrickslabs/ucx/issues/2449)). In this release, we have made significant improvements to our open-source library, particularly in the management of AWS resources and permissions. We have added new methods to handle the creation and deletion of Uber instance profiles and external locations in AWS. The `create_uber_principal` method in the `AWSResourcePermissions` class has been updated to allow more fine-grained control over the migration process and ensure proper creation and configuration of all necessary resources. Additionally, we have introduced a new `AWSResources` class and `aws_resource_permissions` property in the `aws_cli_ctx` fixture to improve the management of AWS resources and permissions. We have also added new unit tests to ensure proper error handling when creating AWS IAM roles for Unity Catalog Migration (ucx) in specific scenarios. These changes enhance the functionality, test coverage, and overall quality of our library.
+* Improve log warning about skipped grants ([#2517](https://github.com/databrickslabs/ucx/issues/2517)). In this release, we have implemented improvements to the warning messages displayed during the verification of Unified Client Context (UCX) behavior for Access Control List (ACL) migration and User-Defined Function (UDF) behavior migration. Previously, generic warning messages were logged when specific Hive metastore grants could not be identified. This release enhances the warning messages by providing more specific information about the skipped grants, including the action type of the Hive metastore grant that failed to be mapped. Additionally, unittest.mock has been utilized to create a mock object for the GroupManager class, and a new method called MigrateGrants has been introduced, which applies a list of grant loaders to a specific table. These changes improve the logging and error handling, ensuring that software engineers have a clear understanding of any skipped grants during UCX and UDF behavior migration.
+* Support sql notebooks in functional tests ([#2513](https://github.com/databrickslabs/ucx/issues/2513)). This pull request introduces support for SQL notebooks in the functional testing framework, expanding its capabilities beyond Python notebooks. The changes include migrating relevant tests from `test_notebook_linter` to functional tests, as well as introducing new classes and methods to support SQL notebook testing. These changes improve the flexibility and scope of the testing framework, enabling developers to test SQL notebooks and ensuring that they meet quality standards. The commit also includes the addition of a new SQL notebook for demonstrating Unity Catalog table migrations, as well as modifications to various tests and regular expressions to accommodate SQL notebooks. Note that environment variables `DATABRICKS_HOST` and `DATABRICKS_TOKEN` are currently hardcoded as `any`, requiring further updates based on the specific testing environment.
+* Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20 ([#2533](https://github.com/databrickslabs/ucx/issues/2533)). In this update, we have updated the `sqlglot` dependency to a version greater than or equal to 25.5.0 and less than 25.20. The previous requirement allowed for versions up to 25.19, but we have chosen to update to a newer version that includes new features and bug fixes. The changelog and commit history for version 25.19 of `sqlglot` are provided in the pull request, highlighting the breaking changes, new features, and bug fixes. By updating to this version, we will benefit from the latest improvements and bug fixes in the `sqlglot` library. We encourage users to review the changelog and test their code to ensure compatibility with the new version.
+* [chore] fixed `make fmt` warnings related to sdk upgrade ([#2534](https://github.com/databrickslabs/ucx/issues/2534)). In this change, warnings related to a recent SDK upgrade have been addressed by modifying the `create` function in the `fixtures.py` file. The `create` function is responsible for generating a `Wait[ServingEndpointDetailed]` object, which contains information about the endpoint name and its core configuration. The `ServedModelInput` instance was previously created using positional arguments, but has been updated to use named arguments instead (`model_name`, `model_version`, `workload_size`, `scale_to_zero_enabled`). This modification enhances code readability and maintainability, making it easier for software engineers to understand and modify the codebase.
+
+Dependency updates:
+
+ * Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20 ([#2533](https://github.com/databrickslabs/ucx/pull/2533)).
+
 ## 0.34.0
 
 * Added a check for No isolation shared clusters and MLR ([#2484](https://github.com/databrickslabs/ucx/issues/2484)). This commit introduces a check for `No isolation shared clusters` utilizing MLR as part of the assessment workflow and cluster crawler, addressing issue [#846](https://github.com/databrickslabs/ucx/issues/846). A new function, `is_mlr`, has been implemented to determine if the Spark version corresponds to an MLR cluster. If the cluster has no isolation and uses MLR, the assessment failure list is appended with an appropriate error message. Thorough testing, including unit tests and manual verification, has been conducted. However, user documentation and new CLI commands, workflows, tables, or unit/integration tests have not been added. Additionally, a new test has been added to verify the behavior of MLR clusters without isolation, enhancing the assessment workflow's accuracy in identifying unsupported configurations.

src/databricks/labs/ucx/__about__.py

@@ -1,2 +1,2 @@
 # DO NOT MODIFY THIS FILE
-__version__ = "0.34.0"
+__version__ = "0.35.0"