v0.35.0

• Added databricks labs ucx delete-credential cmd to delete the UC roles created by UCX (#2504). In this release, we've added several new commands to the labs.yml file for managing Unity Catalog (UC) roles in Databricks, specifically for AWS. The new commands include databricks labs ucx delete-missing-principals and databricks labs ucx delete-credential. The databricks labs ucx delete-missing-principals command helps manage UC roles created through the create-missing-principals cmd by listing all the UC roles in AWS and allowing users to select roles to delete. It also checks for unused roles before deletion. The databricks labs ucx delete-credential command deletes UC roles created by UCX and is equipped with an optional aws-profile flag for authentication purposes. Additionally, we've added a new method delete_uc_role in the access.py file for deleting UC roles and introduced new test cases to ensure correct behavior. These changes resolve issue #2359, improving the overall management of UC roles in AWS.
• Added basic documentation for linter message codes (#2536). A new section, "Linter Message Codes," has been added to the README file, providing detailed explanations, examples, and resolution instructions for various linter message codes related to Unity Catalog (UC) migration. To help users familiarize themselves with the different message codes that may appear during linting, a new command, python tests/integration/source_code/message_codes.py, has been implemented. Running this command will display a list of message codes, including cannot-autofix-table-reference, catalog-api-in-shared-clusters, changed-result-format-in-uc, dbfs-read-from-sql-query, dbfs-usage, dependency-not-found, direct-filesystem-access, implicit-dbfs-usage, jvm-access-in-shared-clusters, legacy-context-in-shared-clusters, not-supported, notebook-run-cannot-compute-value, python-udf-in-shared-clusters, rdd-in-shared-clusters, spark-logging-in-shared-clusters, sql-parse-error, sys-path-cannot-compute-value, table-migrated-to-uc, to-json-in-shared-clusters, and unsupported-magic-line. Users are encouraged to review these message codes and their corresponding explanations to ensure a smooth migration to Unity Catalog.
• Added linters for direct filesystem access in Python and SQL code (#2519). In this release, linters have been added for detecting direct file system access (DFSA) in Python and SQL code, specifically addressing Direct File System Access in the Unity Catalog. Initially, the linters only detect DBFS, but the plan is to expand detection to all DSFSAs. This change is part of issue #2519 and includes new unit tests. The linters will flag code that accesses the file system directly, which is not allowed in Unity Catalog, including SQL queries that read from DBFS and Python code that reads from or displays data using DBFS paths. Developers are required to modify such code to use Unity Catalog tables or volumes instead, ensuring that their code is compatible with Unity Catalog's deprecation of direct file system access and DBFS, ultimately resulting in better project integration and performance.
• Clean up left over uber principal resources for AWS (#2449). In this release, we have made significant improvements to our open-source library, particularly in the management of AWS resources and permissions. We have added new methods to handle the creation and deletion of Uber instance profiles and external locations in AWS. The create_uber_principal method in the AWSResourcePermissions class has been updated to allow more fine-grained control over the migration process and ensure proper creation and configuration of all necessary resources. Additionally, we have introduced a new AWSResources class and aws_resource_permissions property in the aws_cli_ctx fixture to improve the management of AWS resources and permissions. We have also added new unit tests to ensure proper error handling when creating AWS IAM roles for Unity Catalog Migration (ucx) in specific scenarios. These changes enhance the functionality, test coverage, and overall quality of our library.
• Improve log warning about skipped grants (#2517). In this release, we have implemented improvements to the warning messages displayed during the verification of Unified Client Context (UCX) behavior for Access Control List (ACL) migration and User-Defined Function (UDF) behavior migration. Previously, generic warning messages were logged when specific Hive metastore grants could not be identified. This release enhances the warning messages by providing more specific information about the skipped grants, including the action type of the Hive metastore grant that failed to be mapped. Additionally, unittest.mock has been utilized to create a mock object for the GroupManager class, and a new method called MigrateGrants has been introduced, which applies a list of grant loaders to a specific table. These changes improve the logging and error handling, ensuring that software engineers have a clear understanding of any skipped grants during UCX and UDF behavior migration.
• Support sql notebooks in functional tests (#2513). This pull request introduces support for SQL notebooks in the functional testing framework, expanding its capabilities beyond Python notebooks. The changes include migrating relevant tests from test_notebook_linter to functional tests, as well as introducing new classes and methods to support SQL notebook testing. These changes improve the flexibility and scope of the testing framework, enabling developers to test SQL notebooks and ensuring that they meet quality standards. The commit also includes the addition of a new SQL notebook for demonstrating Unity Catalog table migrations, as well as modifications to various tests and regular expressions to accommodate SQL notebooks. Note that environment variables DATABRICKS_HOST and DATABRICKS_TOKEN are currently hardcoded as any, requiring further updates based on the specific testing environment.
• Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20 (#2533). In this update, we have updated the sqlglot dependency to a version greater than or equal to 25.5.0 and less than 25.20. The previous requirement allowed for versions up to 25.19, but we have chosen to update to a newer version that includes new features and bug fixes. The changelog and commit history for version 25.19 of sqlglot are provided in the pull request, highlighting the breaking changes, new features, and bug fixes. By updating to this version, we will benefit from the latest improvements and bug fixes in the sqlglot library. We encourage users to review the changelog and test their code to ensure compatibility with the new version.
• [chore] fixed make fmt warnings related to sdk upgrade (#2534). In this change, warnings related to a recent SDK upgrade have been addressed by modifying the create function in the fixtures.py file. The create function is responsible for generating a Wait[ServingEndpointDetailed] object, which contains information about the endpoint name and its core configuration. The ServedModelInput instance was previously created using positional arguments, but has been updated to use named arguments instead (model_name, model_version, workload_size, scale_to_zero_enabled). This modification enhances code readability and maintainability, making it easier for software engineers to understand and modify the codebase.

Dependency updates:

• Updated sqlglot requirement from <25.19,>=25.5.0 to >=25.5.0,<25.20 (#2533).

Contributors: @asnare, @ericvergnaud, @nfx, @FastLee, @dependabot[bot], @HariGS-DB, @JCZuurmond