v0.32.0

• Added troubleshooting guide for self-signed SSL cert related error (#2346). In this release, we have added a troubleshooting guide to the README file to address a specific error that may occur when connecting from a local machine to a Databricks Account and Workspace using a web proxy and self-signed SSL certificate. This error, SSLCertVerificationError, can prevent UCX from connecting to the Account and Workspace. To resolve this issue, users can now set the REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE environment variables to force the requests library to set verify=False, and set the SSL_CERT_DIR env var pointing to the proxy CA cert for the urllib3 library. This guide will help users understand and resolve this error, making it easier to connect to Databricks Accounts and Workspaces using a web proxy and self-signed SSL certificate.
• Code Compatibility Dashboard: Fix broken links (#2347). In this release, we have addressed and resolved two issues in the Code Compatibility Dashboard of the UCX Migration (Main) project, enhancing its overall usability. Previously, the Markdown panel contained a broken link to the workflow due to an incorrect anchor, and the links in the table widget to the workflow and task definitions did not render correctly. These problems have been rectified, and the dashboard has been manually tested and verified in a staging environment. Additionally, we have updated the invisibleColumns section in the SQL file by changing the fieldName attribute to 'name', which will now display the workflow_id as a link. Before and after screenshots have been provided for visual reference. The corresponding workflow is now referred to as "Jobs Static Code Analysis Workflow".
• Filter out missing import problems for imports within a try-except clause with ImportError (#2332). This release introduces changes to handle missing import problems within a try-except clause that catches ImportError. A new method, _filter_import_problem_in_try_except, has been added to filter out import-not-found issues when they occur in such a clause, preventing unnecessary build failures. The _register_import method now returns an Iterable[DependencyProblem] instead of yielding problems directly. Supporting classes and methods, including Dependency, DependencyGraph, and DependencyProblem from the databricks.labs.ucx.source_code.graph module, as well as FileLoader and PythonCodeAnalyzer from the databricks.labs.ucx.source_code.notebooks.cells module, have been added. The ImportSource.extract_from_tree method has been updated to accept a DependencyProblem object as an argument. Additionally, a new test case has been included for the scenario where a failing import in a try-except clause goes unreported. Issue #1705 has been resolved, and unit tests have been added to ensure proper functionality.
• Fixed report-account-compatibility cli command docstring (#2340). In this release, we have updated the report-account-compatibility CLI command's docstring to accurately reflect its functionality, addressing a previous issue where it inadvertently duplicated the sync-workspace-info command's description. This command now provides a clear and concise explanation of its purpose: "Report compatibility of all workspaces available in the account." Upon execution, it generates a readiness report for the account, specifically focusing on workspaces where ucx is installed. This enhancement improves the clarity of the CLI's functionality for software engineers, enabling them to understand and effectively utilize the report-account-compatibility command.
• Fixed broken table migration workflow links in README (#2286). In this release, we have made significant improvements to the README file of our open-source library, including fixing broken links and adding a mermaid flowchart to demonstrate the table migration workflows. The table migration workflow has been renamed to the table migration process, which includes migrating Delta tables, non-Delta tables, external tables, and views. Two optional workflows have been added for migrating HiveSerDe tables in place and for migrating external tables using CTAS. Additionally, the commands related to table migration have been updated, with the table migration workflow being renamed to the table migration process. These changes are aimed at providing a more comprehensive understanding of the table migration process and enhancing the overall user experience.
• Fixed dashboard queries fail when default catalog is not hive_metastore (#2278). In this release, we have addressed an issue where dashboard queries fail when the default catalog is not set to hive_metastore. This has been achieved by modifying the existing databricks labs ucx install command to always include the hive_metastore namespace in dashboard queries. Additionally, the code has been updated to add the hive_metastore namespace to the DashboardMetadata object used in creating a dashboard from SQL queries in a folder, ensuring queries are executed in the correct database. The commit also includes modifications to the test_install.py unit test file to ensure the installation process correctly handles specific configurations related to the ucx namespace for managing data storage and retrieval. The changes have been manually tested and verified on a staging environment.
• Improve group migration error reporting (#2344). This PR introduces enhancements to the group migration dashboard, focusing on improved error reporting and a more informative user experience. The documentation widgets have been fine-tuned, and the failed-migration widget now provides formatted failure information with a link to the failed job run. The dashboard will display only failures from the latest workflow run, complete with logs. A new link to the job list has been added in the workflows section of the documentation to assist users in identifying and troubleshooting issues. Additionally, the SQL query for retrieving group migration failure information has been refactored, improving readability and extracting relevant data using regular expressions. The changes have been tested and verified on the staging environment, providing clearer and more actionable insights during group migrations. The PR is related to previous work in #2333 and #1914, with updates to the UCX Migration (Groups) dashboard, but no new methods have been added.
• Improve type checking in cli command (#2335). This release introduces enhanced type checking in the command line interface (CLI) of our open-source library, specifically in the lint_local_code function of the cli.py file. By utilizing a newly developed local code linter object, the function now performs more rigorous and accurate type checking for potential issues in the local code. While the functionality remains consistent, this improvement is expected to prevent similar occurrences like issue #2221, ensuring more robust and reliable code. This change underscores our commitment to delivering a high-quality, efficient, and developer-friendly library.
• Lint dependencies in context (#2236). The InheritedContext class has been introduced to gather code fragments from parent files or notebooks during linting of child files or notebooks, addressing issues #2155, #2156, and #2221. This new feature includes the addition of the InheritedContext class, with methods for building instances from a route of dependencies, appending other InheritedContext instances, and finalizing them for use with linting. The DependencyGraph class has been updated to support the new functionality, and various classes, methods, and functions for handling the linter context have been added or updated. Unit, functional, and integration tests have been added to ensure the correct functioning of the changes, which improve the linting functionality by allowing it to consider the broader context of the codebase.
• Make ucx pylsp plugin configurable (#2280). This commit introduces the ability to configure the ucx pylsp plugin with cluster information, which can be provided either in a file or by a client and is managed by the pylsp infrastructure. The Spark Connect linter is now only applied to UC Shared clusters, as Single-User clusters run in Spark Classic mode. A new entry point pylsp_ucx has been added to the pylsp configuration file. The changes affect the pylsp plugin configuration and the application of the Spark Connect linter. Unit tests and manual testing have been conducted, but integration tests and verification on a staging environment are not included in this release.
• New dashboard: group migration, showing groups that failed to migrate (#2333). In this release, we have developed a new dashboard for monitoring group migration in the UCX Migration (Groups) workspace. This dashboard includes a widget displaying messages related to groups that failed to migrate during the migrate-groups-experimental workflow, aiding users in identifying and addressing migration issues. The group migration process consists of several steps, including renaming workspace groups, provisioning account-level groups, and replicating permissions. The release features new methods for displaying and monitoring migration-related messages, as well as links to documentation and workflows for assessing, validating, and removing workspace-level groups post-migration. The new dashboard is currently not connected to the existing system, but it has undergone manual testing and verification on the staging environment. The changes include the addition of a new SQL query file to implement the logic for fetching group migration failures and a new Markdown file displaying the Group Migration Failures section.
• Support spaces in run cmd args (#2330). The recent commit resolves an issue where the system had trouble handling spaces in command-line arguments when running subprocesses. The previous implementation only accepted a full command line, which it would split on spaces, causing problems when the command line contained arguments with spaces. The new implementation supports argument lists, which are passed as is to Popen, allowing for proper handling of command lines with spaces. This change is incorporated in the run_command function of the utils.py file and the _install_pip method of the PythonLibraryResolver class. The shlex.join() function has been replaced with direct string formatting for increased flexibility. The feature is intended for use with the PythonLibraryResolver class and is co-authored by Eric Vergnaud and Andrew Snare. Integration tests have been enabled to ensure the proper functioning of the updated code.
• Updated error messages for SparkConnect linter (#2348). The SparkConnect linter's error messages have been updated to improve clarity and precision. The term UC Shared clusters has been replaced with Unity Catalog clusters in Shared access mode throughout the codebase, affecting messages related to various unsupported functionalities or practices on these clusters. These changes include warnings about direct Spark log level setting, accessing the Spark Driver JVM or its logger, using sc, and employing RDD APIs. This revision enhances user experience by providing more accurate and descriptive error messages, enabling them to better understand and address the issues in their code. The functionality of the linter remains unchanged.
• Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9 (#2279). In this update, we have updated the required version range of the sqlglot dependency in the 'pyproject.toml' file from 'sqlglot>=25.5.0,<25.8' to 'sqlglot>=25.5.0,<25.9'. This change allows the project to utilize any version of sqlglot that is greater than or equal to 25.5.0 and less than 25.9, including the latest version. The update also includes a changelog for the updated version range, sourced from 'sqlglot's official changelog. This changelog includes various bug fixes and new features for several dialects such as BigQuery, DuckDB, and tSQL. Additionally, the parser has undergone some refactors and improvements. The commits section lists the individual commits included in this update.

Dependency updates:

• Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9 (#2279).

Contributors: @asnare, @ericvergnaud, @JCZuurmond, @qziyuan, @vsevolodstep-db, @dependabot[bot]