v0.31.0

• Added handling for corrupted dashboard state in installation process (#2262). This commit introduces a new method, _handle_existing_dashboard, to manage various scenarios related to an existing dashboard during the installation process of UCX, including updating, upgrading from Redash to Lakeview, handling trashed dashboards, and recovering corrupted dashboard references. The _create_dashboard method now requires a non-null parent_path and has updated its docstring. New unit and integration tests have been added to verify the changes, with a particular focus on handling corrupted dashboard state during the installation process.
• Added support for migrating Table ACL for SQL Warehouse cluster in AWS using Instance Profile and Azure using SPN (#2258). This pull request introduces support for migrating Table Access Control (ACL) for SQL Warehouse clusters in Amazon Web Services (AWS) using Instance Profiles and Azure using Service Principal Names (SPNs). It includes identifying SQL Warehouse instances, associated Instance Profiles/SPNs, and principals with access to the warehouse, as well as retrieving their permissions and external locations. The code has been enhanced to represent compute and associated permissions more flexibly, improving the project's handling of both clusters and warehouses. New test cases and a retry mechanism for transient errors have been implemented to ensure robustness. The AzureServicePrincipalCrawler class has been added for managing SQL Warehouses using SPNs in Azure and Instance Profiles in AWS. These changes resolve issue #2238 and enhance the project's ability to handle ACL migration for SQL Warehouse clusters in AWS and Azure.
• Consistently have db-temp- as the backup prefix for renaming groups (#2266). In this release, we are implementing a consistent backup prefix for renaming workspace groups during migration, ensuring codebase and documentation consistency. The db-temp- prefix is now used for renaming groups, replacing the previous ucx-renamed- prefix in the rename_workspace_local_groups task. This change mitigates potential conflicts with account-level groups of the same name. The group migration workflow remains unaltered, except for the rename_workspace_local_groups task that now uses the new prefix. Affected files include 'config.py', 'groups.py', and 'test_groups.py', with corresponding changes in the WorkspaceConfig class, Groups class, and test cases. This feature is experimental, subject to further development, and may change in the future.
• Fixed astroid in the upload_dependencies (#2267). In this update, we have added the astroid library as a dependent library for UCX in the upload_wheel_dependencies function to resolve the reported issue #2257. Previously, the absence of astroid from the dependencies caused problems in certain workspaces. To address this, we modified the _upload_wheel function to include astroid in the list of libraries uploaded as wheel dependencies. This change has been manually tested and confirmed to work in a blocked workspace. No new methods have been added, and existing functionality has been updated within the _upload_wheel function to include astroid in the uploaded dependencies.
• Group migration: improve robustness when renaming groups (#2263). This pull request introduces changes to the group migration functionality to improve its robustness when renaming groups. Instead of assuming that a group rename has taken effect immediately after renaming it, the code now double-checks to ensure that the rename has taken place. This change affects the migrate-groups and migrate-groups-experimental workflows, which have been modified accordingly. Additionally, unit tests and existing integration tests have been updated to account for these changes. The test_rename_groups_should_patch_eligible_groups and test_rename_groups_should_wait_for_renames_to_complete tests have been updated to include a mock sleep function, allowing for more thorough testing of the rename process. The list and get methods of the workspace_client are mocked to return different values at different times, simulating the various stages of the rename process. This allows the tests to thoroughly exercise the code that handles group renames and ensures that it handles failure and success cases correctly. The methods _rename_group, _wait_for_group_rename, and _wait_for_renamed_groups have been added or modified to support this functionality. The _wait_for_workspace_group_deletion and _check_workspace_group_deletion methods have also been updated to support the deletion of original workspace groups. The delete_original_workspace_groups method has been modified to use these new and updated methods for deleting groups and confirming that the deletion has taken effect.
• Install state misses dashboards fields (#2275). In this release, we have resolved a bug related to the installation process in the databricks/labs/blueprint project that resulted in the omission of the dashboards field from the installation state. This bug was introduced in a previous update (#2229) which parallelized the installation process. This commit addresses the issue by saving the installation state at the end of the WorkspaceInstallation.run method, ensuring that the dashboards field is included in the state. Additionally, a new method _install_state.save() has been added to save the installation state. The changes also include adding a new method InstallState.from_installation() and a new test case test_installation_stores_install_state_keys() to retrieve the installation state and check for the presence of specific keys (jobs and dashboards). The test_uninstallation() test case has been updated to ensure that the installation and uninstallation processes work correctly. These changes enhance the installation and uninstallation functionality for the databricks/labs/blueprint project by ensuring that the installation state is saved correctly and that the jobs and dashboards keys are stored as expected, providing improved coverage and increased confidence in the functionality. The changes affect the existing command databricks labs install ucx.
• Use deterministic names to create AWS external locations (#2271). In this release, we have introduced deterministic naming for AWS external locations in our open-source library, addressing issue #2270. The run method in the locations.py file has been updated to generate deterministic names for external locations using the new _generate_external_location_name method. This method generates names based on the lowercase parts of a file path, joined by underscores, instead of using a prefix and a counter. Additionally, test cases for creating external locations in AWS have been updated to use the new naming convention, improving the predictability and consistency of the external location names. These changes simplify the management and validation of external locations, making it easier for software engineers to maintain and control the names of the external locations.

Contributors: @asnare, @HariGS-DB, @JCZuurmond, @pritishpai, @nfx, @FastLee