v0.21.0

• Ensure proper sequencing of view migrations (#1157). In this release, we have introduced a views_migrator module and corresponding test cases to ensure proper sequencing of view migrations, addressing issue #1132. The module contains two main classes: ViewToMigrate and ViewsMigrator. The former is responsible for parsing a view's SQL text and identifying its dependencies, while the latter sequences views based on their dependencies. The commit also adds a new method, __hash__, to the Table class, which returns a hash value of the key of the table, improving the handling of Table objects. Additionally, we have added unit tests and verified the changes on a staging environment. We have also introduced a new file tables_and_views.json for unit testing and added a views_migrator module that takes a TablesCrawler object and returns a sequence of tables (views) that need to be migrated in the correct order. The commit addresses various scenarios such as no views, direct views, indirect views, deep indirect views, invalid SQL, invalid SQL tables, and circular view references. This release is focused on improving the sequencing of view migrations and is accompanied by appropriate tests.
• Experimental support for scanning Delta Tables inside Mount Points (#1095). This commit introduces experimental support for scanning Delta Tables located inside mount points using a new TablesInMounts crawler. Users can now scan specific mount points using the --include-mounts flag and include Parquet files in the scan results with the --include-parquet-files flag. Additionally, the --filter-paths flag allows for filtering paths in a mount point and the --max-depth flag (currently unimplemented) will filter at a specific sub-folder depth in future development. The project dependencies have been updated to use databricks-labs-lsql~=0.3.0. This new feature provides a more granular and flexible way to scan Delta Tables, making the project more user-friendly and adaptable to various use cases.
• Fixed NULL values in ucx.views.table_format to have UNKNOWN value instead (#1156). This commit includes a fix for handling NULL values in the table_format column of Views in the ucx.views.table_format module. Previously, NULL values were displayed as-is, but now they will be replaced with the string "UNKNOWN". This change is part of the fix for issue #115
• Fixing run_workflow functionality for better error handling (#1159). In this release, the run_workflow method in the workflows.py file has been updated to improve error handling by waiting for the job to terminate or skip before raising an error, allowing for a more detailed error message to be generated. A new method, job_initial_run, has been added to initiate a job run and return the run ID, raising a NotFound exception if the job run is not found. The run_workflow functionality in the WorkflowsInstall module has also been enhanced to handle unexpected error types and improve overall error handling during the installation of products. New test cases have been added and existing ones updated to check how the code handles errors when the run ID is not found or when an OperationFailed exception is raised during the installation process. These changes improve the robustness and stability of the system.
• Use experimental Permissions Migration API also for Legacy Table ACLs (#1161). This release introduces several changes to the group permissions migration functionality and associated tests. The experimental Permissions Migration API is now being utilized for Legacy Table ACLs, which has led to the removal of the verification step from the experimental group migration job. The TableAclSupport import and class have been removed, as they are no longer needed. A new apply_to_renamed_groups method has been added for production usage, and a apply_to_groups_with_different_names method has been added for integration testing, both of which are part of the Permissions Migration API. Additionally, two tests have been added to support the experimental permissions migration for a group with the same name in the workspace and account. The permission_manager parameter has been removed from several test functions in the test_generic.py file and replaced with the MigrationState class, which is used directly with the WorkspaceClient object to apply permissions to groups with different names. The test_some_entitlements function in the test_scim.py file has also been updated to use the MigratedGroup class and the MigrationState class's apply_to_groups_with_different_names method. Finally, new tests for the Permissions Migration API have been added to the test_tacl.py file in the tests/integration/workspace_access directory to verify the behavior of the Permissions Migration API when migrating different grants.

Contributors: @ericvergnaud, @qziyuan, @nfx, @FastLee, @william-conti, @dmoore247, @pritishpai