Skip to content

Commit c6019ad

Browse files
nfxnfx
authored
Release v0.1.0 (#213)
# Version changelog ## 0.1.0 Features * Added interactive installation wizard ([#184](#184), [#117](#117)). * Added schedule of jobs as part of `install.sh` flow and created some documentation ([#187](#187)). * Added debug notebook companion to troubleshoot the installation ([#191](#191)). * Added support for Hive Metastore Table ACLs inventory from all databases ([#78](#78), [#122](#122), [#151](#151)). * Created `$inventory.tables` from Scala notebook ([#207](#207)). * Added local group migration support for ML-related objects ([#56](#56)). * Added local group migration support for SQL warehouses ([#57](#57)). * Added local group migration support for all compute-related resources ([#53](#53)). * Added local group migration support for security-related objects ([#58](#58)). * Added local group migration support for workflows ([#54](#54)). * Added local group migration support for workspace-level objects ([#59](#59)). * Added local group migration support for dashboards, queries, and alerts ([#144](#144)). Stability * Added `codecov.io` publishing ([#204](#204)). * Added more tests to group.py ([#148](#148)). * Added tests for group state ([#133](#133)). * Added tests for inventorizer and typed ([#125](#125)). * Added tests WorkspaceListing ([#110](#110)). * Added `make_*_permissions` fixtures ([#159](#159)). * Added reusable fixtures module ([#119](#119)). * Added testing for permissions ([#126](#126)). * Added inventory table manager tests ([#153](#153)). * Added `product_info` to track as SDK integration ([#76](#76)). * Added failsafe permission get operations ([#65](#65)). * Always install the latest `pip` version in `./install.sh` ([#201](#201)). * Always store inventory in `hive_metastore` and make only `inventory_database` configurable ([#178](#178)). * Changed default logging level from `TRACE` to `DEBUG` log level ([#124](#124)). * Consistently use `WorkspaceClient` from `databricks.sdk` ([#120](#120)). * Convert pipeline code to use fixtures. ([#166](#166)). * Exclude mixins from coverage ([#130](#130)). * Fixed codecov.io reporting ([#212](#212)). * Fixed configuration path in job task install code ([#210](#210)). * Fixed a bug with dependency definitions ([#70](#70)). * Fixed failing `test_jobs` ([#140](#140)). * Fixed the issues with experiment listing ([#64](#64)). * Fixed integration testing configuration ([#77](#77)). * Make project runnable on nightly testing infrastructure ([#75](#75)). * Migrated cluster policies to new fixtures ([#174](#174)). * Migrated clusters to the new fixture framework ([#162](#162)). * Migrated instance pool to the new fixture framework ([#161](#161)). * Migrated to `databricks.labs.ucx` package ([#90](#90)). * Migrated token authorization to new fixtures ([#175](#175)). * Migrated experiment fixture to standard one ([#168](#168)). * Migrated jobs test to fixture based one. ([#167](#167)). * Migrated model fixture to the standard fixtures ([#169](#169)). * Migrated warehouse fixture to standard one ([#170](#170)). * Organise modules by domain ([#197](#197)). * Prefetch all account-level and workspace-level groups ([#192](#192)). * Programmatically create a dashboard ([#121](#121)). * Properly integrate Python `logging` facility ([#118](#118)). * Refactored code to use Databricks SDK for Python ([#27](#27)). * Refactored configuration and remove global provider state ([#71](#71)). * Removed `pydantic` dependency ([#138](#138)). * Removed redundant `pyspark`, `databricks-connect`, `delta-spark`, and `pandas` dependencies ([#193](#193)). * Removed redundant `typer[all]` dependency and its usages ([#194](#194)). * Renamed `MigrationGroupsProvider` to `GroupMigrationState` ([#81](#81)). * Replaced `ratelimit` and `tenacity` dependencies with simpler implementations ([#195](#195)). * Reorganised integration tests to align more with unit tests ([#206](#206)). * Run `build` workflow also on `main` branch ([#211](#211)). * Run integration test with a single group ([#152](#152)). * Simplify `SqlBackend` and table creation logic ([#203](#203)). * Updated `migration_config.yml` ([#179](#179)). * Updated legal information ([#196](#196)). * Use `make_secret_scope` fixture ([#163](#163)). * Use fixture factory for `make_table`, `make_schema`, and `make_catalog` ([#189](#189)). * Use new fixtures for notebooks and folders ([#176](#176)). * Validate toolkit notebook test ([#183](#183)). Contributing * Added a note on external dependencies ([#139](#139)). * Added ability to run SQL queries on Spark when in Databricks Runtime ([#108](#108)). * Added some ground rules for contributing ([#82](#82)). * Added contributing instructions link from main readme ([#109](#109)). * Added info about environment refreshes ([#155](#155)). * Clarified documentation ([#137](#137)). * Enabled merge queue ([#146](#146)). * Improved `CONTRIBUTING.md` guide ([#135](#135), [#145](#145)).
1 parent 1c427b3 commit c6019ad

File tree

5 files changed

+116
-73
lines changed

5 files changed

+116
-73
lines changed

‎CHANGELOG.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Version changelog
2+
3+
## 0.1.0
4+
5+
Features
6+
7+
* Added interactive installation wizard ([#184](https://github.com/databricks/ucx/pull/184), [#117](https://github.com/databricks/ucx/pull/117)).
8+
* Added schedule of jobs as part of `install.sh` flow and created some documentation ([#187](https://github.com/databricks/ucx/pull/187)).
9+
* Added debug notebook companion to troubleshoot the installation ([#191](https://github.com/databricks/ucx/pull/191)).
10+
* Added support for Hive Metastore Table ACLs inventory from all databases ([#78](https://github.com/databricks/ucx/pull/78), [#122](https://github.com/databricks/ucx/pull/122), [#151](https://github.com/databricks/ucx/pull/151)).
11+
* Created `$inventory.tables` from Scala notebook ([#207](https://github.com/databricks/ucx/pull/207)).
12+
* Added local group migration support for ML-related objects ([#56](https://github.com/databricks/ucx/pull/56)).
13+
* Added local group migration support for SQL warehouses ([#57](https://github.com/databricks/ucx/pull/57)).
14+
* Added local group migration support for all compute-related resources ([#53](https://github.com/databricks/ucx/pull/53)).
15+
* Added local group migration support for security-related objects ([#58](https://github.com/databricks/ucx/pull/58)).
16+
* Added local group migration support for workflows ([#54](https://github.com/databricks/ucx/pull/54)).
17+
* Added local group migration support for workspace-level objects ([#59](https://github.com/databricks/ucx/pull/59)).
18+
* Added local group migration support for dashboards, queries, and alerts ([#144](https://github.com/databricks/ucx/pull/144)).
19+
20+
Stability
21+
22+
* Added `codecov.io` publishing ([#204](https://github.com/databricks/ucx/pull/204)).
23+
* Added more tests to group.py ([#148](https://github.com/databricks/ucx/pull/148)).
24+
* Added tests for group state ([#133](https://github.com/databricks/ucx/pull/133)).
25+
* Added tests for inventorizer and typed ([#125](https://github.com/databricks/ucx/pull/125)).
26+
* Added tests WorkspaceListing ([#110](https://github.com/databricks/ucx/pull/110)).
27+
* Added `make_*_permissions` fixtures ([#159](https://github.com/databricks/ucx/pull/159)).
28+
* Added reusable fixtures module ([#119](https://github.com/databricks/ucx/pull/119)).
29+
* Added testing for permissions ([#126](https://github.com/databricks/ucx/pull/126)).
30+
* Added inventory table manager tests ([#153](https://github.com/databricks/ucx/pull/153)).
31+
* Added `product_info` to track as SDK integration ([#76](https://github.com/databricks/ucx/pull/76)).
32+
* Added failsafe permission get operations ([#65](https://github.com/databricks/ucx/pull/65)).
33+
* Always install the latest `pip` version in `./install.sh` ([#201](https://github.com/databricks/ucx/pull/201)).
34+
* Always store inventory in `hive_metastore` and make only `inventory_database` configurable ([#178](https://github.com/databricks/ucx/pull/178)).
35+
* Changed default logging level from `TRACE` to `DEBUG` log level ([#124](https://github.com/databricks/ucx/pull/124)).
36+
* Consistently use `WorkspaceClient` from `databricks.sdk` ([#120](https://github.com/databricks/ucx/pull/120)).
37+
* Convert pipeline code to use fixtures. ([#166](https://github.com/databricks/ucx/pull/166)).
38+
* Exclude mixins from coverage ([#130](https://github.com/databricks/ucx/pull/130)).
39+
* Fixed codecov.io reporting ([#212](https://github.com/databricks/ucx/pull/212)).
40+
* Fixed configuration path in job task install code ([#210](https://github.com/databricks/ucx/pull/210)).
41+
* Fixed a bug with dependency definitions ([#70](https://github.com/databricks/ucx/pull/70)).
42+
* Fixed failing `test_jobs` ([#140](https://github.com/databricks/ucx/pull/140)).
43+
* Fixed the issues with experiment listing ([#64](https://github.com/databricks/ucx/pull/64)).
44+
* Fixed integration testing configuration ([#77](https://github.com/databricks/ucx/pull/77)).
45+
* Make project runnable on nightly testing infrastructure ([#75](https://github.com/databricks/ucx/pull/75)).
46+
* Migrated cluster policies to new fixtures ([#174](https://github.com/databricks/ucx/pull/174)).
47+
* Migrated clusters to the new fixture framework ([#162](https://github.com/databricks/ucx/pull/162)).
48+
* Migrated instance pool to the new fixture framework ([#161](https://github.com/databricks/ucx/pull/161)).
49+
* Migrated to `databricks.labs.ucx` package ([#90](https://github.com/databricks/ucx/pull/90)).
50+
* Migrated token authorization to new fixtures ([#175](https://github.com/databricks/ucx/pull/175)).
51+
* Migrated experiment fixture to standard one ([#168](https://github.com/databricks/ucx/pull/168)).
52+
* Migrated jobs test to fixture based one. ([#167](https://github.com/databricks/ucx/pull/167)).
53+
* Migrated model fixture to the standard fixtures ([#169](https://github.com/databricks/ucx/pull/169)).
54+
* Migrated warehouse fixture to standard one ([#170](https://github.com/databricks/ucx/pull/170)).
55+
* Organise modules by domain ([#197](https://github.com/databricks/ucx/pull/197)).
56+
* Prefetch all account-level and workspace-level groups ([#192](https://github.com/databricks/ucx/pull/192)).
57+
* Programmatically create a dashboard ([#121](https://github.com/databricks/ucx/pull/121)).
58+
* Properly integrate Python `logging` facility ([#118](https://github.com/databricks/ucx/pull/118)).
59+
* Refactored code to use Databricks SDK for Python ([#27](https://github.com/databricks/ucx/pull/27)).
60+
* Refactored configuration and remove global provider state ([#71](https://github.com/databricks/ucx/pull/71)).
61+
* Removed `pydantic` dependency ([#138](https://github.com/databricks/ucx/pull/138)).
62+
* Removed redundant `pyspark`, `databricks-connect`, `delta-spark`, and `pandas` dependencies ([#193](https://github.com/databricks/ucx/pull/193)).
63+
* Removed redundant `typer[all]` dependency and its usages ([#194](https://github.com/databricks/ucx/pull/194)).
64+
* Renamed `MigrationGroupsProvider` to `GroupMigrationState` ([#81](https://github.com/databricks/ucx/pull/81)).
65+
* Replaced `ratelimit` and `tenacity` dependencies with simpler implementations ([#195](https://github.com/databricks/ucx/pull/195)).
66+
* Reorganised integration tests to align more with unit tests ([#206](https://github.com/databricks/ucx/pull/206)).
67+
* Run `build` workflow also on `main` branch ([#211](https://github.com/databricks/ucx/pull/211)).
68+
* Run integration test with a single group ([#152](https://github.com/databricks/ucx/pull/152)).
69+
* Simplify `SqlBackend` and table creation logic ([#203](https://github.com/databricks/ucx/pull/203)).
70+
* Updated `migration_config.yml` ([#179](https://github.com/databricks/ucx/pull/179)).
71+
* Updated legal information ([#196](https://github.com/databricks/ucx/pull/196)).
72+
* Use `make_secret_scope` fixture ([#163](https://github.com/databricks/ucx/pull/163)).
73+
* Use fixture factory for `make_table`, `make_schema`, and `make_catalog` ([#189](https://github.com/databricks/ucx/pull/189)).
74+
* Use new fixtures for notebooks and folders ([#176](https://github.com/databricks/ucx/pull/176)).
75+
* Validate toolkit notebook test ([#183](https://github.com/databricks/ucx/pull/183)).
76+
77+
Contributing
78+
79+
* Added a note on external dependencies ([#139](https://github.com/databricks/ucx/pull/139)).
80+
* Added ability to run SQL queries on Spark when in Databricks Runtime ([#108](https://github.com/databricks/ucx/pull/108)).
81+
* Added some ground rules for contributing ([#82](https://github.com/databricks/ucx/pull/82)).
82+
* Added contributing instructions link from main readme ([#109](https://github.com/databricks/ucx/pull/109)).
83+
* Added info about environment refreshes ([#155](https://github.com/databricks/ucx/pull/155)).
84+
* Clarified documentation ([#137](https://github.com/databricks/ucx/pull/137)).
85+
* Enabled merge queue ([#146](https://github.com/databricks/ucx/pull/146)).
86+
* Improved `CONTRIBUTING.md` guide ([#135](https://github.com/databricks/ucx/pull/135), [#145](https://github.com/databricks/ucx/pull/145)).

‎README.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
# UCX - Unity Catalog Migration Toolkit
22

3-
Your best companion for enabling the Unity Catalog.
3+
[![build](https://github.com/databrickslabs/ucx/actions/workflows/push.yml/badge.svg)](https://github.com/databrickslabs/ucx/actions/workflows/push.yml) [![codecov](https://codecov.io/github/databrickslabs/ucx/graph/badge.svg?token=p0WKAfW5HQ)](https://codecov.io/github/databrickslabs/ucx)
4+
5+
Your best companion for enabling the Unity Catalog. It helps you to migrate all Databricks workspace assets:
6+
Entitlements, AWS instance profiles, Clusters, Cluster policies, Instance Pools, Databricks SQL warehouses, Delta Live
7+
Tables, Jobs, MLflow experiments, MLflow registry, SQL Dashboards & Queries, SQL Alerts, Token and Password usage
8+
permissions that are set on the workspace level, Secret scopes, Notebooks, Directories, Repos, Files.
9+
10+
See [contributing instructions](CONTRIBUTING.md) to help improve this project.
411

512
## Installation
613

@@ -18,19 +25,19 @@ export DATABRICKS_CONFIG_PROFILE=ABC
1825
```
1926

2027
You can also specify environment variables in a more direct way, like in this example for installing
21-
on a Azure Databricks Workspace using the Azure CLI authentication:
28+
on an Azure Databricks Workspace using the Azure CLI authentication:
2229

2330
```shell
2431
az login
2532
export DATABRICKS_HOST=https://adb-123....azuredatabricks.net/
2633
./install.sh
2734
```
2835

29-
## Latest working version and how-to
36+
Please follow the instructions in `./install.sh`, which will open a notebook with the description of all jobs to trigger. The journey starts with assessment.
3037

31-
Please note that current project statis is 🏗️ **WIP**, but we have a minimal set of already working utilities.
38+
## Star History
3239

33-
See [contributing instructions](CONTRIBUTING.md).
40+
[![Star History Chart](https://api.star-history.com/svg?repos=databrickslabs/ucx&type=Date)](https://star-history.com/#databrickslabs/ucx)
3441

3542
## Project Support
3643
Please note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.

‎USAGE.md‎

Lines changed: 0 additions & 67 deletions
This file was deleted.

‎docs/logic.md‎ renamed to ‎docs/local-group-migration.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,22 @@
11
# Permissions migration logic and data structures
22

3+
During the UC adoption, it's critical to move the groups from the workspace to account level.
4+
5+
To deliver this migration, the following steps are performed:
6+
7+
| Step description | Relevant API method |
8+
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
9+
| A set of groups to be migrated is identified (either via `groups.selected` config property, or automatically).<br/>Group existence is verified against the account level.<br/>**If there is no group on the account level, an error is thrown.**<br/>Backup groups are created on the workspace level. | `toolkit.prepare_groups_in_environment()` |
10+
| Inventory table is cleaned up. | `toolkit.cleanup_inventory_table()` |
11+
| Workspace local group permissions are inventorized and saved into a Delta Table. | `toolkit.inventorize_permissions()` |
12+
| Backup groups are entitled with permissions from the inventory table. | `toolkit.apply_permissions_to_backup_groups()` |
13+
| Workspace-level groups are deleted. Account-level groups are granted with access to the workspace.<br/>Workspace-level entitlements are synced from backup groups to newly added account-level groups. | `toolkit.replace_workspace_groups_with_account_groups()` |
14+
| Account-level groups are entitled with workspace-level permissions from the inventory table. | `toolkit.apply_permissions_to_account_groups()` |
15+
| Backup groups are deleted | `toolkit.delete_backup_groups()` |
16+
| Inventory table is cleaned up. This step is optional. | `toolkit.cleanup_inventory_table()` |
17+
18+
> Please note that inherited permissions will not be inventorized / migrated. We only cover direct permissions.
19+
320
On a very high-level, the permissions inventorization process is split into two steps:
421

522
1. collect all existing permissions into a persistent storage.

‎src/databricks/labs/ucx/__about__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "0.0.3"
1+
__version__ = "0.1.0"

0 commit comments

Comments
 (0)