Skip to content

Commit 0efead5

Browse files
qziyuanJCZuurmond
andauthored
Fix broken table migration workflow links in README (#2286)
1. Fix broken #table-migration-workflow links in README. 2. Add mermaid flow chart to demonstrate the table migration workflows. --------- Co-authored-by: Cor <[email protected]>
1 parent dcca323 commit 0efead5

File tree

1 file changed

+57
-16
lines changed

1 file changed

+57
-16
lines changed

README.md

Lines changed: 57 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ After [installation](#install-ucx), ensure to [trigger](#ensure-assessment-run-c
88
so that you'll be able to [scope the migration](docs/assessment.md) and execute the [group migration workflow](#group-migration-workflow).
99

1010
The [README notebook](#readme-notebook), which can be found in the installation folder contains further instructions and explanations of the different ucx workflows & dashboards.
11-
Once the migration is scoped, you can start executing the [table migration workflow](#table-migration-workflow).
11+
Once the migration is scoped, you can start with the [table migration process](#Table-Migration).
1212

1313
More workflows, like notebook code migration are coming in future releases.
1414

@@ -269,7 +269,7 @@ Databricks CLI will confirm a few options:
269269
# Migration process
270270

271271
On the high level, the steps in migration process start with the [assessment workflow](#assessment-workflow),
272-
followed by [group migration](#group-migration-workflow), [table migration workflow](#table-migration-workflow),
272+
followed by [group migration](#group-migration-workflow), [table migration process](#Table-Migration),
273273
finalised with the [code migration](#code-migration-commands). It can be described as:
274274

275275
```mermaid
@@ -369,7 +369,7 @@ Databricks SQL warehouses, Delta Live Tables, Jobs, MLflow experiments, MLflow r
369369
SQL Alerts, Token and Password usage permissions that are set on the workspace level, Secret scopes, Notebooks,
370370
Directories, Repos, and Files.
371371

372-
Once done with the group migration, proceed to [table migration workflow](#table-migration-workflow).
372+
Once done with the group migration, proceed to [table migration process](#Table-Migration).
373373

374374
Use [`validate-groups-membership` command](#validate-groups-membership-command) for extra confidence.
375375
If you don't have matching account groups, please run [`create-account-groups` command](#create-account-groups-command).
@@ -536,7 +536,48 @@ We also add a `upgraded_from_workspace_id` property to the upgraded object, to i
536536

537537
The upgrade process can be triggered using the `migrate-tables` [UCX command](#migrate-tables-command)
538538

539-
Or by running the "Migrate Tables" workflow deployed to the workspace.
539+
Or by running the table migration workflows deployed to the workspace.
540+
541+
#### Table migration workflows
542+
543+
```mermaid
544+
flowchart LR
545+
subgraph CLI
546+
migrate-tables[migrate-tables]
547+
end
548+
549+
subgraph mt_workflow[workflow: migrate-tables]
550+
dbfs_root_delta_mt_task[migrate_dbfs_root_delta_tables]
551+
migrate_dbfs_root_non_delta_tables[migrate_dbfs_root_non_delta_tables]
552+
external_tables_sync_mt_task[migrate_external_tables_sync]
553+
view_mt_task[migrate_views]
554+
refresh_migration_status[refresh_migration_status]
555+
external_tables_sync_mt_task --> view_mt_task
556+
dbfs_root_delta_mt_task --> view_mt_task
557+
migrate_dbfs_root_non_delta_tables --> view_mt_task
558+
view_mt_task --> refresh_migration_status
559+
end
560+
561+
subgraph mt_serde_inplace_wf[workflow: migrate-external-hiveserde-tables-in-place-experimental]
562+
serde_inplace_mt_task[migrate_hive_serde_in_place] --> view_mt_task_inplace[migrate_views]
563+
view_mt_task_inplace --> refresh_migration_status_hiveserde[refresh_migration_status]
564+
end
565+
566+
subgraph mt_ctas_wf[workflow: migrate-external-tables-ctas]
567+
ctas_mt_task[migrate_other_external_ctas]
568+
migrate_hive_serde_ctas[migrate_hive_serde_ctas]
569+
view_mt_ctas_task[migrate_views]
570+
refresh_migration_status_ctas[refresh_migration_status]
571+
ctas_mt_task --> view_mt_ctas_task
572+
migrate_hive_serde_ctas --> view_mt_ctas_task
573+
view_mt_ctas_task --> refresh_migration_status_ctas
574+
end
575+
576+
migrate-tables -- 1st --> mt_workflow
577+
migrate-tables -- 2nd, optional \n table migrated here \n will be excluded in ctas workflow --> mt_serde_inplace_wf
578+
migrate-tables -- 3rd --> mt_ctas_wf
579+
```
580+
540581

541582
#### Step 4: Odds and Ends
542583
The following steps can be used to repair/amend the metastore after the upgrade process.
@@ -809,7 +850,7 @@ the metastore id of the metastore you want to assign to the workspace.
809850

810851
# Table migration commands
811852

812-
These commands are vital part of [table migration workflow](#table-migration-workflow) process and require
853+
These commands are vital part of [table migration process](#Table-Migration) process and require
813854
the [assessment workflow](#assessment-workflow) and
814855
[group migration workflow](#group-migration-workflow) to be completed.
815856
See the [migration process diagram](#migration-process) to understand the role of the table migration commands in
@@ -923,7 +964,7 @@ workflows. Once migration is complete, this service principal should be unprovis
923964
On Azure, it creates a principal with `Storage Blob Data Contributor` role assignment on every storage account using
924965
Azure Resource Manager APIs.
925966

926-
This command is one of prerequisites for the [table migration workflow](#table-migration).
967+
This command is one of prerequisites for the [table migration process](#table-migration).
927968

928969
[[back to top](#databricks-labs-ucx)]
929970

@@ -1015,7 +1056,7 @@ multiple runs of the table migration workflow, you can use the [`revert-migrated
10151056
to revert the tables that were migrated in the previous run. You can also skip the tables that you don't want to migrate
10161057
using the [`skip` command](#skip-command).
10171058

1018-
This command is one of prerequisites for the [table migration workflow](#table-migration-workflow).
1059+
This command is one of prerequisites for the [table migration process](#Table-Migration).
10191060

10201061
Once you're done with table migration, proceed to the [code migration](#code-migration-commands).
10211062

@@ -1044,7 +1085,7 @@ Once you're done with table migration, proceed to the [code migration](#code-mig
10441085
databricks labs ucx create-catalogs-schemas
10451086
```
10461087
After [`create-table-mapping` command](#create-table-mapping-command) is executed, you can run this command to have the required UC catalogs and schemas created.
1047-
This command is supposed to be run before migrating tables to UC using [table migration workflow](#table-migration-workflow).
1088+
This command is supposed to be run before migrating tables to UC using [table migration process](#Table-Migration).
10481089

10491090
[[back to top](#databricks-labs-ucx)]
10501091

@@ -1056,7 +1097,7 @@ databricks labs ucx migrate-tables
10561097

10571098
Anytime after [`create-table-mapping` command](#create-table-mapping-command) is executed, you can run this command.
10581099

1059-
This command kicks off the [table migration](#table-migration-workflow) process. It triggers the `migrate-tables` workflow,
1100+
This command kicks off the [table migration](#Table-Migration) process. It triggers the `migrate-tables` workflow,
10601101
and if there are HiveSerDe tables detected, prompt whether to trigger the `migrate-external-hiveserde-tables-in-place-experimental` workflow.
10611102

10621103
[[back to top](#databricks-labs-ucx)]
@@ -1069,7 +1110,7 @@ databricks labs ucx revert-migrated-tables --schema X --table Y [--delete-manage
10691110

10701111
Anytime after [`create-table-mapping` command](#create-table-mapping-command) is executed, you can run this command.
10711112

1072-
This command removes the `upgraded_from` property on a migrated table for re-migration in the [table migration](#table-migration-workflow) process.
1113+
This command removes the `upgraded_from` property on a migrated table for re-migration in the [table migration](#Table-Migration) process.
10731114
This command is useful for developers and administrators who want to revert the migration of a table. It can also be used
10741115
to debug issues related to table migration.
10751116

@@ -1084,7 +1125,7 @@ databricks labs ucx move --from-catalog A --from-schema B --from-table C --to-ca
10841125
```
10851126

10861127
This command moves a UC table/tables from one schema to another schema after
1087-
the [table migration](#table-migration-workflow) process. This is useful for developers and administrators who want
1128+
the [table migration](#Table-Migration) process. This is useful for developers and administrators who want
10881129
to adjust their catalog structure after tables upgrade.
10891130

10901131
Users will be prompted whether tables/view are dropped after moving to new schema. This only applies to `MANAGED` tables and views.
@@ -1116,7 +1157,7 @@ It can also be used to debug issues related to table aliasing.
11161157

11171158
See the [migration process diagram](#migration-process) to understand the role of the code migration commands in the migration process.
11181159

1119-
After you're done with the [table migration](#table-migration-workflow), you can proceed to the code migration.
1160+
After you're done with the [table migration](#Table-Migration), you can proceed to the code migration.
11201161

11211162
Once you're done with the code migration, you can run the [`cluster-remap` command](#cluster-remap-command) to remap the
11221163
clusters to be UC compatible.
@@ -1148,7 +1189,7 @@ With modern IDEs, clicking on the file link opens the file at the problematic li
11481189
databricks labs ucx migrate-local-code
11491190
```
11501191

1151-
**(Experimental)** Once [table migration](#table-migration-workflow) is complete, you can run this command to
1192+
**(Experimental)** Once [table migration](#Table-Migration) is complete, you can run this command to
11521193
migrate all python and SQL files in the current working directory. This command is highly experimental and
11531194
at the moment only supports Python and SQL files and discards code comments and formatting during
11541195
the automated transformation process.
@@ -1161,7 +1202,7 @@ the automated transformation process.
11611202
databricks labs ucx migrate-dbsql-dashboards [--dashboard-id <dashboard-id>]
11621203
```
11631204

1164-
**(Experimental)** Once [table migration](#table-migration-workflow) is complete, you can run this command to
1205+
**(Experimental)** Once [table migration](#Table-Migration) is complete, you can run this command to
11651206
migrate all Databricks SQL dashboards in the workspace. At this moment, this command is highly experimental and discards
11661207
formatting during the automated transformation process.
11671208

@@ -1216,7 +1257,7 @@ databricks labs ucx sync-workspace-info
12161257

12171258
**Requires Databricks Account Administrator privileges.** This command uploads the workspace config to all workspaces
12181259
in the account where `ucx` is installed. This command is necessary to create an immutable default catalog mapping for
1219-
[table migration](#table-migration-workflow) process and is the prerequisite
1260+
[table migration](#Table-Migration) process and is the prerequisite
12201261
for [`create-table-mapping` command](#create-table-mapping-command).
12211262

12221263
If you cannot get account administrator privileges in reasonable time, you can take the risk and
@@ -1287,7 +1328,7 @@ This command is useful for administrators who want to ensure that the groups hav
12871328
used to debug issues related to group membership. See [group migration](docs/local-group-migration.md) and
12881329
[group migration](#group-migration-workflow) for more details.
12891330

1290-
Valid group membership is important to ensure users has correct access after legacy table ACL is migrated in [table migration workflow](#table-migration-workflow)
1331+
Valid group membership is important to ensure users has correct access after legacy table ACL is migrated in [table migration process](#Table-Migration)
12911332

12921333
[[back to top](#databricks-labs-ucx)]
12931334

0 commit comments

Comments
 (0)