Skip to content

Commit f3886f7

Browse files
FastLeeFastLee
authored
Ported internal document about UCX persistence schema (#345)
Generated the table persistence document in docs
1 parent f1db173 commit f3886f7

File tree

1 file changed

+120
-0
lines changed

1 file changed

+120
-0
lines changed

‎docs/table_persistence.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# UCX Persistence
2+
3+
Enumeration of all UCX persistence elements
4+
5+
## Overview
6+
Table Utilization:
7+
8+
| Table | Generate Assessment | Migrate Local Groups | Migrate External Tables | Migrate SQL Warehouses | Upgrade Jobs | Migrate managed tables |
9+
|--------------------|---------------------|----------------------|-------------------------|------------------------|--------------|------------------------|
10+
| tables | RW | | RO | | | RO |
11+
| grants | | | RW | | | RW |
12+
| mounts | RW | | RO | | RO | RO |
13+
| permissions | | RW | | | | |
14+
| jobs | RW | | | | RO | |
15+
| clusters | RW |
16+
| external_locations | RW | | RO |
17+
| workspace | RW | RO | | | RO |
18+
19+
**RW** - Read/Write the job that generates the table
20+
**RO** - Read Only
21+
22+
### Inventory Database
23+
#### _$inventory_.tables
24+
25+
Holds Inventory of all tables in all databases and their relevant metadata.
26+
27+
| Column | Datatype | Description | Comments |
28+
|-----------|----------|-------------|----------|
29+
| catalog | string | Original catalog of the table. _hive_metastore_ by default |
30+
| database | string | Original schema of the table |
31+
| name |string|Name of the table|
32+
|object_type|string|MANAGED, EXTERNAL, or VIEW|
33+
|table_format|string|Table provider. Like delta or json or parquet.|
34+
|location|string|Location of the data for table|
35+
|view_text|nullable string|If the table is the view, then this column holds the definition of the view|
36+
37+
<br/>
38+
39+
#### _$inventory_.table_failures
40+
Holds failures that occurred during crawling HMS tables
41+
42+
| Column | Datatype | Description | Comments |
43+
|-----------|----------|-------------|----------|
44+
|catalog|string|Original catalog of the table. hive_metastore by default|
45+
|database|string|Original schema of the table|
46+
|name|string|Name of the table|
47+
|failures|string|Exception message context|
48+
49+
<br/>
50+
51+
#### _$inventory_.grants
52+
Inventory of all Table ACLs for tables indexed in tables table.
53+
54+
| Column | Datatype | Description | Comments |
55+
|-----------|----------|-------------|----------|
56+
|principal|string|User name, group name, or service principal name|
57+
|action_type|string|Name of GRANT action|
58+
|catalog|string|Original catalog of the table. hive_metastore by default|
59+
|database|Nullable string|Original schema of the table|
60+
|table|Nullable string|Name of the table|
61+
|view|Nullable string|Name of the view|
62+
|any_file|bool|Any file|
63+
|anonymous_function|string|Grant for the anonymous function|
64+
65+
<br/>
66+
67+
#### _$inventory_.mounts
68+
List of DBFS mount points.
69+
70+
| Column | Datatype | Description | Comments |
71+
|-----------|----------|-------------|----------|
72+
|name|string|Name of the mount point|
73+
|source|string|Location of the backing dataset|
74+
|instance_profile|Nullable string|This mount point is accessible only with this AWS IAM instance profile|
75+
76+
<br/>
77+
78+
#### _$inventory_.permissions
79+
Workspace object level permissions
80+
81+
| Column | Datatype | Description | Comments |
82+
|-----------|----------|-------------|----------|
83+
|object_id|string|Either:<br/>Group ID<br/>Workspace Object ID<br/>Redash Object ID<br/>Scope name
84+
|supports|string|One of:<br/>AUTHORIZATION<br/><br/>CLUSTERS<br/>CLUSTER_POLICIES<br/>DIRECTORIES<br/>EXPERIMENTS<br/>FILES<br/>INSTANCE_POOLS<br/>JOBS<br/>NOTEBOOKS<br/>PIPELINES<br/>REGISTERED_MODELS<br/>REPOS<br/>SERVING_ENDPOINTS<br/>SQL_WAREHOUSES
85+
|raw_object_permissions|JSON|JSON-serialized response of:<br/>Generic Permissions<br/>Secret ACL<br/>Group roles and entitlements<br/>Redash permissions|
86+
87+
<br/>
88+
89+
#### _$inventory_.jobs
90+
Holds a list of all jobs with a notation of potential issues.
91+
92+
| Column | Datatype | Description | Comments |
93+
|-----------|----------|-------------|----------|
94+
|job_id|string|Job ID|
95+
|job_name|string|Job Name|
96+
|job_creator|string|UserID of the Job Creator|
97+
|compatible|int|1 or 0, used for percentage reporting|
98+
|failures|string|List of issues identified by the assessment in JSON format|
99+
100+
101+
#### _$inventory_.clusters
102+
Holds a list of all clusters with a notation of potential issues.
103+
104+
| Column | Datatype | Description | Comments |
105+
|-----------|----------|-------------|----------|
106+
|cluster_id|string|Cluster Id|
107+
|cluster_name|string|Cluster Name|
108+
|cluster_creator|string|UserID of the Cluster Creator|
109+
|compatible|int|1 or 0, used for percentage reporting|
110+
|failures|string|List of issues identified by the assessment in JSON format|
111+
112+
113+
#### _$inventory_.external_locations
114+
Holds a list of all external locations that will be required for the migration.
115+
116+
| Column | Datatype | Description | Comments |
117+
|-----------|----------|-------------|----------|
118+
|external_location|string|External Location URL|
119+
120+

0 commit comments

Comments
 (0)