Skip to content

Commit d2048dc

Browse files
asnareasnare
authored
Improve group migration error reporting (#2344)
## Changes This PR updates the dashboard for group migration: - The documentation widgets have been adjusted. - The failed-migration widget now displays formatted information about the failure, and links to the failed job run. - Only failures from the latest workflow run (with logs) are displayed. ### Linked issues Improves upon #2333. Relates to #1914. ### Functionality - Updated dashboard: _UCX Migration (Groups)_ ### Tests - [x] manually tested - [x] verified on staging environment (screenshot attached)
1 parent cdb7709 commit d2048dc

File tree

3 files changed

+76
-16
lines changed

3 files changed

+76
-16
lines changed

‎src/databricks/labs/ucx/queries/migration/groups/00_0_migration_overview.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,15 @@ Workspace Group Migration
1010
Prerequisites for group migration are:
1111

1212
- The assessment workflow has completed.
13-
- For each workspace group a corresponding account-level group must exist.
13+
- For each workspace group a corresponding account-level group must exist. (Refer to the documentation for details about the different strategies that can be configured for associating workspace groups with their corresponding account-level group.)
1414

1515
Group migration performs the following steps:
1616

1717
1. Workspace groups are renamed to a temporary name.
1818
2. The account-level groups are then provisioned into the workspace.
1919
3. Permissions assigned to the workspace groups are replicated on the account-level groups.
2020

21-
These steps are performed by either the `migrate-groups` or the `migrate-groups-experimental` workflows. (The latter uses a faster, but experimental, method to replicate group permissions.)
21+
These steps are performed by either the `migrate-groups` or the `migrate-groups-experimental` [workflows](/jobs). (The latter uses a faster, but experimental, method to replicate group permissions.)
2222

2323
Once group migration has taken place:
2424

‎src/databricks/labs/ucx/queries/migration/groups/01_0_group_migration_failures.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,9 @@
11
---
22
height: 4
3-
width: 2
3+
width: 1
44
---
55

66
Group Migration Failures
77
------------------------
88

99
When migrating groups, errors may occur during the migration of a particular group. Any errors that occurred while migrating groups are displayed here.
10-
11-
The errors for all workflow runs are shown; to clear (old) errors the job run can be deleted.

‎src/databricks/labs/ucx/queries/migration/groups/01_1_group_migration_failures.sql‎

Lines changed: 73 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,77 @@
1-
/* --title 'Group Migration Failures' --height 4 --width 4 */ /*
2-
*
3-
* Messages that we're looking for are of the form:
4-
* failed-group-migration: {name_in_workspace} -> {name_in_account}: {reason}
5-
*/
1+
/*
2+
--title 'Group Migration Failures'
3+
--height 4
4+
--width 5
5+
--overrides '{"spec":{
6+
"encodings":{
7+
"columns": [
8+
{"fieldName": "timestamp", "title": "Time of Failure", "booleanValues": ["false", "true"], "type": "datetime", "displayAs": "datetime", "dateTimeFormat": "ll LTS (z)"},
9+
{"fieldName": "task_name", "title": "Workflow/Task", "booleanValues": ["false", "true"], "type": "string", "displayAs": "link", "linkUrlTemplate": "/jobs/{{ job_id }}", "linkTextTemplate": "{{ workflow_name }}/{{ @ }}", "linkTitleTemplate": "{{ workflow_name }} (ID: {{ job_id }})", "linkOpenInNewTab": true, "useMonospaceFont": true},
10+
{"fieldName": "job_run_id", "title": "Run ID", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "link", "linkUrlTemplate": "/jobs/{{ job_id }}/runs/{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "Job Run: {{ @ }}", "linkOpenInNewTab": true},
11+
{"fieldName": "workspace_group", "title": "Workspace Group", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string"},
12+
{"fieldName": "account_group", "title": "Account Group", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string"},
13+
{"fieldName": "error_message", "title": "Error Message", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string"}
14+
]},
15+
"invisibleColumns": [
16+
{"name": "job_id", "title": "Workflow", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "link", "linkUrlTemplate": "/jobs/{{ job_id }}", "linkTextTemplate": "{{ workflow_name }}", "linkTitleTemplate": "{{ workflow_name }} (ID: {{ job_id }})", "useMonospaceFont": true},
17+
{"name": "workflow_name", "title": "Workflow", "booleanValues": ["false", "true"], "type": "string", "displayAs": "link", "linkUrlTemplate": "/jobs/{{ job_id }}", "linkTextTemplate": "{{ workflow_name }}", "linkTitleTemplate": "{{ workflow_name }} (ID: {{ job_id }})", "useMonospaceFont": true}
18+
]
19+
}}'
20+
*/
21+
WITH latest_job_runs AS (
22+
SELECT
23+
timestamp,
24+
job_id,
25+
job_run_id
26+
FROM (
27+
SELECT
28+
CAST(timestamp AS TIMESTAMP) AS timestamp,
29+
job_id,
30+
job_run_id,
31+
ROW_NUMBER() OVER (PARTITION BY job_id ORDER BY CAST(timestamp AS TIMESTAMP) DESC) = 1 AS latest_run_of_job
32+
FROM inventory.logs
33+
)
34+
WHERE
35+
latest_run_of_job
36+
), logs_latest_job_runs AS (
37+
SELECT
38+
CAST(logs.timestamp AS TIMESTAMP) AS timestamp,
39+
message,
40+
job_run_id,
41+
job_id,
42+
workflow_name,
43+
task_name
44+
FROM inventory.logs
45+
JOIN latest_job_runs
46+
USING (job_id, job_run_id)
47+
WHERE
48+
workflow_name IN ('migrate-groups', 'migrate-groups-experimental')
49+
), migration_failures AS (
50+
/* Messages that we're looking for are of the form:
51+
* failed-group-migration: {name_in_workspace} -> {name_in_account}: {reason}
52+
*/
53+
SELECT
54+
timestamp,
55+
REGEXP_EXTRACT(message, '^failed-group-migration: (.+?) -> (.+?): (.+)$', 1) AS workspace_group,
56+
REGEXP_EXTRACT(message, '^failed-group-migration: (.+?) -> (.+?): (.+)$', 2) AS account_group,
57+
REGEXP_EXTRACT(message, '^failed-group-migration: (.+?) -> (.+?): (.+)$', 3) AS error_message,
58+
job_run_id,
59+
job_id,
60+
workflow_name,
61+
task_name
62+
FROM logs_latest_job_runs
63+
WHERE
64+
STARTSWITH(message, 'failed-group-migration: ')
65+
)
666
SELECT
7-
FROM_UNIXTIME(timestamp) AS timestamp,
67+
timestamp,
68+
workspace_group,
69+
account_group,
70+
error_message,
871
job_run_id,
9-
level,
10-
message
11-
FROM inventory.logs
12-
WHERE
13-
workflow_name IN ('migrate-groups', 'migrate-groups-experimental') AND STARTSWITH(message, 'failed-group-migration:')
72+
job_id,
73+
workflow_name,
74+
task_name
75+
FROM migration_failures
1476
ORDER BY
1577
1

0 commit comments

Comments
 (0)