Skip to content

Conversation

@JCZuurmond
Copy link
Contributor

@JCZuurmond JCZuurmond commented Sep 26, 2024

Changes

Add and populate workflow runs table

Linked issues

Resolves #2600

Functionality

  • added relevant user documentation
  • modified existing workflow: migration-process-experimental

Tests

  • manually tested
  • added unit tests
  • added integration tests

TODO

  • Handle concurrent writes, see
  • Decide on getting workflow run status from parse_log_task --> only add it to the migration progress workflow for now

@JCZuurmond JCZuurmond added feat/workflow triggered as a Databricks Job managed by UCX feat/migration-progress Issues related to the migration progress workflow labels Sep 26, 2024
@JCZuurmond JCZuurmond self-assigned this Sep 26, 2024
@JCZuurmond
Copy link
Contributor Author

JCZuurmond commented Sep 26, 2024

  • TODO: Populate the workflows_runs table

@JCZuurmond JCZuurmond force-pushed the feat/add-workspace-scans-table branch from f5f3ac2 to cbaba94 Compare September 26, 2024 12:31
Copy link
Contributor Author

@JCZuurmond JCZuurmond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to hit submit on this review, maybe outdated

@JCZuurmond
Copy link
Contributor Author

Note that this task will fail when there is no UCX catalog with the multiworkspace schema and workflow_runs table. The following issue checks at the start of the assessment workflow if the catalog exists: #2577

@JCZuurmond
Copy link
Contributor Author

Note that this task will fail when there is no UCX catalog with the multiworkspace schema and workflow_runs table. The following issue checks at the start of the assessment workflow if the catalog exists: #2577

Added try-except with NotFound and log error. The log crawler will pick that up

@JCZuurmond JCZuurmond marked this pull request as ready for review September 27, 2024 10:18
@JCZuurmond JCZuurmond requested a review from a team as a code owner September 27, 2024 10:18
@JCZuurmond JCZuurmond changed the title Add and populate UCX workflow_runs artifact table Add and populate UCX workflow_runs table Sep 27, 2024
@JCZuurmond JCZuurmond force-pushed the feat/add-workspace-scans-table branch from affe04a to 9502991 Compare September 27, 2024 10:19
@github-actions
Copy link

github-actions bot commented Sep 27, 2024

❌ 108/109 passed, 1 failed, 3 skipped, 3h48m58s total

❌ test_running_real_migration_progress_job: AssertionError: assert False (24m9.909s)
AssertionError: assert False
 +  where False = validate_step('migration-progress-experimental')
 +    where validate_step = <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7fee81781c60>.validate_step
 +      where <databricks.labs.ucx.installer.workflows.DeployedWorkflows object at 0x7fee81781c60> = <tests.integration.conftest.MockInstallationContext object at 0x7fee805a0820>.deployed_workflows
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
12:34 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.GQbr/config.yml) doesn't exist.
12:34 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
12:34 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
12:34 INFO [databricks.labs.ucx.install] Fetching installations...
12:34 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
12:34 DEBUG [tests.integration.conftest] Waiting for clusters to start...
12:34 DEBUG [tests.integration.conftest] Waiting for clusters to start...
12:34 INFO [databricks.labs.ucx.install] Installing UCX v0.38.1+7220241002123441
12:34 INFO [databricks.labs.ucx.install] Creating ucx schemas...
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
12:34 INFO [databricks.labs.ucx.install] Creating dashboards...
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.GQbr/README for the next steps.
12:34 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/435388802390838
12:34 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/435388802390838/runs/309107610272846
12:34 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of assessment job: https://DATABRICKS_HOST#job/435388802390838/runs/309107610272846
12:54 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 309107610272846 with state: RunResultState.SUCCESS
12:54 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 309107610272846 duration: 0:19:31.848000 (2024-10-02 12:34:59.213000+00:00 thru 2024-10-02 12:54:31.061000+00:00)
12:54 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/435388802390838
12:54 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
12:54 DEBUG [databricks.labs.ucx.installer.workflows] starting migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621
12:54 INFO [databricks.labs.ucx.installer.workflows] Started migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621/runs/470621098601968
12:54 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621/runs/470621098601968
12:58 INFO [databricks.labs.ucx.installer.workflows] Completed migration-progress-experimental job run 470621098601968 with state: RunResultState.SUCCESS_WITH_FAILURES (The job run succeeded with 1 failed task)
12:58 INFO [databricks.labs.ucx.installer.workflows] Completed migration-progress-experimental job run 470621098601968 duration: 0:03:41.345000 (2024-10-02 12:54:45.435000+00:00 thru 2024-10-02 12:58:26.780000+00:00)
12:58 DEBUG [databricks.labs.ucx.installer.workflows] Validating migration-progress-experimental workflow: https://DATABRICKS_HOST#job/280165764390621
12:34 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.GQbr/config.yml) doesn't exist.
12:34 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
12:34 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
12:34 INFO [databricks.labs.ucx.install] Fetching installations...
12:34 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
12:34 DEBUG [tests.integration.conftest] Waiting for clusters to start...
12:34 DEBUG [tests.integration.conftest] Waiting for clusters to start...
12:34 INFO [databricks.labs.ucx.install] Installing UCX v0.38.1+7220241002123441
12:34 INFO [databricks.labs.ucx.install] Creating ucx schemas...
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
12:34 INFO [databricks.labs.ucx.install] Creating dashboards...
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
12:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
12:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
12:34 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.GQbr/README for the next steps.
12:34 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/435388802390838
12:34 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/435388802390838/runs/309107610272846
12:34 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of assessment job: https://DATABRICKS_HOST#job/435388802390838/runs/309107610272846
12:54 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 309107610272846 with state: RunResultState.SUCCESS
12:54 INFO [databricks.labs.ucx.installer.workflows] Completed assessment job run 309107610272846 duration: 0:19:31.848000 (2024-10-02 12:34:59.213000+00:00 thru 2024-10-02 12:54:31.061000+00:00)
12:54 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/435388802390838
12:54 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
12:54 DEBUG [databricks.labs.ucx.installer.workflows] starting migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621
12:54 INFO [databricks.labs.ucx.installer.workflows] Started migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621/runs/470621098601968
12:54 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of migration-progress-experimental job: https://DATABRICKS_HOST#job/280165764390621/runs/470621098601968
12:58 INFO [databricks.labs.ucx.installer.workflows] Completed migration-progress-experimental job run 470621098601968 with state: RunResultState.SUCCESS_WITH_FAILURES (The job run succeeded with 1 failed task)
12:58 INFO [databricks.labs.ucx.installer.workflows] Completed migration-progress-experimental job run 470621098601968 duration: 0:03:41.345000 (2024-10-02 12:54:45.435000+00:00 thru 2024-10-02 12:58:26.780000+00:00)
12:58 DEBUG [databricks.labs.ucx.installer.workflows] Validating migration-progress-experimental workflow: https://DATABRICKS_HOST#job/280165764390621
12:58 INFO [databricks.labs.ucx.install] Deleting UCX v0.38.1+7220241002123441 from https://DATABRICKS_HOST
12:58 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_s5skr
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=770524603642148, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1115588870294615, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=324668988215569, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=435388802390838, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=280165764390621, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=114289376397196, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1012092358731991, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=628589497548849, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=6584618534011, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=484530241945676, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=213703251823109, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1021419190635484, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=334553193075351, as it is no longer needed
12:58 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=616081349347128, as it is no longer needed
12:58 INFO [databricks.labs.ucx.install] Deleting cluster policy
12:58 INFO [databricks.labs.ucx.install] Deleting secret scope
12:58 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

Running from acceptance #6259

Copy link
Contributor

@asnare asnare left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment it seems to me that some of the requirements (or purpose) of this table aren't clear. To avoid rework I think this needs to be resolved before we can complete this PR.

Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@JCZuurmond JCZuurmond requested a review from asnare October 2, 2024 10:05
@JCZuurmond
Copy link
Contributor Author

Integration test fails due to a time out on the assessment job. This is happening also for the assessment integration test, but went unnoticed as the CI is not able to create issues anymore. This is blocking the PR at the moment

@nfx nfx merged commit cb40b60 into main Oct 2, 2024
5 of 6 checks passed
@nfx nfx deleted the feat/add-workspace-scans-table branch October 2, 2024 15:11
@nfx nfx mentioned this pull request Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat/migration-progress Issues related to the migration progress workflow feat/workflow triggered as a Databricks Job managed by UCX