Skip to content

[NEP-19579] set default catalog values on duplication #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

[NEP-19579] set default catalog values on duplication #45

wants to merge 4 commits into from

Conversation

jbat
Copy link
Member

@jbat jbat commented Apr 28, 2025 •
edited
Loading

https://jobready.atlassian.net/browse/NEP-19579

Relates to database catalogs support being added apache/superset#28416
Fixes an issue on Transfer and Duplicate dashboards relating to a new attribute catalog on datasets.

Both Transfer and Duplicate now update the catalog to the target database catalog.

NOTE: Both SS Staging and SS Prod envs have both been updated to SS 5.0.0.
We can now proceed to test this branch changes.
( now is un-blocked by https://jobready.atlassian.net/browse/NEP-19161 currently )

Staging Env Test .. on User Audit Board

# example of dashboard duplicate
Superset::Services::DuplicateDashboard.new(source_dashboard_id: 986, target_schema: 'aimbig_stage', target_database_id: 7 ).perform
=> {:new_dashboard_id=>992, 
       :new_dashboard_url=>"https://staging.ready-superset.jobready.io/superset/dashboard/992/",
       :published=>false}


Superset::Dashboard::Datasets::List.new(dashboard_id: 992, include_filter_datasets: true).list
+------+------------------------------------------------------+----------+------------------+------------------+--------------+-------------+
|                                                        992 Jobseeker Visits Audit                                                         |
+------+------------------------------------------------------+----------+------------------+------------------+--------------+-------------+
| Id   | Datasource name                                      | Database | Database name    | Database backend | Schema       | Filter only |
+------+------------------------------------------------------+----------+------------------+------------------+--------------+-------------+
| 2475 | Jobseeker Audit-aimbig_stage                         | 7        | Jobready-Staging | postgresql       | aimbig_stage |             |
| 2476 | Jobseeker Audit Full Details Past Month-aimbig_stage | 7        | Jobready-Staging | postgresql       | aimbig_stage |             |
+------+------------------------------------------------------+----------+------------------+------------------+--------------+-------------+
=> nil

(ENV:STAGING)> Superset::Dataset::Get.new(2475).result['catalog']
D, [2025-09-01T20:29:29.442615 #12788] DEBUG -- : Happi: GET https://staging.ready-superset.jobready.io/api/v1/dataset/2470, {}
=> "neptune_stagingapp01_dlm_sensitive"

(ENV:STAGING)> Superset::Dataset::Get.new(2476).result['catalog']
D, [2025-09-01T20:29:35.102821 #12788] DEBUG -- : Happi: GET https://staging.ready-superset.jobready.io/api/v1/dataset/2471, {}
=> "neptune_stagingapp01_dlm_sensitive"

Copy Board from staging to production

Following https://github.com/rdytech/superset-client/blob/develop/doc/migrating_dashboards_across_environments.md

# on ss PROD api
# import to jobready_prod1
file = '/Users/jonathon.batson/proj/superset-acumania/dashboard-backups/au/staging/jobready/986/dashboard_986_export_20250910.zip'

prod_p1_db_config = '/Users/jonathon.batson/proj/superset-acumania/database-backups/au/production/1/databases/JobReady-P1.yaml'

new_import_zip = Superset::Services::ImportDashboardAcrossEnvironments.new(
  dashboard_export_zip:      file,
  target_database_yaml_file: prod_p1_db_config,
  target_database_schema:    'jobready_prod1',
  ).perform

Superset::Dashboard::Import.new(source_zip_file: new_import_zip).perform
=> {Ok}
 Superset::Dashboard::Datasets::List.new(dashboard_id: 422).list
D, [2025-09-10T21:38:23.299273 #99850] DEBUG -- : Happi: GET https://ready-superset.jobready.io/api/v1/dashboard/422, {}
D, [2025-09-10T21:38:23.806577 #99850] DEBUG -- : Happi: GET https://ready-superset.jobready.io/api/v1/dashboard/422/datasets, {}
+------+-----------------------------------------+----------+---------------+------------------+----------------+-------------+
|                                                   422 User Activity Audit                                                   |
+------+-----------------------------------------+----------+---------------+------------------+----------------+-------------+
| Id   | Datasource name                         | Database | Database name | Database backend | Schema         | Filter only |
+------+-----------------------------------------+----------+---------------+------------------+----------------+-------------+
| 1722 | Jobseeker Audit Full Details Past Month | 1        | JobReady-P1   | postgresql       | jobready_prod1 |             |
| 1721 | Jobseeker Audit                         | 1        | JobReady-P1   | postgresql       | jobready_prod1 |             |
+------+-----------------------------------------+----------+---------------+------------------+----------------+-------------+

# COPY 422 board TO ATWORK SCHEMA
Superset::Services::DuplicateDashboard.new(source_dashboard_id: 422, target_schema: 'atwork', target_database_id: 1 ).perform
=> {:new_dashboard_id=>423, 
       :new_dashboard_url=>"https://ready-superset.jobready.io/superset/dashboard/423/", 
       :published=>false}

# COPY 422 board TO COACT SCHEMA
# FAILS ... DATASETS STILL POINT TO DATABASE CATALOG 1 ???
Superset::Services::DuplicateDashboard.new(source_dashboard_id: 422, target_schema: 'coact', target_database_id: 2 ).perform

jbat
jbat commented Apr 28, 2025
View reviewed changes
lib/superset/services/import_dashboard_across_environment.rb
@@ -78,6 +80,10 @@ def update_dataset_configs
dashboard_config[:datasets].each do |dataset|
dataset[:content][:database_uuid] = dashboard_config[:databases].first[:content][:uuid]
dataset[:content][:schema] = target_database_schema

# by clearing out the source's catalog, this automatically allows superset to set to the default catalog on the target
dataset[:content][:catalog] = nil
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note .. at RDY we do not use catalogs atm for high level namespace architecture
so for our purposes currently nil is fine.

@jbat jbat mentioned this pull request Apr 28, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jbat