Merge pull request #37 from rdytech/NEP-18548-import-dashboard-across-envs

NEP-18548 Transfer Dashboard across Envs

Author: jbat

CHANGELOG.md

@@ -1,5 +1,9 @@
 ## Change Log
 
+## 0.2.2 - 2024-10-10
+
+* add ImportDashboardAcrossEnvironments class for transfering between superset environments
+
 ## 0.2.1 - 2024-09-17
 
 * add Superset::Database::Export class for exporting database configurations

Gemfile.lock

@@ -14,7 +14,7 @@ GIT
 PATH
   remote: .
   specs:
-    superset (0.2.1)
+    superset (0.2.2)
       dotenv (~> 2.7)
       enumerate_it (~> 1.7.0)
       faraday (~> 1.0)

README.md

@@ -66,12 +66,17 @@ More examples [listed here](https://github.com/rdytech/superset-client/tree/deve
 
 ## Duplicating Dashboards
 
-The Primary motivation behind this gem was to use the Superset API to duplicate dashboards, charts, datasets across multiple database connections.  
+One Primary motivation behind this gem was to use the Superset API to duplicate dashboards, charts, datasets across multiple database connections.
 
 Targeted use case was for superset embedded functionality implemented in a application resting on multi tenanted database setup.
 
 See examples in [Duplicate Dashboards](https://github.com/rdytech/superset-client/tree/develop/doc/duplicate_dashboards.md)
 
+## Moving / Transferring Dashboards across Environments
+
+With a few configuration changes to an import file, the process can be codified to transfer a dashboard between environments.
+
+See example in [Transferring Dashboards across Environments](https://github.com/rdytech/superset-client/tree/develop/doc/migrating_dashboards_across_environments.md)
 
 ## Contributing
 

doc/migrating_dashboards_across_environments.md

@@ -0,0 +1,173 @@
+# Transferring Dashboards across Environments
+
+In this document, we will discuss how to transfer dashboards across Superset hosting environments with the goal of heading towards an API call to automate the process.
+
+Current process is limited to dashboards with all datasets based on a single database connection.
+
+## Short Version
+
+Assuming you want to transfer a dashboard from Env1 to Env2.
+
+You will need the following:
+- a Env1 Dashboard Export Zip file
+- a Env2 Database config export yaml
+- a Env2 schema to point your datasets to
+
+Assuming your API env for ruby is setup for your target superset environment.
+( ie using API creds for Env2 for this example )
+
+```ruby
+
+new_import_zip = Superset::Services::ImportDashboardAcrossEnvironments.new(
+  dashboard_export_zip:      'path_to/dashboard_101_export_20241010.zip',
+  target_database_yaml_file: 'path_to/env2_db_config.yaml', 
+  target_database_schema:    'acme', 
+  ).perform
+
+# now import the adjusted zip to the target superset env
+Superset::Dashboard::Import.new(source_zip_file: new_import_file).perform
+
+```
+
+## Background
+
+A common practice is to set up infrastructure to deploy multiple Superset environments. For example, a simple setup might be:
+- Local development environment for testing version upgrades and feature exploration
+- Staging Superset environment for testing in a production-like environment
+- Production Superset environment that requires a higher level of stability and uptime
+
+For the above example, the Superset staging environment often holds connections to staging databases, and the Superset production environment will hold connections to the production databases.
+
+In the event where the database schema structure for the local development, staging, and production databases are exactly the same, dashboards can be replicated and transferred across Superset hosting environments.
+
+That process does require some manual updating of the exported YAML files before importing them into the target environment. Also required is some understanding of the underlying dashboard export structure and how the object UUIDs work and relate to each other, especially in the context of databases and datasets.
+
+## Dashboard Export/Import within the Same Environment
+
+This is a fairly straightforward process.
+
+There are multiple methods for exporting a dashboard:
+- Export from the dashboard list page in the GUI
+- Export via the Superset API
+- Export via the Superset CLI
+
+Each export method will result in a zip file that contains a set of YAML files as per this list below, which is an export of customized version of the test Sales dashboard from the default example dashboards.
+
+Test fixture is: https://github.com/rdytech/superset-client/blob/develop/spec/fixtures/dashboard_18_export_20240322.zip
+
+```
+└── dashboard_export_20240321T214117
+    ├── charts
+    │   ├── Boy_Name_Cloud_53920.yaml
+    │   ├── Names_Sorted_by_Num_in_California_53929.yaml
+    │   ├── Number_of_Girls_53930.yaml
+    │   ├── Pivot_Table_53931.yaml
+    │   └── Top_10_Girl_Name_Share_53921.yaml
+    ├── dashboards
+    │   └── Birth_Names_18.yaml
+    ├── databases
+    │   └── examples.yaml
+    ├── datasets
+    │   └── examples
+    │       └── birth_names.yaml
+    └── metadata.yaml
+```
+
+Each of the above YAML files holds UUID values for the primary object and any related objects.
+
+- Database YAMLs hold the database connection string as well as a UUID for the database
+- Dataset YAMLs have their own UUID as well as a reference to the database UUID
+- Chart YAMLs have their own UUID as well as a reference to their dataset UUID
+
+Example of the database YAML file:
+
+```
+cat databases/examples.yaml
+database_name: examples
+sqlalchemy_uri: postgresql+psycopg2://superset:XXXXXXXXXX@superset-host:5432/superset
+cache_timeout: null
+expose_in_sqllab: true
+allow_run_async: true
+allow_ctas: true
+allow_cvas: true
+allow_dml: true
+allow_file_upload: true
+extra:
+  metadata_params: {}
+  engine_params: {}
+  metadata_cache_timeout: {}
+  schemas_allowed_for_file_upload:
+  - examples
+  allows_virtual_table_explore: true
+uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+version: 1.0.0
+```
+
+If we grep the database/examples.yaml we can see the UUID of the database.
+
+```
+grep -r uuid databases/
+  databases/examples.yaml:uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+
+```
+
+Now if we look at the UUID values in the datasets, you will see both the dataset UUID and the reference to the database UUID.
+
+```
+grep -r uuid datasets
+datasets/examples/birth_names.yaml:uuid: 283f5023-0814-40f6-b12d-96f6a86b984f
+datasets/examples/birth_names.yaml:database_uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+```
+
+If the above dashboard zip file `dashboard_18_export_20240322.zip` was imported as is to the same superset environment as it was exported from, this would mean all UUID's would already exist in superset and these objects would be found and updated with the imported zip data.
+
+If the above zip file was imported as is to a different target Superset environment, it would fail as there would be no matching database UUID entry in that target Superset environment.
+
+**Key Point:** When importing a dashboard to a different Superset environment than the original environment, the database configuration in the zip export must exist in the target Superset environment and all datasets must point to the database config.
+
+## Migrate a Dashboard to a Different Superset Environment
+
+With the above knowledge, we can now think about how to migrate dashboards between Superset environments.
+
+Each Superset object is given a UUID. Within the exported dashboard files, we are primarily concerned with:
+- Replacing the staging database configuration with the production configuration
+- Updating all staging datasets to point to the new production database UUID
+
+Given we have a request to 'transfer' a dashboard across to a different environment, say staging to production, how would we then proceed?
+
+With the condition that the database in staging and production are structurally exactly the same schema, from the above discussion on UUIDs, you can then see that if we want to import a staging dashboard export into the production environment, we will need to perform the following steps:
+
+1. Export the staging dashboard and unzip
+2. Note the staging database UUIDs in the `databases/` directory
+3. Get a copy of the production database YAML configuration file
+4. In the exported dashboard files, replace the staging database YAML with the production YAML
+5. In the dataset YAML files, replace all instances of the previously noted staging database UUID with the new production UUID
+6. Zip the files and import them to the production environment
+
+The process above assumes that whoever is migrating the dashboard has a copy of the target database YAML files so that in steps 3 and 4 we can then replace the staging database YAML with the production one.
+
+## Requirements
+
+The overall process requires the following:
+- The source dashboard zip file
+- The target Superset environment database YAML file
+- Ability to copy and manipulate the source dashboard zip file
+- The ability to import via API to the target Superset environment
+
+
+## Gotchas!
+
+Migrating a dashboard once to a new target environment, database, schema will result in:
+- Creating a new dashboard with the UUID from the import zip
+- Creating a new set of charts with their UUIDs from the import zip
+- Creating a new set of datasets with their UUIDs from the import zip
+
+Migrating the same dashboard a second time to the same target environment, database, but different schema will NOT create a new dashboard.
+
+It will attempt to update the same dashboard as the UUID for the dashboard has not changed. It will also NOT change any of the datasets to the new schema. This appears to be a limitation of the import process, which may lead to some confusing results.
+
+## References
+
+Some helpful references relating to cross-environment workflows:
+- [Managing Content Across Workspaces](https://docs.preset.io/docs/managing-content-across-workspaces)
+- [Superset Slack AI Explanation](https://apache-superset.slack.com/archives/C072KSLBTC1/p1722382347022689)

lib/superset/dashboard/import.rb

@@ -20,7 +20,6 @@
 module Superset
   module Dashboard
     class Import < Request
-
       attr_reader :source_zip_file, :overwrite
 
       def initialize(source_zip_file: , overwrite: true)
@@ -30,7 +29,6 @@ def initialize(source_zip_file: , overwrite: true)
 
       def perform
         validate_params
-
         response
       end
 
@@ -46,7 +44,9 @@ def response
       def validate_params
         raise ArgumentError, 'source_zip_file is required' if source_zip_file.nil?
         raise ArgumentError, 'source_zip_file does not exist' unless File.exist?(source_zip_file)
+        raise ArgumentError, 'source_zip_file is not a zip file' unless File.extname(source_zip_file) == '.zip'
         raise ArgumentError, 'overwrite must be a boolean' unless [true, false].include?(overwrite)
+        raise ArgumentError, "zip target database does not exist: #{zip_database_config_not_found_in_superset}" if zip_database_config_not_found_in_superset.present?
       end
 
       def payload
@@ -59,6 +59,26 @@ def payload
       def route
         "dashboard/import/"
       end
+
+      def zip_database_config_not_found_in_superset
+        zip_databases_details.select {|s| !superset_database_uuids_found.include?(s[:uuid]) }
+      end
+
+      def superset_database_uuids_found
+        @superset_database_uuids_found ||= begin
+          zip_databases_details.map {|i| i[:uuid]}.map do |uuid|
+            uuid if Superset::Database::List.new(uuid_equals: uuid).result.present?
+          end.compact
+        end
+      end
+
+      def zip_databases_details
+        zip_dashboard_config[:databases].map{|d| {uuid: d[:content][:uuid], name: d[:content][:database_name]} }
+      end
+
+      def zip_dashboard_config
+        @zip_dashboard_config ||= Superset::Services::DashboardLoader.new(dashboard_export_zip: source_zip_file).perform
+      end
     end
   end
 end

lib/superset/database/list.rb

@@ -4,10 +4,11 @@
 module Superset
   module Database
     class List < Superset::Request
-      attr_reader :title_contains
+      attr_reader :title_contains, :uuid_equals
 
-      def initialize(page_num: 0, title_contains: '')
+      def initialize(page_num: 0, title_contains: '', uuid_equals: '')
         @title_contains = title_contains
+        @uuid_equals = uuid_equals
         super(page_num: page_num)
       end
 
@@ -34,6 +35,7 @@ def filters
         # TODO filtering across all list classes can be refactored to support multiple options in a more flexible way
         filter_set = []
         filter_set << "(col:database_name,opr:ct,value:'#{title_contains}')" if title_contains.present?
+        filter_set << "(col:uuid,opr:eq,value:'#{uuid_equals}')" if uuid_equals.present?
         unless filter_set.empty?
           "filters:!(" + filter_set.join(',') + "),"
         end
@@ -45,6 +47,7 @@ def list_attributes
 
       def validate_constructor_args
         raise InvalidParameterError, "title_contains must be a String type" unless title_contains.is_a?(String)
+        raise InvalidParameterError, "uuid_equals must be a String type" unless uuid_equals.is_a?(String)
       end
     end
   end

lib/superset/request.rb

@@ -5,7 +5,6 @@ class Request
     class InvalidParameterError < StandardError; end
     class ValidationError < StandardError; end
 
-
     PAGE_SIZE = 100
 
     attr_accessor :page_num

lib/superset/services/dashboard_loader.rb

@@ -0,0 +1,69 @@
+# Given a path, load all yaml files
+
+require 'superset/file_utilities'
+require 'yaml'
+
+module Superset
+  module Services
+    class DashboardLoader
+      include FileUtilities
+
+      TMP_PATH = '/tmp/superset_dashboard_imports'.freeze
+
+      attr_reader :dashboard_export_zip
+
+      def initialize(dashboard_export_zip:)
+        @dashboard_export_zip = dashboard_export_zip
+      end
+
+      def perform
+        unzip_source_file
+        dashboard_config
+      end
+
+      def dashboard_config
+        @dashboard_config ||= DashboardConfig.new(
+                                dashboard_export_zip:    dashboard_export_zip, 
+                                tmp_uniq_dashboard_path: tmp_uniq_dashboard_path).config
+      end
+
+      private
+
+      def unzip_source_file
+        @extracted_files = unzip_file(dashboard_export_zip, tmp_uniq_dashboard_path)
+      end
+
+      def tmp_uniq_dashboard_path
+        @tmp_uniq_dashboard_path ||= File.join(TMP_PATH, uuid)
+      end
+
+      def uuid
+        SecureRandom.uuid
+      end
+
+      class DashboardConfig < ::OpenStruct
+        def config
+            {
+              tmp_uniq_dashboard_path: tmp_uniq_dashboard_path,
+              dashboards: load_yamls_for('dashboards'),
+              databases:  load_yamls_for('databases'),
+              datasets:   load_yamls_for('datasets'),
+              charts:     load_yamls_for('charts'),
+              metadata:   load_yamls_for('metadata.yaml', pattern_sufix: nil),
+            }
+        end
+
+        def load_yamls_for(object_path, pattern_sufix: '**/*.yaml')
+          pattern = File.join([tmp_uniq_dashboard_path, '**', object_path, pattern_sufix].compact)
+          Dir.glob(pattern).map do |file|
+            { filename: file, content: load_yaml_and_symbolize_keys(file) } if File.file?(file)
+          end.compact
+        end
+
+        def load_yaml_and_symbolize_keys(path)
+          YAML.load_file(path).deep_symbolize_keys
+        end
+      end
+    end
+  end
+end