NEP-18548 Add Dashboard Config Loader

Author: jbat

doc/migrating_dashboards_across_environments.md

@@ -0,0 +1,146 @@
+# Transferring Dashboards across Environments
+
+In this document, we will discuss how to transfer dashboards across Superset hosting environments with the goal of heading towards an API call to automate the process.
+
+## Background
+
+A common practice is to set up infrastructure to deploy multiple Superset environments. For example, a simple setup might be:
+- Local development environment for testing version upgrades and feature exploration
+- Staging Superset environment for testing in a production-like environment
+- Production Superset environment that requires a higher level of stability and uptime
+
+For the above example, the Superset staging environment often holds connections to staging databases, and the Superset production environment will hold connections to the production databases.
+
+In the event where the database schema structure for the local development, staging, and production databases are exactly the same, dashboards can be replicated and transferred across Superset hosting environments.
+
+That process does require some manual updating of the exported YAML files before importing them into the target environment. Also required is some understanding of the underlying dashboard export structure and how the object UUIDs work and relate to each other, especially in the context of databases and datasets.
+
+## Dashboard Export/Import within the Same Environment
+
+This is a fairly straightforward process.
+
+There are multiple methods for exporting a dashboard:
+- Export from the dashboard list page in the GUI
+- Export via the Superset API
+- Export via the Superset CLI
+
+Each export method will result in a zip file that contains a set of YAML files as per this list below, which is an export of customized version of the test Sales dashboard from the default example dashboards.
+
+Test fixture is: https://github.com/rdytech/superset-client/blob/develop/spec/fixtures/dashboard_18_export_20240322.zip
+
+```
+└── dashboard_export_20240321T214117
+    ├── charts
+    │   ├── Boy_Name_Cloud_53920.yaml
+    │   ├── Names_Sorted_by_Num_in_California_53929.yaml
+    │   ├── Number_of_Girls_53930.yaml
+    │   ├── Pivot_Table_53931.yaml
+    │   └── Top_10_Girl_Name_Share_53921.yaml
+    ├── dashboards
+    │   └── Birth_Names_18.yaml
+    ├── databases
+    │   └── examples.yaml
+    ├── datasets
+    │   └── examples
+    │       └── birth_names.yaml
+    └── metadata.yaml
+```
+
+Each of the above YAML files holds UUID values for the primary object and any related objects.
+
+- Database YAMLs hold the database connection string as well as a UUID for the database
+- Dataset YAMLs have their own UUID as well as a reference to the database UUID
+- Chart YAMLs have their own UUID as well as a reference to their dataset UUID
+
+Example of the database YAML file:
+
+```
+cat databases/examples.yaml
+database_name: examples
+sqlalchemy_uri: postgresql+psycopg2://superset:XXXXXXXXXX@superset-host:5432/superset
+cache_timeout: null
+expose_in_sqllab: true
+allow_run_async: true
+allow_ctas: true
+allow_cvas: true
+allow_dml: true
+allow_file_upload: true
+extra:
+  metadata_params: {}
+  engine_params: {}
+  metadata_cache_timeout: {}
+  schemas_allowed_for_file_upload:
+  - examples
+  allows_virtual_table_explore: true
+uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+version: 1.0.0
+```
+
+If we grep the database/examples.yaml we can see the UUID of the database.
+
+```
+grep -r uuid databases/
+  databases/examples.yaml:uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+
+```
+
+Now if we look at the UUID values in the datasets, you will see both the dataset UUID and the reference to the database UUID.
+
+```
+grep -r uuid datasets
+datasets/examples/birth_names.yaml:uuid: 283f5023-0814-40f6-b12d-96f6a86b984f
+datasets/examples/birth_names.yaml:database_uuid: a2dc77af-e654-49bb-b321-40f6b559a1ee
+```
+
+If the above dashboard zip file `dashboard_18_export_20240322.zip` was imported as is to the same superset environment as it was exported from, this would mean all UUID's would already exist in superset and these objects would be found and updated with the imported zip data.
+
+If the above zip file was imported as is to a different target Superset environment, it would fail as there would be no matching database UUID entry in that target Superset environment.
+
+**Key Point:** When importing a dashboard to a different Superset environment than the original environment, the database configuration in the zip export must exist in the target Superset environment and all datasets must point to the database config.
+
+## Migrate a Dashboard to a Different Superset Environment
+
+With the above knowledge, we can now think about how to migrate dashboards between Superset environments.
+
+Each Superset object is given a UUID. Within the exported dashboard files, we are primarily concerned with:
+- Replacing the staging database configuration with the production configuration
+- Updating all staging datasets to point to the new production database UUID
+
+Given we have a request to 'transfer' a dashboard across to a different environment, say staging to production, how would we then proceed?
+
+With the condition that the database in staging and production are structurally exactly the same schema, from the above discussion on UUIDs, you can then see that if we want to import a staging dashboard export into the production environment, we will need to perform the following steps:
+
+1. Export the staging dashboard and unzip
+2. Note the staging database UUIDs in the `databases/` directory
+3. Get a copy of the production database YAML configuration file
+4. In the exported dashboard files, replace the staging database YAML with the production YAML
+5. In the dataset YAML files, replace all instances of the previously noted staging database UUID with the new production UUID
+6. Zip the files and import them to the production environment
+
+The process above assumes that whoever is migrating the dashboard has a copy of the target database YAML files so that in steps 3 and 4 we can then replace the staging database YAML with the production one.
+
+## Requirements
+
+The overall process requires the following:
+- The source dashboard zip file
+- The target Superset environment database YAML file
+- Ability to copy and manipulate the source dashboard zip file
+- The ability to import via API to the target Superset environment
+
+
+## Gotchas!
+
+Migrating a dashboard once to a new target environment, database, schema will result in:
+- Creating a new dashboard with the UUID from the import zip
+- Creating a new set of charts with their UUIDs from the import zip
+- Creating a new set of datasets with their UUIDs from the import zip
+
+Migrating the same dashboard a second time to the same target environment, database, but different schema will NOT create a new dashboard.
+
+It will attempt to update the same dashboard as the UUID for the dashboard has not changed. It will also NOT change any of the datasets to the new schema. This appears to be a limitation of the import process, which may lead to some confusing results.
+
+## References
+
+Some helpful references relating to cross-environment workflows:
+- [Managing Content Across Workspaces](https://docs.preset.io/docs/managing-content-across-workspaces)
+- [Superset Slack AI Explanation](https://apache-superset.slack.com/archives/C072KSLBTC1/p1722382347022689)

lib/superset/services/dashboard_loader.rb

@@ -0,0 +1,68 @@
+# Given a path, load all yaml files
+
+require 'superset/file_utilities'
+require 'yaml'
+
+module Superset
+  module Services
+    class DashboardLoader
+      include FileUtilities
+      TMP_PATH = '/tmp/superset_dashboard_imports'.freeze
+
+      attr_reader :dashboard_export_zip
+
+      def initialize(dashboard_export_zip:)
+        @dashboard_export_zip = dashboard_export_zip
+      end
+
+      def perform
+        unzip_source_file
+        dashboard_config
+      end
+
+      def dashboard_config
+        @dashboard_config ||= DashboardConfig.new(
+                                dashboard_export_zip:    dashboard_export_zip, 
+                                tmp_uniq_dashboard_path: tmp_uniq_dashboard_path).config
+      end
+
+      private
+
+      def unzip_source_file
+        @extracted_files = unzip_file(dashboard_export_zip, tmp_uniq_dashboard_path)
+      end
+
+      def tmp_uniq_dashboard_path
+        @tmp_uniq_dashboard_path ||= File.join(TMP_PATH, uuid)
+      end
+
+      def uuid
+        SecureRandom.uuid
+      end
+
+      class DashboardConfig < ::OpenStruct
+        def config
+            {
+              dashboards: load_yamls_for('dashboards'),
+              databases:  load_yamls_for('databases'),
+              datasets:   load_yamls_for('datasets'),
+              charts:     load_yamls_for('charts'),
+              metadata:   load_yamls_for('metadata.yaml', pattern_sufix: nil),
+            }
+        end
+
+        def load_yamls_for(object_path, pattern_sufix: '**/*.yaml')
+          pattern = File.join([tmp_uniq_dashboard_path, '**', object_path, pattern_sufix].compact)
+          Dir.glob(pattern).map do |file|
+            { filename: file, content: load_yaml_and_symbolize_keys(file) } if File.file?(file)
+          end.compact
+        end
+
+        def load_yaml_and_symbolize_keys(path)
+          yaml = YAML.load_file(path)
+          yaml.deep_symbolize_keys
+        end
+      end
+    end
+  end
+end

spec/superset/services/dashboard_loader_spec.rb

@@ -0,0 +1,70 @@
+require 'superset/services/dashboard_loader'
+
+RSpec.describe Superset::Services::DashboardLoader do
+  let(:loader) { described_class.new(dashboard_export_zip: dashboard_export_zip) }
+  let(:dashboard_export_zip) { 'spec/fixtures/dashboard_18_export_20240322.zip' }
+
+  describe '#perform' do
+    before { loader.perform }
+
+    it 'populates dashboard_config with each objects filename and content' do
+      expect(loader.dashboard_config.keys).to contain_exactly(:dashboards, :datasets, :databases, :charts, :metadata)
+      loader.dashboard_config.keys.each do |object|
+        expect(loader.dashboard_config[object]).to all(include(:filename, :content))
+      end
+    end
+
+    it 'loads the metadata yaml file' do
+      metadata = loader.dashboard_config[:metadata].first
+      expect(File.basename(metadata[:filename])).to eq('metadata.yaml')
+      expect(metadata[:content].keys).to contain_exactly(:version, :type, :timestamp)
+    end
+
+    it 'loads the dashboards yaml' do
+      dashboards = loader.dashboard_config[:dashboards]
+      expect(dashboards.size).to eq(1)
+      expect(File.basename(dashboards.first[:filename])).to eq('Birth_Names_18.yaml')
+      expect(dashboards.first[:content].keys).to match_array(
+        [ :dashboard_title, :description, :css, :slug, :certified_by, :certification_details, :published,
+          :uuid, :position, :metadata, :version])
+      expect(dashboards.first[:content][:dashboard_title]).to eq('Birth Names')
+    end
+
+    it 'loads the databases yaml' do
+      databases = loader.dashboard_config[:databases]
+      expect(databases.size).to eq(1)
+      expect(File.basename(databases.first[:filename])).to eq('examples.yaml')
+      expect(databases.first[:content].keys).to match_array(
+        [:allow_ctas, :allow_cvas, :allow_dml, :allow_file_upload, :allow_run_async, :cache_timeout, 
+        :database_name, :expose_in_sqllab, :extra, :sqlalchemy_uri, :uuid, :version])
+      expect(databases.first[:content][:database_name]).to eq('examples')
+    end
+
+    it 'loads the datasets yaml' do
+      datasets = loader.dashboard_config[:datasets]
+      expect(datasets.size).to eq(1)
+      expect(File.basename(datasets.first[:filename])).to eq('birth_names.yaml')
+      expect(datasets.first[:content].keys).to match_array([
+          :table_name, :main_dttm_col, :description, :default_endpoint, :offset, :cache_timeout, :schema, :sql,
+          :params, :template_params, :filter_select_enabled, :fetch_values_predicate, :extra, :normalize_columns, :always_filter_main_dttm,
+          :uuid, :metrics, :columns, :version, :database_uuid])
+      expect(datasets.first[:content][:table_name]).to eq('birth_names')
+    end
+
+    it 'loads the charts yaml' do
+      charts = loader.dashboard_config[:charts]
+binding.pry
+      expect(charts.size).to eq(5)
+      expect(charts.map {|c| File.basename(c[:filename]) }).to match_array([
+        "Boy_Name_Cloud_53920.yaml",
+        "Names_Sorted_by_Num_in_California_53929.yaml",
+        "Number_of_Girls_53930.yaml",
+        "Pivot_Table_53931.yaml",
+        "Top_10_Girl_Name_Share_53921.yaml"])
+      expect(charts.first[:content].keys).to match_array([
+        :cache_timeout, :certification_details, :certified_by, :dataset_uuid, :description, :params, :query_context, 
+        :slice_name, :uuid, :version, :viz_type])
+      expect(charts.first[:content][:slice_name]).to eq('Boy Name Cloud')
+    end
+  end
+end