Feature/account mapping#1457
Open
martrost wants to merge 8 commits intoaws-solutions-library-samples:mainfrom
Open
Feature/account mapping#1457martrost wants to merge 8 commits intoaws-solutions-library-samples:mainfrom
martrost wants to merge 8 commits intoaws-solutions-library-samples:mainfrom
Conversation
…le, and file discovery Add AutoDiscovery class to account_mapper_helpers.py with intelligent auto-selection logic and interactive prompts: - discover_databases: Auto-select single database or prompt for multiple - discover_tables: Auto-select single table or prompt for multiple - discover_tag_keys: Query and parse hierarchytags to extract available keys - discover_account_id_column: Pattern matching for common account ID columns - prompt_file_selection: InquirerPy file picker with extension filtering Implements requirements 3.4-3.11, 4.5-4.7, 9.1-9.2, 16.2-16.3, 16.10, 16.12 Phase 3 of account mapper refactor complete.
Refactor account mapper into modular classes (AutoDiscovery, ConfigManager, DataLoader, TransformEngine, AthenaWriter, UnifiedWorkflow) replacing the monolithic map_config.py. Key improvements: - Add --simple flag for backwards compatibility - Use managementaccountid instead of payer_id from org table - Add payer naming feature with CASE WHEN SQL generation - Add file source view lifecycle management (create, reuse, cleanup) - Add name_split dimension support with split_part() in Athena - Add checkbox retry-on-empty-selection UX helper - Add dimension name sanitization (spaces to underscores) - Sort taxonomy dimension columns alphabetically in output - Persist and reload payer_names from config view - Add stale file_source_view cleanup when no file dims exist
Remove unused functions, methods, and module-level state: - _discover_database/_discover_table wrappers (replaced by discover_source) - extract_payer_info (replaced by payer_names config + CASE WHEN SQL) - _get_config_hash, clear_athena_cache, _athena_data_cache (never called) - Account.add_tag/get_tags/get_tag/get_business_unit (never called) - DataLoader.get_available_columns (never called) - DataLoader.auto_detect_account_column (duplicate of AutoDiscovery) - DataLoader.validate_account_ids and validate_ids param (never used) - Account._account_tags init field (no longer needed)
…ble commands and the map command
Author
|
Added the docs/cid-cmd update so the documentation is available as well |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cid-cmd map — Account Mapping
The
mapcommand creates anaccount_mapAthena view that enriches your AWS account data with custom taxonomy dimensions (business unit, environment, cost center, etc.). These dimensions can then be used in Cloud Intelligence Dashboards for grouping, filtering, and reporting.How It Works
The command follows a six-phase workflow:
organization_datatable (from AWS Organizations data collection) and the target Athena databaseaccount_map,account_map_config, and optionallyaccount_map_file_source)Usage
Options
--file PATH--simpleaccount_mapview with justaccount_idandaccount_namefrom the organization_data table--view-name TEXTaccount_map)-v/--verbose-vv)-y/--yesTaxonomy Dimension Sources
During interactive configuration you choose one or more data sources for your taxonomy dimensions:
Tags from source table
Extracts values from the
hierarchytagscolumn inorganization_data. Each selected tag key becomes a column in the output view.Example: if your accounts are tagged with
Environment=ProductionandCostCenter=Engineering, selecting those tag keys producesenvironmentandcost_centercolumns.Additional file (
--file)Joins columns from a CSV/Excel/JSON file by account ID. The file must contain an account ID column; all other selected columns become taxonomy dimensions.
Example file (
accounts.csv):Split account name
Extracts dimensions by splitting the
account_namestring on a separator character. You specify the separator and the positional index to extract.Example: for account name
aws-retail-prod, splitting by-at index 1 yieldsretail, and at index 2 yieldsprod.Configuration Persistence
The command saves its configuration as an Athena view (
<view_name>_config). On subsequent runs it detects the existing config and offers to reuse it, so you don't have to reconfigure every time.Views Created
account_mapaccount_map_configaccount_map_file_source--fileis used) Stores the file data as an Athena view for joinsPrerequisites
organization_datatable in Athena (typically created by the CID data collection CFN stack)Examples
Create an account map using AWS Organization tags:
Create an account map enriched with data from a spreadsheet:
Re-run with saved configuration (no prompts):