Tidy-ups

Author: RoelantVos

README.md

@@ -1,16 +1,14 @@
-# Data Solution Framework
+# Design and Solution Patterns
 
-A documentation library containing reusable design- and solution patterns.
+A documentation library containing reusable design- and solution patterns, supporting [Data Engine Thinking](https://dataenginethinking.com/en/).
 
 ## Getting started
 
 Please have a look at [the introduction documentation](./docs/index.md) to get started!
 
-The automatically-generated documentation site is available on [GitHub pages](https://data-solution-automation-engine.github.io/data-solution-framework/).
-
 ## Implementation
 
-This repository is intended to be cloned and modified for organisation-specific scenarios. All files are text-based (MarkDown format, by default) for convenient editing and collaboration using Git. A DocFX file is also provided to generate static HTML from the repository's contents.
+This repository is intended to be cloned and modified for organization-specific scenarios. All files are text-based (MarkDown format, by default) for convenient editing and collaboration using Git. A DocFX file is also provided to generate static HTML from the repository's contents.
 
 To generate the content as a website (on localhost port 8081), please run the following from the 'docs' directory of the repository:
 

docs/design-patterns/design-pattern-data-vault-hub.md

@@ -4,73 +4,75 @@ uid: design-pattern-data-vault-hub
 
 # Design Pattern - Data Vault - Hub
 
-> [!WARNING]
-> This design pattern requires a major update to refresh the content.
-
 ## Purpose
 
 This design pattern describes how to define, and load data into, Data Vault Hub style tables.
 
 ## Motivation
 
-A Data Vault Hub is the physical implementation of a Core Business Concept. These are the the key identified 'things' that can be meaningfully identified as part of an organization's business processes.
+A Data Vault Hub is the physical implementation of a Core Business Concept (CBC). These are the essential 'things' that can be meaningfully identified as part of an organization's business processes.
 
 Loading data into Hub tables is a relatively straightforward process with a clearly defined location in the architecture: it is applied when loading data from the Staging Layer to the Integration Layer.
 
-The Hub is a vital component of a Data Vault solution, making sure that Data Warehouse keys are distributed properly and at the right point in time.
+The Hub is a vital component of a Data Vault solution, making sure that Data Warehouse keys are distributed properly, and at the right point in time.
 
 Decoupling key distribution and managing historical information (changes over time) is essential to reduce loading dependencies. It also simplifies (flexible) storage design in the Data Warehouse.
 
 Also known as:
 
-- Core Business Concept (Ensemble modelling)
-- Hub (Data Vault modelling concept)
-- Surrogate Key (SK) or Hash Key (HSH) distribution, as commonly used implementations of the concept
-- Data Warehouse key distribution
+- Core Business Concept (Ensemble Modeling).
+- Hub (Data Vault Modeling concept).
+- Surrogate Key (SK) or Hash Key (HSH) distribution, as commonly used implementations of the concept.
+- Data Warehouse key distribution.
 
 ## Applicability
 
-This pattern is applicable for the process of loading from the Staging Layer into Hub tables. It is used in all Hubs in the Integration Layer. Derived (Business Data Vault) Hub ETL processes follow the same pattern.
+This pattern is applicable for the process of loading from the Staging Layer into Hub tables. It is used in all Hubs in the Integration Layer. Derived (Business Data Vault) Hub data logistics processes follow the same pattern.
 
 ## Structure
 
-A Hub table contains the unique list of business key, and the corresponding Hub ETL process can be described as an 'insert only' of the unique business keys that are not yet in the the target Hub.
+A Hub table contains the unique list of business key, and the corresponding Hub data logistics process can be described as an 'insert only' of the unique business keys that are not yet in the the target Hub.
 
-The process performs a distinct selection on the business key attribute(s) in the Staging Area table and performs a key lookup to verify if the available business keys already exists in the target Hub table. If the business key already exists the row can be discarded, if not it can be inserted.
+The process performs a distinct selection on the business key attribute(s) in the Landing Area table and performs a key lookup to verify if the available business keys already exists in the target Hub table. If the business key already exists the row can be discarded, if not it can be inserted.
 
 During the selection the key distribution approach is implemented to make sure a dedicated Data Warehouse key is created. This can be an integer value, a hash key (i.e. MD5 or SHA1) or a natural business key.
 
 ## Implementation guidelines
 
-Hubs are core business concepts which must be immediately and uniquely identifiable through their name.
+Hubs must be immediately and uniquely identifiable through their name.
 
-Loading a Hub table from a specific Staging Layer table is a single, modular, ETL process. This is a requirement for flexibility in loading information as it enables full parallel processing.
+Loading a Hub table from a specific Staging Layer table is a single, modular, data logistics process. This is a requirement for flexibility in loading information as it enables full parallel processing.
 
 Multiple passes of the same source table or file are usually required for various tasks. The first pass will insert new keys in the Hub table; the other passes may be needed to populate the Satellite and Link tables.
-The designated business key (usually the source natural key, but not always!) is the ONLY non-process or Data Warehouse related attribute in the Hub table.
+The designated business key (sometimes the source natural key, but not always!) is the ONLY non-process or Data Warehouse related attribute in the Hub table.
 
-The Load Date / Time Stamp (LDTS) is copied (inherited) from the Staging Layer. This improves ETL flexibility. The Staging Area ETL is designed to label every record which is processed by the same module with the correct date/time: the date/time the record has been loaded into the Data Warehouse environment (event date/time). The ETL process control framework will track when records have been loaded physically through the Insert Module Instance ID.
+The Inscription Timestamp is copied (inherited) from the Staging Layer. This improves data logistics flexibility. The Landing Area data logistics is designed to label every record which is processed by the same module with the correct timestamp, indicating when the record has been loaded into the Data Warehouse environment. The data logistics process control framework will track when records have been loaded physically through the Audit Trail Id.
 
-Multiple ETL processes may load the same business key into the corresponding Hub table if the business key exists in more than one table. This also means that ETL software must implement dynamic caching to avoid duplicate inserts when running similar processes in parallel.
+Multiple data logistics processes may load the same business key into the corresponding Hub table if the business key exists in more than one table. This also means that data logistics software must implement dynamic caching to avoid duplicate inserts when running similar processes in parallel.
 
-By default the DISTINCT function is executed on database level to reserve resources for the ETL engine but this can be executed in ETL as well if this supports proper resource distribution (i.e. light database server but powerful ETL server).
+By default the DISTINCT function is executed on database level to reserve resources for the data logistics engine, but this can be executed inline in data logistics as well if this supports proper resource distribution (i.e. light database server but powerful data logistics server).
 
-The logic to create the initial (dummy) Satellite record can both be implemented as part of the Hub ETL process, as a separate ETL process which queries all keys that have no corresponding dummy or as part of the Satellite ETL process. This depends on the capabilities of the ETL software since not all are able to provide and reuse sequence generators or able to write to multiple targets in one process. The default and arguably most flexible way is to incorporate this concept as part of the Satellite ETL since it does not require rework when additional Satellites are associated with the Hub. This means that each Satellite ETL must perform a check if a dummy record exists before starting the standard process (and be able to roll back the dummy records if required).
+The logic to create the initial (dummy) zero key record can both be implemented as part of the Hub data logistics process, as a separate data logistics process which queries all keys that have no corresponding dummy, or issued when the Hub table is created.
 
-When modeling the Hub tables try to be conservative when defining the business keys. Not every foreign key in the source indicates a business key and therefore a Hub table. A true business key is a concept that is known and used throughout the organisation (and systems) and is self-standing and meaningful.
+When modeling the Hub tables try to be conservative when defining the business keys. Not every foreign key in the source indicates a business key and therefore a Hub table. A true business key is a concept that is known and used throughout the organization (and systems) and is self-standing and meaningful.
 
-To cater for a situation where multiple Load Date / Time stamp values exist for a single business key, the minimum Load Date / Time stamp should be the value passed through with the HUB record. This can be implemented in ETL logic, or passed through to the database.  When implemented at a database level, instead of using a SELECT DISTINCT, using the MIN function with a GROUP BY the business key can achieve both a distinct selection, and minimum Load Date / Time stamp in one step.
+To cater for a situation where multiple Load Date / Time stamp values exist for a single business key, the minimum Load Date / Time stamp should be the value passed through with the HUB record. This can be implemented in data logistics logic, or passed through to the database.  When implemented at a database level, instead of using a SELECT DISTINCT, using the MIN function with a GROUP BY the business key can achieve both a distinct selection, and minimum Load Date / Time stamp in one step.
 
 ## Considerations and consequences
 
 Multiple passes on the same Staging Layer data set are likely to be required: once for the Hub table(s) but also for any corresponding Link and Satellite tables.
 
-Defining Hub ETL processes as atomic modules, as defined in this Design Pattern, means that many Staging Layer tables load data to the same central Hub table. All processes will be very similar with the only difference being the mapping between the Staging Layer business key attribute and the target Hub business key counterpart.
+Defining Hub data logistics processes as atomic modules, as defined in this Design Pattern, means that many Staging Layer tables load data to the same central Hub table. All processes will be very similar with the only difference being the mapping between the Staging Layer business key attribute and the target Hub business key counterpart.
+
+Misidentifying business keys or over-keying leads to proliferation of Hubs and downstream complexity; validate business key selection with domain experts.
+
+Hash-key generation must be deterministic and collision-resistant; standardize hash inputs (case, trim, date formats) to avoid false duplicates.
 
 ## Related patterns
 
-- [Design Pattern - Logical - Core Business Concept](xref:design-pattern-logical-core-business-concept)
-- Design Pattern 006 - Generic - Using Start, Process and End Dates
-- Design Pattern 009 - Data Vault - Loading Satellite tables
-- Design Pattern 010 - Data Vault - Loading Link tables
-- Design Pattern 023 - Data Vault - Missing keys and placeholders
+- [Design Pattern - Logical - Core Business Concept](xref:design-pattern-logical-core-business-concept).
+- Design Pattern 006 - Generic - Using Start, Process and End Dates.
+- Design Pattern 009 - Data Vault - Loading Satellite tables.
+- Design Pattern 010 - Data Vault - Loading Link tables.
+- Design Pattern 023 - Data Vault - Missing keys and placeholders.
+

docs/design-patterns/design-pattern-data-vault-link-satellite-driving-key.md

@@ -42,6 +42,6 @@ To avoid data redundancy, it is recommended to manage this process into the targ
 
 ## Related patterns
 
-* Design Pattern - Using Start, Process and End Dates
-* Design Pattern - Satellite
-* Design Pattern - Link
+* Design Pattern - Using Start, Process and End Dates.
+* Design Pattern - Satellite.
+* Design Pattern - Link.

docs/design-patterns/design-pattern-data-vault-link-satellite.md

@@ -23,20 +23,30 @@ To provide a generic approach for loading Link Satellites.
 
 This pattern is only applicable for loading data to Link-Satellite tables from:
 
-* The Staging Area into the Integration Area.
+* The Landing Area into the Integration Area.
 * The Integration Area into the Interpretation Area.
-* The only difference to the specified ETL template is any business logic required in the mappings towards the Interpretation Area tables.
+* The only difference to the specified data logistics template is any business logic required in the mappings towards the Interpretation Area tables.
 
 ## Structure
 
- Standard Link-Satellites use the Driving Key concept to manage the ending of old relationships.
+Standard Link-Satellites use the Driving Key concept to manage the ending of old relationships. The driving key defines which key in the Link controls history (for example, the transaction id) so that related attributes expire correctly when that driving key changes.
 
 ## Implementation guidelines
 
+* Identify the driving key for the Link; use it to manage effective/expiry dates in the Link-Satellite.
+* Insert-only SCD2 pattern: close the current record (set expiry), insert a new record when the driving key/value combination changes.
+* Carry metadata from the Link (load timestamps, source identifiers) to keep lineage intact.
+* Use hash keys/checksums to detect attribute changes when applicable; avoid unnecessary updates.
+* Keep Link-Satellites narrow—store only relationship attributes (e.g., status, type, reason) and not Hub-level attributes.
+
 ## Considerations and consequences
 
+* Choosing the wrong driving key results in incorrect timelines; validate with business owners.
+* Link-Satellites are optional; if relationship attributes are stable or modeled elsewhere, they may be unnecessary.
+* Overuse of Link-Satellites can add joins and complexity; apply only when relationship history is required.
+
 ## Related patterns
 
-* Design Pattern - Using Start, Process and End Dates
-* Design Pattern - Satellite
-* Design Pattern - Link
+* Design Pattern - Using Start, Process and End Dates.
+* Design Pattern - Satellite.
+* Design Pattern - Link.