Major re-review

Still in progress. Need to redo all markups, go through documentation end to end.

Author: RoelantVos

1000_Design_Patterns/Design Pattern - Data Vault - Loading Hub tables.md

@@ -2,15 +2,18 @@
 
 ## Purpose
 This Design Pattern describes how to load data into Data Vault Hub style entities.
-Motivation
+
+## Motivation
 Loading data into Hub tables is a relatively straightforward process with a fixed location in the scheduling of loading data from the Staging Layer to the Integration Layer. It is a vital component of the Data Warehouse architecture, making sure that Data Warehouse keys are distributed properly and at the right point in time. Decoupling key distribution and historical information is an essential requirement for parallel processing and for reducing dependencies in the loading process. This pattern specifies how this process works and why it is important to follow. In a Data Vault based Enterprise Data Warehouse solution, the Hub tables (and corresponding ETL) are the only places where Data Warehouse keys are distributed.
 Also known as
 Hub (Data Vault modelling concept)
 Surrogate Key (SK) or Hash Key (HSH) distribution
 Data Warehouse key distribution
-Applicability
+
+## Applicability
 This pattern is applicable for the process of loading from the Staging Layer into the Integration Area Hub tables only.
-Structure
+
+## Structure
 The ETL process can be described as an ‘insert only’ set of the unique business keys. The process performs a SELECT DISTINCT on the Staging Area table and a key lookup to retrieve the OMD_RECORD_SOURCE_ID based on the value in the Staging Layer table.  If no entry for the record source is found the ETL process is set to fail because this indicates a major error in the ETL Framework configuration (i.e. this must be tested during unit and UAT testing).
 Using this value and the source business key the process performs a key lookup (outer join) to verify if that specific business key already exists in the target Hub table (for that particular record source). If it exists, the row can be discarded, if not it can be inserted.
 Business Insights > Design Pattern 008 - Data Vault - Loading Hub tables > image2015-4-29 14:54:58.png
@@ -20,7 +23,8 @@ The Hub ETL processes are the first ones that need to be executed in the Integra
  Business Insights > Design Pattern 008 - Data Vault - Loading Hub tables > BI2.png
 Figure 2: Dependencies
 Logically the creation of the initial Satellite record is part of the ETL process for Hub tables and is a prerequisite for further processing of the Satellites.
-Implementation guidelines
+
+## Implementation Guidelines
 Use a single ETL process, module or mapping to load the Hub table, thus improving flexibility in processing. This means that no Hub keys will be distributed as part of another ETL process.
 Multiple passes of the same source table or file are usually required for various tasks. The first pass will insert new keys in the Hub table; the other passes may be needed to populate the Satellite and Link tables.
 The designated business key (usually the source natural key, but not always!) is the ONLY non-process or Data Warehouse related attribute in the Hub table.
@@ -30,16 +34,15 @@ By default the DISTINCT function is executed on database level to reserve resour
 The logic to create the initial (dummy) Satellite record can both be implemented as part of the Hub ETL process, as a separate ETL process which queries all keys that have no corresponding dummy or as part of the Satellite ETL process. This depends on the capabilities of the ETL software since not all are able to provide and reuse sequence generators or able to write to multiple targets in one process. The default and arguably most flexible way is to incorporate this concept as part of the Satellite ETL since it does not require rework when additional Satellites are associated with the Hub. This means that each Satellite ETL must perform a check if a dummy record exists before starting the standard process (and be able to roll back the dummy records if required).
 When modeling the Hub tables try to be conservative when defining the business keys. Not every foreign key in the source indicates a business key and therefore a Hub table. A true business key is a concept that is known and used throughout the organisation (and systems) and is ‘self-standing’ and meaningful.
 To cater for a situation where multiple OMD_INSERT_DATETIME values exist for a single business key, the minimum OMD_INSERT_DATETIME should be the value passed through with the HUB record. This can be implemented in ETL logic, or passed through to the database.  When implemented at a database level, instead of using a SELECT DISTINCT, using the MIN function with a GROUP BY the business key can achieve both a distinct selection, and minimum OMD_INSERT_DATETIME in one step.
-Consequences
+
+## Considerations and Consequences
 Multiple passes on source data is likely to be required: once for Hub tables and subsequently for Link and Satellite tables. Defining Hub ETL processes in the atomic way as defined in this Design Pattern means that many files load data to the same central Hub table; all processes will be very similar with the only difference the mapping between the source attribute which represents the business key and the Hub counterpart.
 A single Hub may be loaded by many Modules from a single source system, and there may be several Satellites for the source system hanging off this Hub. It needs to be ensured that all corresponding Satellites are populated by the Hub ETL.
 Known uses
 This type of ETL process is to be used in all Hub or Surrogate Key tables in the Integration Area. The Interpretation Area Hub tables, if used, have similar characteristics but the ETL process contains business logic.
-Related patterns
+
+## Related Patterns
 Design Pattern 006 – Generic – Using Start, Process and End Dates
 Design Pattern 009 – Data Vault – Loading Satellite tables
 Design Pattern 010 – Data Vault – Loading Link tables
-Design Pattern 023 – Data Vault – Missing keys and placeholders
-
-Discussion items (not yet to be implemented or used until final)
-The OMD_INSERT_DATETIME that represents the implementation of the Event Date/Time concept is currently populated in a different way than similar OMD information (such as the OMD_UPDATE_DATETIME). It may be easier to introduce an OMD_EVENT_DATETIME attribute that captures this information.
+Design Pattern 023 – Data Vault – Missing keys and placeholders

1000_Design_Patterns/Design Pattern - Data Vault - Loading Link Satellite tables.md

@@ -2,19 +2,23 @@
 
 ## Purpose
 This Design Pattern describes how to load data into Link-Satellite tables within a ‘Data Vault’ EDW architecture. In Data Vault, Link-Satellite tables manage the change for relationships over time.
-Motivation
+
+## Motivation
 
 Also known as
 Link-Satellite (Data Vault modelling concept).
 History or INT tables.
-Applicability
+
+## Applicability
 This pattern is only applicable for loading data to Link-Satellite tables from:
 The Staging Area into the Integration Area.
 The Integration Area into the Interpretation Area.
 The only difference to the specified ETL template is any business logic required in the mappings towards the Interpretation Area tables.
-Structure
+
+## Structure
  Standard Link-Satellites use the Driving Key concept to manage the ending of ‘old’ relationships.
-Implementation guidelines
+
+## Implementation Guidelines
 Multiple passes of the same source table or file are usually required. The first pass will insert new keys in the Hub table; the other passes are needed to populate the Satellite and Link tables.
 Select all records for the Link Satellite which have more than one open effective date / current record indicator but are not the most recent (because that record does not need to be closed
 WITH MyCTE (<Link SK>, <Driving Key SK>, OMD_EFFECTIVE_DATE, OMD_EXPIRY_DATE, RowVersion)
@@ -44,10 +48,12 @@ LEFT JOIN MyCTE LEAD ON BASE.<Driving Key SK> = LEAD.<Driving Key SK>
 LEFT JOIN MyCTE LAG ON BASE.<Driving Key SK> = LAG.<Driving Key SK>
   AND BASE.RowVersion = LAG.RowVersion+1
 WHERE BASE.OMD_EXPIRY_DATE = '99991231'
-Consequences
+
+## Considerations and Consequences
 Multiple passes on source data are likely to be required.
 Known uses
-Related patterns
+
+## Related Patterns
 Design Pattern 006 – Using Start, Process and End Dates
 Design Pattern 009 – Loading Satellite tables.
 Design Pattern 010 – Loading Link tables.

1000_Design_Patterns/Design Pattern - Data Vault - Loading Link tables.md

@@ -2,14 +2,17 @@
 
 ## Purpose
 This design pattern describes the loading process for ‘Link’ tables in the Data Vault concept.
-Motivation
+
+## Motivation
 The Link concept in Data Vault provides the flexibility of this data modeling approach. Links are sets of (hub) keys that indicate that a relationship between those Hubs has existed at some point in time. A Link table is similar in concept to the Hub table, but only stores key pairs. The structure of the relationship tables (including the Link table) is documented in the Integration FrameworkA120 – Integration Layer document of the Outline Architecture. Even though the Data Vault concept allows for adding attributes in the Link table it is strongly recommended (for flexibility reasons) to only store the generated hashkey (meaningless key), the Hub surrogate hashkeys and the date/time information. Doing this will ensure compatibility with both stationary facts (time dependent facts such as balances) and pure transactions.
 Link to file Eru Marumaru
 Also known As
 Relationship table
-Applicability
+
+## Applicability
 This pattern is only applicable for loading processes from the Staging Area into the Integration Area and from the Integration Area to the Interpretation Area. The pattern varies slightly for the type of Link table specified (such as Transactional Link, Same-As Link, Hierarchical Link or Low-Value Links) or whether it contains a degenerate attribute. In most cases the Link table will contain the default attributes (Link Key, Hub Keys and metadata attributes) but in the case of a pure transactional Link table it can contain the transaction attributes as well.
-Structure
+
+## Structure
 The ETL process can be described as an ‘insert only’ set of the unique combination of Data Warehouse keys. Depending on the type of source table, the process will do the following:
 Source Area to Integration Area: the process executes a SELECT DISTINCT query on business keys and performs key lookups (outer join) on the corresponding Hub tables to obtain the Hub Data Warehouse keys. The resulting key combination is then verified using a key lookup against the target Link table to verify if that specific combination of Data Warehouse keys already exists. If it exists, the row can be discarded, if not it can be inserted.
 Integration Area to Interpretation Area: the process executes a SELECT DISTINCT query on Data Warehouse keys (likely after combining multiple tables first) and performs a key lookup against the target Link table to verify if that specific combination of Data Warehouse keys already exists. If it exists, the row can be discarded, if not it can be inserted.
@@ -18,7 +21,8 @@ The following diagram displays the ETL process for Link tables;
 Business Insights > Design Pattern 010 - Data Vault - Loading Link tables > image2015-4-29 16:24:14.png
 This image needs updating to reflect DV 2.0 (hashkey) useage Eru Marumaru
 In a pure relational Link it is required that a dummy key is available in each corresponding Link-Satellite to complete the timelines. This is handled as part of the Link-Satellite processing as a Link can contain multiple Link-Satellites. Dummy records are only required to be inserted for each driving key as a view in time across the driving key is ultimately required. Inserting a dummy record for every Link key will cause issues in the timeline. This is explained in more detail in the Link-Satellite Design Pattern.
-Implementation Guidelines
+
+## Implementation Guidelines
 Use a single ETL process, module or mapping to load the Link table, thus improving flexibility in processing. Every ETL process should have a distinct function.
 Multiple passes of the same source table or file are usually required. The first pass will insert new keys in the Link table; the other passes are needed to populate the Link Satellite tables (if any).
 By default, create a sequence / meaningless key for each unique key combination in a Link table.
@@ -27,11 +31,13 @@ Date/time information is copied from the Staging Area tables and not generated b
 The logic to create the initial (dummy) Satellite record can both be implemented as part of the Link ETL process, as a separate ETL process which queries all keys that have no corresponding dummy or as part of the Link-Satellite ETL process. This depends on the capabilities of the ETL software since not all are able to provide and reuse sequence generators or able to write to multiple targets in one process. The default and arguably most flexible way is to incorporate this concept as part of the Link-Satellite ETL since it does not require rework when additional Link-Satellites are associated with the Link. This means that each Link-Satellite ETL must perform a check if a dummy record exists before starting the standard process (and be able to roll back the dummy records if required).
 Depending on how the Link table is modelled (what kind of relationship it manages) the Link table may contains a relationship type attribute. If a link table contains multiple, or changing, relationships (types) this attributes is moved to the Link-Satellite table.
 Ending /closing relationships is always done in the Link-Satellite table, typically using a separate ETL process.
-Consequences
+
+## Considerations and Consequences
 Multiple passes on source data is likely to be required. In extreme cases a single source table might be used (branch out) to Hubs, Satellites, Links and Link Satellites.
  Known Uses
 This type of ETL process is to be used for loading all link tables in both the Integration Area as well as the Interpretation Area. This is because the Link table is also used to relate raw (Integration Area) data and cleansed (Interpretation Area) data together.
- Related Patterns
+
+## Related Patterns
 Design Pattern 006 – Generic – Using Start, Process and End Dates
 Design Pattern 008 – Data Vault – Loading Hub tables
 Design Pattern 009 – Data Vault – Loading Satellite tables

1000_Design_Patterns/Design Pattern - Data Vault - Loading Satellite tables.md

@@ -2,17 +2,20 @@
 
 ## Purpose
 This Design Pattern describes how to load data into Satellite tables within a ‘Data Vault’ EDW architecture. The concept can be applied to any SCD2 mechanism as well.
-Motivation
+
+## Motivation
 The Design Pattern to load data into Satellite style tables aims to simplify and streamline the way ETL design is done for these tables. The process is essentially straightforward and does not require any business logic other than the definition of the business key. This is already done as part of the data modelling and Hub definition steps.
 Also known as
 Satellite (Data Vault modelling concept).
 History or INT tables.
-Applicability
+
+## Applicability
 This pattern is only applicable for loading data to Satellite tables from:
 The Staging Area into the Integration Area.
 The Integration Area into the Interpretation Area.
 The only difference to the specified ETL template is any business logic required in the mappings towards the Interpretation Area Satellite tables.
-Structure
+
+## Structure
 The ETL process can be described as a slowly changing dimension / history update of all attributes except the business key (which is stored in the Hub table). This is explained in the following diagram. Most attribute values, including some of the OMD values are copied from the Staging Area table. This includes:
 OMD_INSERT_DATETIME (used for the target OMD_EFFECTIVE_DATE and OMD_UPDATE_DATETIME attributes).
 OMD_SOURCE_ROW_ID.
@@ -24,7 +27,8 @@ Figure 1: Satellite ETL process
 The Satellite ETL processes can only be run after the Hub process has finished, but can run in parallel with the Link ETL process. This is displayed in the following diagram:
 Business Insights > Design Pattern 009 - Data Vault - Loading Satellite Tables > BI.png
 Figure 2: Dependencies
-Implementation guidelines
+
+## Implementation Guidelines
 Multiple passes of the same source table or file are usually required. The first pass will insert new keys in the Hub table; the other passes are needed to populate the Satellite and Link tables.
 The process in Figure 1 shows the entire ETL in one single process. For specific tools this way of developing ETL might be relatively inefficient. Therefore, the process can also be broken up into two separate mappings; one for inserts and one for updates. Logically the same actions will be executed, but physically two separate mappings can be used. This can be done in two ways:
 Follow the same logic, with the same selects, but place filters for the update and insert branches. This leads to an extra pass on the source table, at the possible benefit of running the processes in parallel.
@@ -39,13 +43,13 @@ WHERE  (            satellite.OMD_EXPIRY_DATE IS NULL AND
 ORDER BY 1,2 DESC
 If you have a Change Data Capture based source, the attribute comparison is not required because the source system supplies the information whether the record in the Staging Area is new, updated or deleted.
 Use hash values to detect changes, instead of comparing attributes separately. The hash value is created from all attributes except the business key and OMD values.
-Consequences
+
+## Considerations and Consequences
 Multiple passes on source data are likely to be required.
 Known uses
 This type of ETL process is to be used in all Hub or SK tables in the Integration Area. The Cleansing Area Hub tables, if used, have similar characteristics but the ETL process contains business logic.
-Related patterns
+
+## Related Patterns
 Design Pattern 006 – Generic – Using Start, Process and End Dates
 Design Pattern 009 – Data Vault – Loading Satellite tables.
-Design Pattern 010 – Data Vault – Loading Link tables.
-Discussion items (not yet to be implemented or used until final)
-None.
+Design Pattern 010 – Data Vault – Loading Link tables.