Skip to content

Commit 755dfc7

Browse files
committed
Tidying-up continues
1 parent 9b08118 commit 755dfc7

File tree

41 files changed

+307
-358
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+307
-358
lines changed

docs/design-patterns/design-pattern-data-vault-hub.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Multiple passes on the same Staging Layer data set are likely to be required: on
6767

6868
Defining Hub ETL processes as atomic modules, as defined in this Design Pattern, means that many Staging Layer tables load data to the same central Hub table. All processes will be very similar with the only difference being the mapping between the Staging Layer business key attribute and the target Hub business key counterpart.
6969

70-
## Related Patterns
70+
## Related patterns
7171

7272
- [Design Pattern - Logical - Core Business Concept](xref:design-pattern-logical-core-business-concept)
7373
- Design Pattern 006 - Generic - Using Start, Process and End Dates

docs/design-patterns/design-pattern-data-vault-link-satellite-driving-key.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ To avoid data redundancy, it is recommended to manage this process into the targ
4040

4141
## Considerations and consequences
4242

43-
## Related Patterns
43+
## Related patterns
4444

4545
* Design Pattern - Using Start, Process and End Dates
4646
* Design Pattern - Satellite

docs/design-patterns/design-pattern-data-vault-link-satellite.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ This pattern is only applicable for loading data to Link-Satellite tables from:
3535

3636
## Considerations and consequences
3737

38-
## Related Patterns
38+
## Related patterns
3939

4040
* Design Pattern - Using Start, Process and End Dates
4141
* Design Pattern - Satellite

docs/design-patterns/design-pattern-data-vault-link.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Multiple passes on source data is likely to be required. In extreme cases a sing
6262

6363
This type of ETL process is to be used for loading all link tables in both the Integration Area as well as the Interpretation Area. This is because the Link table is also used to relate raw (Integration Area) data and cleansed (Interpretation Area) data together.
6464

65-
## Related Patterns
65+
## Related patterns
6666

6767
* Design Pattern - Generic - Using Start, Process and End Dates
6868
* [Design Pattern - Data Vault - Hub tables](xref:design-pattern-data-vault-hub)

docs/design-patterns/design-pattern-data-vault-missing-keys-and-placeholders.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
1+
---
2+
uid: design-pattern-data-vault-missing-keys-and-placeholders
3+
---
4+
15
# Design Pattern - Data Vault - Missing Keys and Placeholders
26

7+
> [!WARNING]
8+
> This design pattern requires a major update to refresh the content.
9+
310
## Purpose
411

512
This Design Pattern documents how to handle situations where there are mismatches with the source business keys leading to values not being available in some cases. Due to the strict approach towards key lookups this would lead to errors in ETL. This is where placeholders are applied. The pattern assumes that source files are always first processed against Hub tables including loading any transactional tables against the Hubs.
@@ -43,5 +50,5 @@ Implementing a full taxonomy of potential unknown values as hard business rules
4350
Known uses
4451
This type of ETL process is to be used in all Hub or Surrogate Key tables in the Integration Area. The Interpretation Area Hub tables, if used, have similar characteristics but the ETL process contains business logic.
4552

46-
## Related Patterns
53+
## Related patterns
4754
Design Pattern 008 Data Vault Loading Hub tables.

docs/design-patterns/design-pattern-data-vault-satellite.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Use hash values to detect changes, instead of comparing attributes separately. T
5757

5858
Multiple passes on source data are likely to be required.
5959

60-
## Related Patterns
60+
## Related patterns
6161

6262
Design Pattern 006 - Generic - Using Start, Process and End Dates
6363
Design Pattern 009 - Data Vault - Loading Satellite tables

docs/design-patterns/design-pattern-dimensional-loading-from-the-persistent-staging-area.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ LEFT JOIN Table1 ON NOT (Table1.EFFECTIVE_DATETIME >= R.EXPIRY_DATETIME OR Table
140140
LEFT JOIN Table2 ON NOT (Table2.EFFECTIVE_DATETIME >= R.EXPIRY_DATETIME OR Table2.EXPIRY_DATETIME <= R.EFFECTIVE_DATETIME)
141141
```
142142

143-
## Related Patterns
143+
## Related patterns
144144

145145
* Design Pattern 019 - Data Vault - Creating Dimensions from Hub tables.
146146
* Design Pattern 006 - Generic - Managing temporality by using Start, Process and End dates.

docs/design-patterns/design-pattern-dimensional-time-dimension.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,5 @@ The decision not to copy the data types from the file definitions but to check a
2424
Known uses
2525
None.
2626

27-
## Related Patterns
27+
## Related patterns
2828
Design Pattern 015 � Generic � Loading Staging Area tables.
Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,11 @@
1-
# Design Pattern - Staging Layer - Staging Area Guidelines
1+
---
2+
uid: design-pattern-generic-control-framework
3+
---
4+
5+
# Design Pattern - Generic - Control Framework
6+
7+
> [!WARNING]
8+
> This design pattern is a placeholder awaiting content
29
310
## Purpose
411

@@ -8,12 +15,12 @@ This Design Pattern describes the reasoning for, and functionality provided by,
815

916
## Applicability
1017

11-
This pattern is only applicable for every process in the data solution.
18+
This pattern is only applicable for _every_ process in the data solution.
1219

1320
## Structure
1421

1522
## Implementation guidelines
1623

1724
## Considerations and consequences
1825

19-
## Related Patterns
26+
## Related patterns
Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,27 @@
11
# Design Pattern - Generic - Data extraction from internal systems
22

3+
> [!WARNING]
4+
> This design pattern is a placeholder awaiting content
5+
36
## Purpose
7+
48
This Design Pattern describes the overarching concepts related to extracting data from internal systems.
59

610
## Motivation
7-
-
811

912
## Applicability
1013

11-
1214
## Structure
13-
* Data must be extracted from the sources that created the data (as opposed to using copied data as a source). This is a broader Data Governance principle
14-
* Impacts on operational systems must be assessed and documented as part of the interface specification
15-
* Source system extract processes should not include the Data Warehouse transformation, aggregation and consolidation rules, this is applied later (separation of concerns)
16-
* Source system extracts should include control data to enable audit and reconciliation, e.g. record count, hash totals, etc. This is by default supported by the ETL process control model
17-
* The standard data integration tool must be used to extract data from the source systems, unless another efficient data extract utility is provided as part of the application package (this may include using SQL for ETL)
18-
* Implement incremental extracts where possible, as this is more scalable.
1915

20-
## Implementation guidelines
16+
- Data must be extracted from the sources that created the data (as opposed to using copied data as a source). This is a broader Data Governance principle
17+
- Impacts on operational systems must be assessed and documented as part of the interface specification
18+
- Source system extract processes should not include the Data Warehouse transformation, aggregation and consolidation rules, this is applied later (separation of concerns)
19+
- Source system extracts should include control data to enable audit and reconciliation, e.g. record count, hash totals, etc. This is by default supported by the ETL process control model
20+
- The standard data integration tool must be used to extract data from the source systems, unless another efficient data extract utility is provided as part of the application package (this may include using SQL for ETL)
21+
- Implement incremental extracts where possible, as this is more scalable.
2122

23+
## Implementation guidelines
2224

2325
## Considerations and consequences
2426

25-
26-
## Related Patterns
27+
## Related patterns

0 commit comments

Comments
 (0)