Skip to content

Commit d7c134d

Browse files
committed
Minior changes in wording
1 parent 7542382 commit d7c134d

2 files changed

+16
-5
lines changed

Data Integration Framework - Reference Solution Architecture - 1 - Overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
The reference Solution Architecture ('reference architecture') is designed to facilitate a platform-independent, flexible and manageable data solution.
44

5-
The fundamental principle of the reference architecture is to *design for change* by decoupling 'technical' logic and 'business' logic and ensuring each data integration process can run independently and in parallel with built-in recovery mechanisms.
5+
The fundamental principles of the reference architecture is to *design for change* by decoupling 'technical' and 'business' logic, and to ensure each data integration process can run independently and in parallel with built-in recovery mechanisms. Both of these principles are means to enable scalability and flexibility.
66

77
The reference architecture aims to provide guidelines for decoupling (functional separation) of the various elements of the data solution, so new or changed requirements can be incorporated without re-engineering the data solution foundations.
88

Data Integration Framework - Reference Solution Architecture - 2 - Staging Layer.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
The Staging Layer covers the first series of ETL process steps within the reference architecture. The processes involved with the Staging Layer introduce data from many (often disparate) source applications into the Data Warehouse environment.
44

5-
In this sense, the Staging Layer is for the most part literally a place where the data is collected onto the Data Warehouse environment before being integrated in the core Data Warehouse or loaded for other use-cases (i.e. analytics, ad-hoc reporting).
5+
In this sense, the Staging Layer is for the most part literally a place where the data is collected in the Data Warehouse environment, before being integrated in the core Data Warehouse layer (Integration Layer) or made available for other use-cases (i.e. analytics, ad-hoc reporting) as a more raw data feed.
66

77
But even then many fundamental decisions are required that have repercussions throughout the rest of the design. This document defines the Staging Layer and describes the required process steps and available solutions.
88

@@ -14,7 +14,20 @@ The position of the Staging Layer in the overall architecture is outlined in the
1414

1515
![1547519184139](.\Images\Staging_Layer_1_Overview.png)
1616

17-
## Staging Layer
17+
Core requirements and principles
18+
19+
* Disconnected (no Referential Integrity)
20+
* Transient, except for the Persistent Staging Area
21+
* Raw data, original events
22+
* Standardised data types
23+
* Nullable
24+
* Sequenced
25+
* Time-stamped & time-ordered
26+
* Record source identifiable
27+
28+
29+
30+
## Areas of the Staging Layer
1831

1932
The Staging Layer consists of the **Staging Area** and the **Persistent Staging Area**. The main purpose of this layer is to collect source data and optionally store it in a source data archive. The Staging Layer prepares and collects data for further process into the Integration Layer.
2033

@@ -26,8 +39,6 @@ An option in the Data Warehouse design is to load the source data into a History
2639

2740
Objects in the Staging Layer are not accessible for end-users or Business Intelligence and analytics software (e.g. Cognos). This is because for most scenarios information has not yet been prepared for consumption. There is an exception to this rule; for specific data mining or statistical analysis it is often preferable for analysts to access the raw / unprocessed data. This means this access can be granted for the Staging Layer which contains essentially raw time variant data. Allow access serves a purpose in prototyping and local self-service BI / visualisation.
2841

29-
30-
3142
The Staging Layer, or the process from source to staging, consists of two separate parts (areas):
3243

3344
* The Staging Area, and

0 commit comments

Comments
 (0)