You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Data Integration Framework - Reference Solution Architecture - 1 - Overview.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
The reference Solution Architecture ('reference architecture') is designed to facilitate a platform-independent, flexible and manageable data solution.
4
4
5
-
The fundamental principle of the reference architecture is to *design for change* by decoupling 'technical' logic and 'business' logic and ensuring each data integration process can run independently and in parallel with built-in recovery mechanisms.
5
+
The fundamental principles of the reference architecture is to *design for change* by decoupling 'technical' and 'business' logic, and to ensure each data integration process can run independently and in parallel with built-in recovery mechanisms. Both of these principles are means to enable scalability and flexibility.
6
6
7
7
The reference architecture aims to provide guidelines for decoupling (functional separation) of the various elements of the data solution, so new or changed requirements can be incorporated without re-engineering the data solution foundations.
Copy file name to clipboardExpand all lines: Data Integration Framework - Reference Solution Architecture - 2 - Staging Layer.md
+15-4Lines changed: 15 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
The Staging Layer covers the first series of ETL process steps within the reference architecture. The processes involved with the Staging Layer introduce data from many (often disparate) source applications into the Data Warehouse environment.
4
4
5
-
In this sense, the Staging Layer is for the most part literally a place where the data is collected onto the Data Warehouse environment before being integrated in the core Data Warehouse or loaded for other use-cases (i.e. analytics, ad-hoc reporting).
5
+
In this sense, the Staging Layer is for the most part literally a place where the data is collected in the Data Warehouse environment, before being integrated in the core Data Warehouse layer (Integration Layer) or made available for other use-cases (i.e. analytics, ad-hoc reporting) as a more raw data feed.
6
6
7
7
But even then many fundamental decisions are required that have repercussions throughout the rest of the design. This document defines the Staging Layer and describes the required process steps and available solutions.
8
8
@@ -14,7 +14,20 @@ The position of the Staging Layer in the overall architecture is outlined in the
* Transient, except for the Persistent Staging Area
21
+
* Raw data, original events
22
+
* Standardised data types
23
+
* Nullable
24
+
* Sequenced
25
+
* Time-stamped & time-ordered
26
+
* Record source identifiable
27
+
28
+
29
+
30
+
## Areas of the Staging Layer
18
31
19
32
The Staging Layer consists of the **Staging Area** and the **Persistent Staging Area**. The main purpose of this layer is to collect source data and optionally store it in a source data archive. The Staging Layer prepares and collects data for further process into the Integration Layer.
20
33
@@ -26,8 +39,6 @@ An option in the Data Warehouse design is to load the source data into a History
26
39
27
40
Objects in the Staging Layer are not accessible for end-users or Business Intelligence and analytics software (e.g. Cognos). This is because for most scenarios information has not yet been prepared for consumption. There is an exception to this rule; for specific data mining or statistical analysis it is often preferable for analysts to access the raw / unprocessed data. This means this access can be granted for the Staging Layer which contains essentially raw time variant data. Allow access serves a purpose in prototyping and local self-service BI / visualisation.
28
41
29
-
30
-
31
42
The Staging Layer, or the process from source to staging, consists of two separate parts (areas):
0 commit comments