Skip to content

Commit 822d91c

Browse files
authored
Update Design Pattern - Generic - Fundamental ETL Requirements.md
1 parent ab3b30f commit 822d91c

File tree

1 file changed

+1
-10
lines changed

1 file changed

+1
-10
lines changed

1000_Design_Patterns/Design Pattern - Generic - Fundamental ETL Requirements.md

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,6 @@ The purpose of this Design Pattern is to define a set of minimal requirements ev
66
## Motivation
77
Regardless of (place in the) architecture or purpose every ETL process should be created to follow a distinct set of base rules. Essential concepts such as having ETL processes check if they have already run before inserting duplicates or corrupting data makes testing, maintenance and troubleshooting a more straightforward task. The ultimate motivation is to develop ETL which cannot cause errors due to unplanned or unwanted execution. Essentially, ETL must be able to be run and re-run at any point in time to support a fully flexible scheduling and implementation.
88

9-
## Also known as
10-
ETL guidelines
11-
Basic ETL architecture
12-
139
## Applicability
1410
This Design Pattern applies to every ETL process.
1511

@@ -36,15 +32,10 @@ An example is as follows:
3632

3733
ETL processes are recommended to be placed in the directory/folder where they pull data _to_. For instance the ETL logic for ‘Staging to History’ exists in the ‘150_History_Area’ folder and loads data from the ‘100_Staging_Area’.
3834

39-
## Consequences and considerations
35+
## Considerations and considerations
4036
In some situations specific properties of the ETL process may seem overkill or perhaps even redundant. This (perceived) additional effort will have its impact on developing duration.
4137

4238
But in the context of maintaining a generic design (e.g. to support ETL generation and maintenance) this will still be necessary. Concessions may be made per architectural Layer (all ETL processes within a certain architecture step) but this is recommended to be motivated in the customised (i.e. project specific) Solution Architecture documentation.
4339

44-
## Known uses
45-
All data integration processes designed and developed within the context of the Data Integration framework and architecture.
46-
4740
## Related patterns
4841
In the various Design and Implementation Patterns where detailed ETL design for a specific task is documented the requirements in this pattern will be adhered to.
49-
Discussion items (not yet to be implemented or used until final)
50-
None.

0 commit comments

Comments
 (0)