Skip to content

Commit 98e8082

Browse files
author
Bhooshan Mogal
authored
Merge pull request #9 from data-integrations/update-docs
Update SCD2-sparkcompute.md
2 parents 1246c25 + 674ab49 commit 98e8082

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

docs/SCD2-sparkcompute.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,23 @@ Use Case
99
This plugin is used to integrate the incoming data to the Slowly Changing Dimension Type 2 (SCD2) target tables, and to track the history of the record.
1010

1111
It supports the following patterns:
12+
1213
**New record integration**
1314
If a new record is received from the input flow (with new natural key), it is added as active, with end date equal to a dummy value (9999-12-31).
15+
1416
**Changed record integration**
1517
If a record is received with at least an update, the previous version is closed, with end date = start date of the new record minus
1618
one second) and the new record is added as active, with end date equal to the dummy value (9999-12-31).
19+
1720
**False delta record cut off**
1821
This capability checks if at least one of the fields of the input record has changed from the previous version already present in
1922
the SCD2 table. If it does it is integrated to the table, otherwise it is discarded. This feature allows this plug-in to be agnostic about the input data loading
2023
(delta or full), since it manages only the changed records and avoids integrating records without variations.
24+
2125
**Late arriving data**
2226
The plugin checks the date of completion of each incoming record in order to add them in the right time interval (start/end date) and
2327
to handle records with the date of completion older than the current active version. For example:
24-
Initial status of the SCD2:
28+
Initial status of the SCD2 dataset:
2529

2630
| SubId | TariffId | Status | Start_Date | End_Date |
2731
|-------|----------|--------|------------|----------|

0 commit comments

Comments
 (0)