concatTR

concatTR Transformer

Purpose

The purpose of this transformer is to concatenate (or make the equivalent of a SQL union) several data sources.

Configuration by using the SQL directly in the configuration file

The specific configuration (as a Datasource) in the configuration file section parameters includes no specifics parameters.

Data sources configuration

• Inputs : several inputs
• Outputs : 1 output

Configuration example:

    "transformers":  [ { 
        "id": "Concat Transformer",
        "inputs" : [ "E1", "E2" ],
        "outputs" : [ "L1" ],
        "classname": "pipelite.transformers.concatTR"
    },
    ... ] ...

Example

graph TD;
    id1[Read Data Source E1]-->id2[Dataset E1];
    id10[Read Data Source E2]-->id12[Dataset E2];
    id2[Dataset S1]-->id3[Concat Datasets E1 and E2];
    id12[Dataset E2]-->id3[Concat Datasets E1 and E2];
    id3[Concat Datasets E1 and E2]-->id4[Create Dataset O1];
    id4[Create Dataset L1]-->id5[passthroughTR O1 to L1];
    id5[passthroughTR O1 to L1]-->id6[Write Dataset L1]

Loading

The pipelite report must looks like this:

--- PIPELITE REPORT ---
           Type                                    Description                Start                  End  Duration Rows Processed Order
id                                                                
E1    Extractor                      csvFileDS -> Output: [E1]  2023-12-04 08:35:21  2023-12-04 08:35:21  0.012083              3     1
E2    Extractor                      csvFileDS -> Output: [E2]  2023-12-04 08:35:21  2023-12-04 08:35:21  0.004318              4     2
T1  Transformer    concatTR -> Inputs: [E1,E2] / Outputs: [O1]  2023-12-04 08:35:21  2023-12-04 08:35:21  0.011393              7     3
T2  Transformer  passthroughTR -> Inputs: [O1] / Outputs: [L1]  2023-12-04 08:35:21  2023-12-04 08:35:21  0.000311              7     4
L1       Loader                       csvFileDS -> Input: [L1]  2023-12-04 08:35:21  2023-12-04 08:35:21  0.003783              7     5

See here