Skip to content

[DOP-23867] Adjust components to handle new lineage response schemas#21

Merged
dolfinus merged 1 commit intodevelopfrom
feature/DOP-23867
Feb 17, 2025
Merged

[DOP-23867] Adjust components to handle new lineage response schemas#21
dolfinus merged 1 commit intodevelopfrom
feature/DOP-23867

Conversation

@dolfinus
Copy link
Member

@dolfinus dolfinus commented Feb 13, 2025

API response for lineage API endpoint was changed from:

{
    "relations": [
        {
            "kind": "PARENT",
            "from": {"kind": "JOB", "id": 123},
            "to": {"kind": "RUN", "id": "00000000-0000-0000-0000-000000000000"},
        },
        {
            "kind": "SYMLINK",
            "from": {"kind": "DATASET", "id": 234},
            "to": {"kind": "DATASET", "id": 999},
        },
        {
            "kind": "INPUT",
            "from": {"kind": "DATASET", "id": 234},
            "to": {"kind": "OPERATION", "id": "11111111-1111-1111-1111-111111111111"},
        },
        {
            "kind": "OUTPUT",
            "from": {"kind": "OPERATION", "id": "11111111-1111-1111-1111-111111111111"},
            "to": {"kind": "DATASET", "id": 234},
        },
    ],
    "nodes": [
        {"kind": "DATASET", "id": 123, "name": "abc"},
        {"kind": "JOB", "id": 234, "name": "cde"},
        {
            "kind": "RUN",
            "id": "00000000-0000-0000-0000-000000000000",
            "external_id": "def",
        },
        {
            "kind": "OPERATION",
            "id": "11111111-1111-1111-1111-111111111111",
            "name": "efg",
        },
    ],
}

to:

{
    "relations": {
        "parents": [
            {
                "from": {
                    "kind": "JOB",
                    "id": "123"
                },
                "to": {
                    "kind": "RUN",
                    "id": "00000000-0000-0000-0000-000000000000"
                },
            },
        ],
        "symlinks": [
            {
                "from": {
                    "kind": "DATASET",
                    "id": "234"
                },
                "to": {
                    "kind": "DATASET",
                    "id": "999"
                },
            },
        ],
        "inputs": [
            {
                "from": {
                    "kind": "DATASET",
                    "id": "234"
                },
                "to": {
                    "kind": "OPERATION",
                    "id": "11111111-1111-1111-1111-111111111111",
                },
            },
        ],
        "outputs": [
            {
                "from": {
                    "kind": "OPERATION",
                    "id": "11111111-1111-1111-1111-111111111111",
                },
                "to": {
                    "kind": "DATASET",
                    "id": "234"
                },
            },
        ],
    },
    "nodes": {
        "datasets": {
            "123": {
                "id": "123",
                "name": "abc"
            },
        },
        "jobs": {
            "234": {
                "id": "234",
                "name": "cde"
            },
        },
        "runs": {
            "00000000-0000-0000-0000-000000000000": {
                "id": "00000000-0000-0000-0000-000000000000",
                "external_id": "def",
            },
        },
        "operations": {
            "11111111-1111-1111-1111-111111111111": {
                "id": "11111111-1111-1111-1111-111111111111",
                "name": "efg",
            },
        },
    },
}

See MTSWebServices/data-rentgen#164.

Updated lineage components to be aligned with new schemas.

@dolfinus dolfinus self-assigned this Feb 13, 2025
@dolfinus dolfinus requested a review from Zabilsya February 13, 2025 14:46
@dolfinus dolfinus merged commit 4ae7469 into develop Feb 17, 2025
3 checks passed
@dolfinus dolfinus deleted the feature/DOP-23867 branch February 17, 2025 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants