High Latency and bottleneck: Adding list of properties to a vertex

[n5nk]

Hi,

Usecase: Add attribute of type array as properties to a vertex.

We have a usecase, where one of the attributes is of type array for an entity. Internally, we iterate over each entry in that array, and add it as a property to the vertex.

Code Flow:

AtlasJanusVertex::addProperty() --> AbstractVertex::property() 

Observation:
One observation is that, as the size of the array increases to as big as 10k entries, AbstractVertex::property() happens to be the bottleneck.
Since internally a lock is created on the vertex, and a property is added, and the lock is released. We have observed that this whole process is literally time-consuming, and latency just spikes to mins ( considering 10k entries for an attribute in a single request)

Questions:

• Is there a better way to add a set of entries as properties in a single shot to a vertex, by acquiring the lock only once, rather than locking per entry ?
• Or is there a better way, which Im missing out !

Example Models:

{
  "entityDefs": [
    {
      "name": "Strategy",
      "description": "Strategy Assets",
      "typeVersion": "1.0",
      "serviceType": "atlan",
      "attributeDefs": [
        {
          "name": "reportNames",
          "typeName": "array<string>"
        }
      ]
    }
  ]
}
--------------------------------------------
{
  "entityDefs": [
    {
      "superTypes": [
        "Strategy"
      ],
      "name": "StrategyProject",
      "typeVersion": "1.0",
      "serviceType": "atlan",
      "attributeDefs": []
    }
  ],
  "relationshipDefs": []
}

--------------------------------------------
{
  "entityDefs": [
    {
      "superTypes": [
        "Strategy"
      ],
      "name": "StrategyReport",
      "typeVersion": "1.0",
      "serviceType": "atlan",
      "attributeDefs": [
        {
          "name": "strategyReportType",
          "typeName": "string"
        }
      ]
    }
  ],
  "relationshipDefs": [
    {
      "name": "strategy_project_strategy_reports",
      "typeVersion": "1.0",
      "relationshipCategory": "AGGREGATION",
      "relationshipLabel": "__StrategyProject.strategyReports",
      "serviceType": "atlan",
      "endDef1": {
        "type": "StrategyProject",
        "name": "strategyReports",
        "isContainer": true,
        "cardinality": "SET",
        "isLegacyAttribute": false
      },
      "endDef2": {
        "type": "StrategyReport",
        "name": "strategyProject",
        "isContainer": false,
        "cardinality": "SINGLE",
        "isLegacyAttribute": false
      },
      "propagateTags": "ONE_TO_TWO"
    }
  ]
}
--------------------------------------------
{
    "entityDefs":
    [
        {
            "superTypes":
            [
                "Strategy"
            ],
            "name": "StrategyAttribute",
            "description": "Strategy Attribute Asset",
            "attributeDefs":
            [
                {
                    "name": "strategyAttributeForms",
                    "typeName": "string",
                    "cardinality": "SINGLE"
                }
            ]
        }
    ],
    "relationshipDefs":
    [
        {
            "name": "strategy_project_strategy_attributes",
            "typeVersion": "1.0",
            "relationshipCategory": "AGGREGATION",
            "relationshipLabel": "StrategyProject.strategyAttributes",
            "serviceType": "atlan",
            "endDef1":
            {
                "type": "StrategyProject",
                "name": "strategyAttributes",
                "isContainer": true,
                "cardinality": "SET",
                "isLegacyAttribute": false
            },
            "endDef2":
            {
                "type": "StrategyAttribute",
                "name": "strategyProject",
                "isContainer": false,
                "cardinality": "SINGLE",
                "isLegacyAttribute": false
            },
            "propagateTags": "ONE_TO_TWO"
        },
        {
            "name": "strategy_reports_strategy_attributes",
            "typeVersion": "1.0",
            "relationshipCategory": "ASSOCIATION",
            "relationshipLabel": "StrategyReport.strategyAttributes",
            "serviceType": "atlan",
            "endDef1":
            {
                "type": "StrategyReport",
                "name": "strategyAttributes",
                "isContainer": false,
                "cardinality": "SET",
                "isLegacyAttribute": false
            },
            "endDef2":
            {
                "type": "StrategyAttribute",
                "name": "strategyReports",
                "isContainer": false,
                "cardinality": "SET",
                "isLegacyAttribute": false
            },
            "propagateTags": "ONE_TO_TWO"
        }
    ]
}

Sample Data

{
    "entities": [
       {
            "typeName": "StrategyProject",
            "attributes": {
                "name": "WF Tests",
                "qualifiedName": "default/strategy/123/projects/test"
            },
            
            "status": "ACTIVE",
            "displayText": "WF Tests",
            "meaningNames": [],
            "meanings": [],
            "isIncomplete": false,
            "labels": []
        }
    ]
}

-------
{
    "entities": [
        {
            "typeName": "StrategyAttribute",
            "status": "ACTIVE",
            "attributes": {
                "name": "TestAttribute1",
                "qualifiedName": "default/strategy/123/projects/test/attributes/1",
                "reportNames": [
                    "FY16 BAG & FTWR SALES TY"      
                ],
                "strategyProject": {
                    "typeName": "StrategyProject",
                    "uniqueAttributes": {
                        "qualifiedName": "default/strategy/123/projects/test"
                    }
                }
            }
        }
    ]
}

-------

{
    "entities": [
        {
            "typeName": "StrategyReport",
            "status": "ACTIVE",
            "attributes": {
                "name": "repor991091",
                "description": "",
                "qualifiedName": "default/strategy/123/projects/test/reports/report123",
                "strategyProjectQualifiedName": "default/strategy/123/projects/test",
                "strategyProjectName": "Global Operational",
                "strategyProject": {
                    "typeName": "StrategyProject",
                    "uniqueAttributes": {
                        "qualifiedName": "default/strategy/123/projects/test"
                    }
                },
                "strategyAttributes": [
                    {
                        "typeName": "StrategyAttribute",
                        "uniqueAttributes": {
                            "qualifiedName": "default/strategy/123/projects/test/attributes/1"
                        }
                    },
                    {
                        "typeName": "StrategyAttribute",
                        "uniqueAttributes": {
                            "qualifiedName": "default/strategy/123/projects/test/attributes/2"
                        }
                    }
                    ....
                    
                ],
                "strategyMetrics": []
            }
        }
    ]
}

[n5nk]

If we see in above samples, using reportNames array type, with 10k entries. Total time took to insert this asset was almost 20mins