Performance issues while requesting catalog (Catalog V3)

[trueH3]

Bug Report

Describe the Bug

were facing performance problems while executing

curl --location 'http://localhost:29193/management/v3/catalog/request'
--header 'Content-Type: application/json'
--data-raw '{
"@context": {
"@vocab": "https://w3id.org/edc/v0.0.1/ns/"
},
"counterPartyAddress": "http://localhost:19194/protocol",
"protocol": "dataspace-protocol-http",
"querySpec": {
"limit": 100,
"filterExpression": [
{
"operandLeft": "https://w3id.org/edc/v0.0.1/ns/id",
"operator": "=",
"operandRight": "example-s3-asset",
"@type": "Criterion"
}
]
}

}
'
We are using version 0.10.1 but I checked latests version 0.16.0 and i believe that the problem is still there.

after investigation it was discovered that method ContractDefinitionResolverImpl.resolveFor located in org.eclipse.edc.connector.controlplane.catalog

is ignoring my filterExpression statement from request and fetches all entries from edc_contract_definitions, then for each entry fetched it make call to db to fetch respective access policy from edc_policydefinitions table. So given we have scenario that for example we currently have in database 3000 entries of different contract definitions and each contract has unique access policy assigned to it in order to execute code logic in this method we need to make 3000 + 1 calls to database which takes a lot of times and often causing timeout exceptions thrown by this endpoint. I Checked implementation of this method in latest release (0.16.0) and I saw that select statement to fetch edc_contract_definitions has where statement based on a field called participant_context_id , but I guess in provided example given all edc_contract_definitions will belong to the same participant_context_id then issue will be still there. 

proposition to improve performance for this case would be to 
decrease entries fetched from edc_contract_definitions based on filter expression provided in request and then for retrieved list of access policies ids create one call for all ids to fetch all policy entries at once during one database call . This way database call will be reduced to 1 + 1 .

this is just a suggestion I do not think im edc expert therefore maybe I do not understand whole flow and why it was done this way not another.

kind regards Szymon

Expected Behavior

endpoint execution should not throw timeouts

Observed Behavior

endpoint execution time is very long given database have many entries in database. it is so long that sometimes connection timeout errors are thrown

Steps to Reproduce

Steps to reproduce the behavior:

curl --location 'http://localhost:29193/management/v3/catalog/request'
--header 'Content-Type: application/json'
--data-raw '{
"@context": {
"@vocab": "https://w3id.org/edc/v0.0.1/ns/"
},
"counterPartyAddress": "http://localhost:19194/protocol",
"protocol": "dataspace-protocol-http",
"querySpec": {
"limit": 100,
"filterExpression": [
{
"operandLeft": "https://w3id.org/edc/v0.0.1/ns/id",
"operator": "=",
"operandRight": "example-s3-asset",
"@type": "Criterion"
}
]
}

}
'

Context Information

used EDC version 0.10.1 but i believe that in latest version 0.16.0 issue is still reproducable

Detailed Description

If applicable, add screenshots and logs to help explain your problem.

Possible Implementation

proposition to improve performance for this case would be to 
decrease entries fetched from edc_contract_definitions based on filter expression provided in request and then for retrieved list of access policies ids create one call for all ids to fetch all policy entries at once during one database call . This way database call will be reduced to 1 + 1 .

this is just a suggestion I do not think im edc expert therefore maybe I do not understand whole flow and why it was done this way not another.



kind regards Szymon

[github-actions]

Thanks for your contribution 🔥 We will take a look asap 🚀

[paullatzelsperger]

this is a question we've seen a number of times.

constructing the catalog is a bit more complicated than resolving rows from the database: contract-defs and assets must be resolved and cross-referenced with policies, etc., so a large number of either of them will negatively impact the e2e time.

i don't know your exact use case, but having a lot of assets/contract-defs usually is an indication of problematic data modeling. lets say you're modelling objects that are accessible via a private REST API, then modelling each object as one asset, and - to make matter worse - defining one contract def for each of those assets would be the worst case.

instead, it is recommended to model the API access as one asset and perform object-level authorization in the data plane.

[paullatzelsperger]

oh and another thing: markdown code fences exist for a reason. use them. it costs you nothing to properly format text.