-
Notifications
You must be signed in to change notification settings - Fork 12
Open
Labels
enhancementNew feature or requestNew feature or request
Description
The connector partitions the graph to allow Spark to read it in parallel. But Spark does not know anything about the partitioning. Say the connector partitions the graph by predicate and uid range, Spark would not know that and repartition / shuffle the read data if it wanted to join on partition or uid. If Spark would know the exact partitioning scheme, it could avoid un-needed shuffle steps.
Check to what extend Spark allows data sources to tell it about its partitioning.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request