-
Notifications
You must be signed in to change notification settings - Fork 1
Automate parameter selection #11
Description
From JIRA:
As of today, most parameters we provide to Spark are based on either indications provided by the official documentation, or mostly on trial-and-error tests after we find issues. Some of these parameters, such as the memory consumption limit per driver / executor, can be automized based on the input parameters. For instance, we can launch a mock simulation just to understand the input sizes and then configure Spark properly. It would also be good to recover after we detect a timeout exception, tweaking whatever parameters caused the problem automatically. The final .properties file can then be stored, to be used again if needed in other simulations of the same campaign.
As a consequence, the goal of this task is to explore such functionality in the context of the functionalizer, preventing out-of-memory exceptions, timeout exceptions, and so on.
This should indeed be tuned to account for the available resources.