You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The results of the experiment will be stored in the directory `RunnerConfig.results_output_path/RunnerConfig.name` as defined by your config variables.
76
76
77
-
### Events
77
+
**More information about the profilers and use cases can be found in the [Wiki tab](https://github.com/S2-group/experiment-runner/wiki).**
78
78
79
-
When a user experiment is run, the following list of events are raised in order automatically by Experiment Runner:
80
-
81
-
-`BEFORE_EXPERIMENT` - Invoked only once.
82
-
- For each variation, the following events are raised in order:
83
-
1.`BEFORE_RUN` Invoked before each variation
84
-
2.`START_RUN`
85
-
3.`START_MEASUREMENT`
86
-
4.`INTERACT`
87
-
5.`CONTINUE` - Only to be used by `OperationType.SEMI` configs. (Not automatically subscribed to by the generated config.)
88
-
6.`STOP_MEASUREMENT`
89
-
7.`STOP_RUN`
90
-
8.`POPULATE_RUN_DATA`
91
-
9. Wait for `RunnerConfig.time_between_runs_in_ms` milliseconds
92
-
-`AFTER_EXPERIMENT` - Invoked only once.
93
-
94
-
*TODO: Add visualization similar to [robot-runner timeline of events](documentation/ICSE_2021.pdf)*
95
-
96
-
Variations are automatically created by the Experiment Runner in accordance to the user-defined run table (Factors, Treatment levels, and variation exclusions).
97
-
98
-
Further detailed description of the events and their expected callback behavior can be found in the generated config. One thing to notice in the config is that, each callback function that is associated with a variation, accepts a `context: RunnerContext` parameter that describes the current variation.
99
-
100
-
## Internal Details
101
-
102
-
The framework offers an automation of the infrastructure overhead for measurement-based empirical experiments, as a consequence of its design, produced by the following **design drivers**:
103
-
104
-
-**User Authority**: Give the user full authority over the experiment execution in the Python-based configuration file.
105
-
-**Focus on Orchestration**: Orchestrate the experiment on the basis of *events*. These events are characterized by their *moment of execution* in any experiment.
106
-
-**Focus on Supporting Infrastructure**: Offer the user all, potentially necessary, supporting features (e.g. factors and treatment levels).
107
-
108
-
Experiment Runner consists of the following **components**:
109
-
110
-
-**Experiment orchestrator**: Is in charge of executing the whole experiment according to the experiment configuration provided by the user.
111
-
-**Event manager**: Provides the user with subscribable events, to which callback methods can be set, which are called at the appropriate time by the Experiment Orchestrator.
112
-
-**Progress manager**: Keeps track of the execution of each run of the experiment.
113
-
-**Config Validator**: Provides a validation of a user's configuration file and checks system readiness.
114
-
115
-
*TODO: Add visualization similar to [robot-runner overview](documentation/overview.png)*
116
-
117
-
When Experiment Runner is passed a user-defined experiment config via command line arguments, it will:
118
-
119
-
- Validate the config
120
-
- Output the config's values as read by Experiment Runner in the console for user validation
121
-
- Create the experiment folder
122
-
- Create the run table (.csv), and persist it in the experiment folder
123
-
- Execute the experiment on a per-variation basis, going over each variation with its specified treatments in the run table.
0 commit comments