Add documentation on incumbent calculation for non-deterministic runs

Author: Sarah Krebs

docs/plugins/configuration_footprint.rst

@@ -41,7 +41,9 @@ Performance plot
 ----------------
 Perhaps the most useful plot of the two is to see how much we know about which configuration will
 achieve which **objective** score. To view this, mousing over the **Incumbent** will tell you the
-best configuration found for the given *objective* and *budget*. The **Evaluated Configuration**
+best configuration found for the given *objective* and *budget*. For non-deterministic runs (i.e.
+multiple seeds evaluated per configuration), only configurations evaluated on the maximum number of
+seeds are considered to choose the best configuration from. The **Evaluated Configuration**
 points will have a true objective score, all be it possibly noisy if the objective is noisy.
 Using these two kind of points, we can try to infer what the performance for the rest of the
 configuration space will be.

docs/plugins/cost_over_time.rst

@@ -13,6 +13,11 @@ deviation too.
     The configuration spaces of the selected runs should be equal. Otherwise, a good comparison
     is not possible. They can, however, still be displayed in the same graph.
 
+.. note::
+    For non-deterministic runs (i.e. multiple seeds evaluated per configuration), only
+    configurations evaluated on the maximum number of seeds are considered to choose the best
+    configuration at a given time from.
+
 This plugin is capable of answering following questions:
 
 * Does the optimizer converge?