Skip to content

Commit b8f1cac

Browse files
peanutfunpeanutfun
committed
Add explanation of BayesianOptimizerController to tutorial
1 parent 01f0f1c commit b8f1cac

File tree

1 file changed

+198
-168
lines changed

1 file changed

+198
-168
lines changed

‎doc/tutorial/climada_util_calibrate.ipynb‎

Lines changed: 198 additions & 168 deletions
Original file line numberDiff line numberDiff line change
@@ -2071,6 +2071,29 @@
20712071
"source": [
20722072
"## Execute the Calibration\n",
20732073
"\n",
2074+
"We created a class `BayesianOptimizerController` to control and guide the calibration process.\n",
2075+
"It is intended to walk through several optimization iterations and stop the process if the buest guess cannot be improved.\n",
2076+
"The optimization works as follows:\n",
2077+
"\n",
2078+
"1. The optimizer randomly samples the parameter space `BayesianOptimizerController.init_points` times.\n",
2079+
"2. The optimizer uses a Gaussian regression process to \"smartly\" sample the parameter space at most `BayesianOptimizerController.n_iter` times.\n",
2080+
" * The process uses an \"Upper Confidence Bound\" sampling method whose parameter `BayesianOptimizerController.kappa` indicates how close the sampled points are to the buest guess.\n",
2081+
" Higher `kappa` means more exploration of the parameter space, lower `kappa` means more exploitation.\n",
2082+
" * After each sample, the parameter `kappa` is reduced by the factor `BayesianOptimizerController.kappa_decay`.\n",
2083+
" By default, this parameter is set such that `kappa` equals `BayesianOptimizerController.kappa_min` at the last step.\n",
2084+
" This way, the sampling becomes more exploitative the more steps are taken.\n",
2085+
"3. The controller tracks the improvements of the buest guess for parameters.\n",
2086+
" If `BayesianOptimizerController.min_improvement_count` consecutive improvements are lower than `BayesianOptimizerController.min_improvement`, the smart sampling is stopped.\n",
2087+
" In this case, the `BayesianOptimizerController.iterations` count is increased and the process repeated from step 1.\n",
2088+
"4. If an entire iteration did not show any improvement, the optimization is stopped.\n",
2089+
" It is also stopped when the `BayesianOptimizerController.max_iterations` count is reached.\n",
2090+
"\n",
2091+
"Users can control the \"density\", and thus the accuracy of the sampling by adjusting the controller parameters.\n",
2092+
"Increasing `init_points`, `n_iter`, `min_improvement_count`, and `max_iterations`, and decreasing `min_improvement` generally increases density and accuracy, but leads to longer runtimes.\n",
2093+
"\n",
2094+
"We suggest using the `from_input` classmethod for a convenient choice of sampling density based on the parameter space.\n",
2095+
"The two parameters `init_points` and `n_iter` are set to $b^N$, where $N$ is the number of estimated parameters and $b$ is the `sampling_base` parameter, which defaults to 4.\n",
2096+
"\n",
20742097
"Now we can finally execute our calibration task!\n",
20752098
"We will plug all input parameters in an instance of `Input`, and then create the optimizer instance with it.\n",
20762099
"The `Optimizer.run` method returns an `Output` object, whose `params` attribute holds the optimal parameters determined by the calibration.\n",
@@ -2122,11 +2145,185 @@
21222145
"\n",
21232146
" # Create and run the optimizer\n",
21242147
" opt = BayesianOptimizer(input)\n",
2125-
" controller = BayesianOptimizerController.from_input(inp=input)\n",
2148+
" controller = BayesianOptimizerController.from_input(input)\n",
21262149
" bayes_output = opt.run(controller)\n",
21272150
" bayes_output.params # The optimal parameters"
21282151
]
21292152
},
2153+
{
2154+
"cell_type": "markdown",
2155+
"metadata": {},
2156+
"source": [
2157+
"## Evaluate Output"
2158+
]
2159+
},
2160+
{
2161+
"attachments": {},
2162+
"cell_type": "markdown",
2163+
"metadata": {},
2164+
"source": [
2165+
"The Bayesian Optimizer returns the entire paramter space it sampled.\n",
2166+
"We can find out a lot about the relation of the fitted parameters by investigating how the cost function value depends on them.\n",
2167+
"We can retrieve the parameter space as `pandas.DataFrame` via `BayesianOptimizerOutput.p_space_to_dataframe`.\n",
2168+
"This dataframe has MultiIndex columns.\n",
2169+
"One group are the `Parameters`, the other holds information on the `Calibration` for each parameter set.\n",
2170+
"Notice that the optimal parameter set is not necessarily the last entry in the parameter space!"
2171+
]
2172+
},
2173+
{
2174+
"cell_type": "code",
2175+
"execution_count": 14,
2176+
"metadata": {},
2177+
"outputs": [
2178+
{
2179+
"data": {
2180+
"text/html": [
2181+
"<div>\n",
2182+
"<style scoped>\n",
2183+
" .dataframe tbody tr th:only-of-type {\n",
2184+
" vertical-align: middle;\n",
2185+
" }\n",
2186+
"\n",
2187+
" .dataframe tbody tr th {\n",
2188+
" vertical-align: top;\n",
2189+
" }\n",
2190+
"\n",
2191+
" .dataframe thead tr th {\n",
2192+
" text-align: left;\n",
2193+
" }\n",
2194+
"\n",
2195+
" .dataframe thead tr:last-of-type th {\n",
2196+
" text-align: right;\n",
2197+
" }\n",
2198+
"</style>\n",
2199+
"<table border=\"1\" class=\"dataframe\">\n",
2200+
" <thead>\n",
2201+
" <tr>\n",
2202+
" <th></th>\n",
2203+
" <th colspan=\"2\" halign=\"left\">Parameters</th>\n",
2204+
" <th>Calibration</th>\n",
2205+
" </tr>\n",
2206+
" <tr>\n",
2207+
" <th></th>\n",
2208+
" <th>scale</th>\n",
2209+
" <th>v_half</th>\n",
2210+
" <th>Cost Function</th>\n",
2211+
" </tr>\n",
2212+
" <tr>\n",
2213+
" <th>Iteration</th>\n",
2214+
" <th></th>\n",
2215+
" <th></th>\n",
2216+
" <th></th>\n",
2217+
" </tr>\n",
2218+
" </thead>\n",
2219+
" <tbody>\n",
2220+
" <tr>\n",
2221+
" <th>0</th>\n",
2222+
" <td>0.422852</td>\n",
2223+
" <td>115.264302</td>\n",
2224+
" <td>2.726950</td>\n",
2225+
" </tr>\n",
2226+
" <tr>\n",
2227+
" <th>1</th>\n",
2228+
" <td>0.010113</td>\n",
2229+
" <td>63.349706</td>\n",
2230+
" <td>4.133135</td>\n",
2231+
" </tr>\n",
2232+
" <tr>\n",
2233+
" <th>2</th>\n",
2234+
" <td>0.155288</td>\n",
2235+
" <td>37.268453</td>\n",
2236+
" <td>0.800611</td>\n",
2237+
" </tr>\n",
2238+
" <tr>\n",
2239+
" <th>3</th>\n",
2240+
" <td>0.194398</td>\n",
2241+
" <td>68.718642</td>\n",
2242+
" <td>1.683610</td>\n",
2243+
" </tr>\n",
2244+
" <tr>\n",
2245+
" <th>4</th>\n",
2246+
" <td>0.402800</td>\n",
2247+
" <td>92.721038</td>\n",
2248+
" <td>2.046407</td>\n",
2249+
" </tr>\n",
2250+
" <tr>\n",
2251+
" <th>...</th>\n",
2252+
" <td>...</td>\n",
2253+
" <td>...</td>\n",
2254+
" <td>...</td>\n",
2255+
" </tr>\n",
2256+
" <tr>\n",
2257+
" <th>246</th>\n",
2258+
" <td>0.956750</td>\n",
2259+
" <td>45.733468</td>\n",
2260+
" <td>0.804684</td>\n",
2261+
" </tr>\n",
2262+
" <tr>\n",
2263+
" <th>247</th>\n",
2264+
" <td>0.788425</td>\n",
2265+
" <td>48.043636</td>\n",
2266+
" <td>0.768636</td>\n",
2267+
" </tr>\n",
2268+
" <tr>\n",
2269+
" <th>248</th>\n",
2270+
" <td>0.826704</td>\n",
2271+
" <td>49.932348</td>\n",
2272+
" <td>0.764991</td>\n",
2273+
" </tr>\n",
2274+
" <tr>\n",
2275+
" <th>249</th>\n",
2276+
" <td>0.880736</td>\n",
2277+
" <td>49.290532</td>\n",
2278+
" <td>0.767437</td>\n",
2279+
" </tr>\n",
2280+
" <tr>\n",
2281+
" <th>250</th>\n",
2282+
" <td>0.744523</td>\n",
2283+
" <td>51.596642</td>\n",
2284+
" <td>0.770761</td>\n",
2285+
" </tr>\n",
2286+
" </tbody>\n",
2287+
"</table>\n",
2288+
"<p>251 rows × 3 columns</p>\n",
2289+
"</div>"
2290+
],
2291+
"text/plain": [
2292+
" Parameters Calibration\n",
2293+
" scale v_half Cost Function\n",
2294+
"Iteration \n",
2295+
"0 0.422852 115.264302 2.726950\n",
2296+
"1 0.010113 63.349706 4.133135\n",
2297+
"2 0.155288 37.268453 0.800611\n",
2298+
"3 0.194398 68.718642 1.683610\n",
2299+
"4 0.402800 92.721038 2.046407\n",
2300+
"... ... ... ...\n",
2301+
"246 0.956750 45.733468 0.804684\n",
2302+
"247 0.788425 48.043636 0.768636\n",
2303+
"248 0.826704 49.932348 0.764991\n",
2304+
"249 0.880736 49.290532 0.767437\n",
2305+
"250 0.744523 51.596642 0.770761\n",
2306+
"\n",
2307+
"[251 rows x 3 columns]"
2308+
]
2309+
},
2310+
"execution_count": 14,
2311+
"metadata": {},
2312+
"output_type": "execute_result"
2313+
}
2314+
],
2315+
"source": [
2316+
"p_space_df = bayes_output.p_space_to_dataframe()\n",
2317+
"p_space_df"
2318+
]
2319+
},
2320+
{
2321+
"cell_type": "markdown",
2322+
"metadata": {},
2323+
"source": [
2324+
"In contrast, the controller only tracks the consecutive improvements of the best guess."
2325+
]
2326+
},
21302327
{
21312328
"cell_type": "code",
21322329
"execution_count": 13,
@@ -2337,173 +2534,6 @@
23372534
"controller.improvements()"
23382535
]
23392536
},
2340-
{
2341-
"cell_type": "markdown",
2342-
"metadata": {},
2343-
"source": [
2344-
"## Evaluate Output"
2345-
]
2346-
},
2347-
{
2348-
"attachments": {},
2349-
"cell_type": "markdown",
2350-
"metadata": {},
2351-
"source": [
2352-
"The Bayesian Optimizer returns the entire paramter space it sampled.\n",
2353-
"We can find out a lot about the relation of the fitted parameters by investigating how the cost function value depends on them.\n",
2354-
"We can retrieve the parameter space as `pandas.DataFrame` via `BayesianOptimizerOutput.p_space_to_dataframe`.\n",
2355-
"This dataframe has MultiIndex columns.\n",
2356-
"One group are the `Parameters`, the other holds information on the `Calibration` for each parameter set.\n",
2357-
"Notice that the optimal parameter set is not necessarily the last entry in the parameter space!"
2358-
]
2359-
},
2360-
{
2361-
"cell_type": "code",
2362-
"execution_count": 14,
2363-
"metadata": {},
2364-
"outputs": [
2365-
{
2366-
"data": {
2367-
"text/html": [
2368-
"<div>\n",
2369-
"<style scoped>\n",
2370-
" .dataframe tbody tr th:only-of-type {\n",
2371-
" vertical-align: middle;\n",
2372-
" }\n",
2373-
"\n",
2374-
" .dataframe tbody tr th {\n",
2375-
" vertical-align: top;\n",
2376-
" }\n",
2377-
"\n",
2378-
" .dataframe thead tr th {\n",
2379-
" text-align: left;\n",
2380-
" }\n",
2381-
"\n",
2382-
" .dataframe thead tr:last-of-type th {\n",
2383-
" text-align: right;\n",
2384-
" }\n",
2385-
"</style>\n",
2386-
"<table border=\"1\" class=\"dataframe\">\n",
2387-
" <thead>\n",
2388-
" <tr>\n",
2389-
" <th></th>\n",
2390-
" <th colspan=\"2\" halign=\"left\">Parameters</th>\n",
2391-
" <th>Calibration</th>\n",
2392-
" </tr>\n",
2393-
" <tr>\n",
2394-
" <th></th>\n",
2395-
" <th>scale</th>\n",
2396-
" <th>v_half</th>\n",
2397-
" <th>Cost Function</th>\n",
2398-
" </tr>\n",
2399-
" <tr>\n",
2400-
" <th>Iteration</th>\n",
2401-
" <th></th>\n",
2402-
" <th></th>\n",
2403-
" <th></th>\n",
2404-
" </tr>\n",
2405-
" </thead>\n",
2406-
" <tbody>\n",
2407-
" <tr>\n",
2408-
" <th>0</th>\n",
2409-
" <td>0.422852</td>\n",
2410-
" <td>115.264302</td>\n",
2411-
" <td>2.726950</td>\n",
2412-
" </tr>\n",
2413-
" <tr>\n",
2414-
" <th>1</th>\n",
2415-
" <td>0.010113</td>\n",
2416-
" <td>63.349706</td>\n",
2417-
" <td>4.133135</td>\n",
2418-
" </tr>\n",
2419-
" <tr>\n",
2420-
" <th>2</th>\n",
2421-
" <td>0.155288</td>\n",
2422-
" <td>37.268453</td>\n",
2423-
" <td>0.800611</td>\n",
2424-
" </tr>\n",
2425-
" <tr>\n",
2426-
" <th>3</th>\n",
2427-
" <td>0.194398</td>\n",
2428-
" <td>68.718642</td>\n",
2429-
" <td>1.683610</td>\n",
2430-
" </tr>\n",
2431-
" <tr>\n",
2432-
" <th>4</th>\n",
2433-
" <td>0.402800</td>\n",
2434-
" <td>92.721038</td>\n",
2435-
" <td>2.046407</td>\n",
2436-
" </tr>\n",
2437-
" <tr>\n",
2438-
" <th>...</th>\n",
2439-
" <td>...</td>\n",
2440-
" <td>...</td>\n",
2441-
" <td>...</td>\n",
2442-
" </tr>\n",
2443-
" <tr>\n",
2444-
" <th>246</th>\n",
2445-
" <td>0.956750</td>\n",
2446-
" <td>45.733468</td>\n",
2447-
" <td>0.804684</td>\n",
2448-
" </tr>\n",
2449-
" <tr>\n",
2450-
" <th>247</th>\n",
2451-
" <td>0.788425</td>\n",
2452-
" <td>48.043636</td>\n",
2453-
" <td>0.768636</td>\n",
2454-
" </tr>\n",
2455-
" <tr>\n",
2456-
" <th>248</th>\n",
2457-
" <td>0.826704</td>\n",
2458-
" <td>49.932348</td>\n",
2459-
" <td>0.764991</td>\n",
2460-
" </tr>\n",
2461-
" <tr>\n",
2462-
" <th>249</th>\n",
2463-
" <td>0.880736</td>\n",
2464-
" <td>49.290532</td>\n",
2465-
" <td>0.767437</td>\n",
2466-
" </tr>\n",
2467-
" <tr>\n",
2468-
" <th>250</th>\n",
2469-
" <td>0.744523</td>\n",
2470-
" <td>51.596642</td>\n",
2471-
" <td>0.770761</td>\n",
2472-
" </tr>\n",
2473-
" </tbody>\n",
2474-
"</table>\n",
2475-
"<p>251 rows × 3 columns</p>\n",
2476-
"</div>"
2477-
],
2478-
"text/plain": [
2479-
" Parameters Calibration\n",
2480-
" scale v_half Cost Function\n",
2481-
"Iteration \n",
2482-
"0 0.422852 115.264302 2.726950\n",
2483-
"1 0.010113 63.349706 4.133135\n",
2484-
"2 0.155288 37.268453 0.800611\n",
2485-
"3 0.194398 68.718642 1.683610\n",
2486-
"4 0.402800 92.721038 2.046407\n",
2487-
"... ... ... ...\n",
2488-
"246 0.956750 45.733468 0.804684\n",
2489-
"247 0.788425 48.043636 0.768636\n",
2490-
"248 0.826704 49.932348 0.764991\n",
2491-
"249 0.880736 49.290532 0.767437\n",
2492-
"250 0.744523 51.596642 0.770761\n",
2493-
"\n",
2494-
"[251 rows x 3 columns]"
2495-
]
2496-
},
2497-
"execution_count": 14,
2498-
"metadata": {},
2499-
"output_type": "execute_result"
2500-
}
2501-
],
2502-
"source": [
2503-
"p_space_df = bayes_output.p_space_to_dataframe()\n",
2504-
"p_space_df"
2505-
]
2506-
},
25072537
{
25082538
"cell_type": "markdown",
25092539
"metadata": {},

0 commit comments

Comments
 (0)