Skip to content

Commit f6e7d03

Browse files
drewoldagCopilot
drewoldag
and
Copilot
authored
Adding example notebook showing how configs work (#631)
* Adding a basic notebook about manually updating config values and providing a custom config file. * Update docs/notebooks/config_basics.ipynb Co-authored-by: Copilot <[email protected]> * Update docs/notebooks/config_basics.ipynb Co-authored-by: Copilot <[email protected]> * Update docs/notebooks/config_basics.ipynb Co-authored-by: Copilot <[email protected]> * Responding to PR comments. Added default_config page to the reference section. --------- Co-authored-by: Copilot <[email protected]>
1 parent abdb12f commit f6e7d03

File tree

5 files changed

+217
-1
lines changed

5 files changed

+217
-1
lines changed

‎docs/common_workflows.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,7 @@
11
Common workflows
22
================
3+
4+
.. toctree::
5+
:maxdepth: 1
6+
7+
Configuration basics <notebooks/config_basics>

‎docs/hyrax_default_config.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
.. _complete_default_config:
2+
3+
Hyrax default configuration
4+
---------------------------
5+
6+
The following is the complete default configuration file for Hyrax.
7+
This file is used when no custom configuration file is provided.
8+
9+
.. literalinclude:: ../src/hyrax/hyrax_default_config.toml
10+
:language: bash
11+
:linenos:

‎docs/notebooks/config_basics.ipynb‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "e44e86b0",
6+
"metadata": {},
7+
"source": [
8+
"# Configurations in Hyrax\n",
9+
"\n",
10+
"Hyrax uses configuration files to make it easier to track all of the parameters used for a given process.\n",
11+
"Hyrax attempts to provide a set of reasonable configuration values for all of the operations a user will perform, with exceptions.\n",
12+
"\n",
13+
"When you create a Hyrax instance, the built in defaults will be used automatically."
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"id": "42f8f15d",
20+
"metadata": {},
21+
"outputs": [],
22+
"source": [
23+
"from hyrax import Hyrax\n",
24+
"\n",
25+
"h = Hyrax()"
26+
]
27+
},
28+
{
29+
"cell_type": "markdown",
30+
"id": "c161d3b0",
31+
"metadata": {},
32+
"source": [
33+
"Once a Hyrax instance is created, the configuration is available as the ``h.config`` dictionary.\n",
34+
"We can view any particular value by using the keys of the dictionary.\n",
35+
"For instance, if we wanted to know what the default filename will be for the trained model weights file, we can use the following:"
36+
]
37+
},
38+
{
39+
"cell_type": "code",
40+
"execution_count": null,
41+
"id": "2a58fdc0",
42+
"metadata": {},
43+
"outputs": [],
44+
"source": [
45+
"h.config[\"train\"][\"weights_filename\"]"
46+
]
47+
},
48+
{
49+
"cell_type": "markdown",
50+
"id": "349ccd65",
51+
"metadata": {},
52+
"source": [
53+
"If you're interested, the complete Hyrax default configuration file can be seen here:\n",
54+
"https://hyrax.readthedocs.io/en/latest/hyrax_default_config.html\n",
55+
"\n",
56+
"A small portion of the default config can be seen below."
57+
]
58+
},
59+
{
60+
"cell_type": "code",
61+
"execution_count": null,
62+
"id": "0ed312fc",
63+
"metadata": {},
64+
"outputs": [],
65+
"source": [
66+
"import json\n",
67+
"\n",
68+
"print(json.dumps(h.config[\"general\"], indent=2))"
69+
]
70+
},
71+
{
72+
"cell_type": "markdown",
73+
"id": "0212fbb8",
74+
"metadata": {},
75+
"source": [
76+
"## Updating config values\n",
77+
"\n",
78+
"While it is possible to set the values in the dictionaries directly, using `h.set_config()`\n",
79+
"will rerun the Hyrax configuration resolution algorithm to ensure that any\n",
80+
"secondary configuration updates are incorporated."
81+
]
82+
},
83+
{
84+
"cell_type": "code",
85+
"execution_count": null,
86+
"id": "a2d20e7d",
87+
"metadata": {},
88+
"outputs": [],
89+
"source": [
90+
"print(f\"Default number of epochs: {h.config['train']['epochs']}\")\n",
91+
"h.set_config(\"train.epochs\", 5)\n",
92+
"print(f\"Updated number of epochs: {h.config['train']['epochs']}\")"
93+
]
94+
},
95+
{
96+
"cell_type": "markdown",
97+
"id": "d4fa0d6d",
98+
"metadata": {},
99+
"source": [
100+
"> ## Warning\n",
101+
"> **Use caution with ``None``**\n",
102+
"> When setting config values, it is generally a good idea to avoid using ``None``.\n",
103+
"> It is preferable to use ``False`` or an empty string, ``\"\"`` when using .toml."
104+
]
105+
},
106+
{
107+
"cell_type": "markdown",
108+
"id": "c382d1f7",
109+
"metadata": {},
110+
"source": [
111+
"## User configuration files\n",
112+
"\n",
113+
"It is not always convenient to manually modify configurations each time\n",
114+
"Hyrax runs.\n",
115+
"Configurations can be overridden with a custom .toml file provided to Hyrax during\n",
116+
"instantiation.\n",
117+
"\n",
118+
"In this example, we'll name our file ``user_config.toml`` (any file name is fine), and set the log level and weights filename like so:\n",
119+
"\n",
120+
"``` toml\n",
121+
"[general]\n",
122+
"log_level = \"error\"\n",
123+
"\n",
124+
"[train]\n",
125+
"weights_filename = \"model_weights.pth\"\n",
126+
"```\n",
127+
"\n",
128+
"> ## Note\n",
129+
"> User configuration files don't need to be a complete copy of the default\n",
130+
"> configuration file.\n",
131+
"> Hyrax will start with the defaults and override the specific keys with values\n",
132+
"> from the user's configuration file."
133+
]
134+
},
135+
{
136+
"cell_type": "code",
137+
"execution_count": null,
138+
"id": "bc1da3ba",
139+
"metadata": {},
140+
"outputs": [],
141+
"source": [
142+
"import json\n",
143+
"\n",
144+
"h = Hyrax(config_file=\"./user_config.toml\")\n",
145+
"\n",
146+
"print(\"Only the log level has been changed in the 'general' section:\")\n",
147+
"print(json.dumps(h.config[\"general\"], indent=2))\n",
148+
"\n",
149+
"print(\"Additionally, the trained weights file name is now:\")\n",
150+
"print(h.config[\"train\"][\"weights_filename\"])"
151+
]
152+
},
153+
{
154+
"cell_type": "markdown",
155+
"id": "e192d03d",
156+
"metadata": {},
157+
"source": [
158+
"## Config immutability\n",
159+
"\n",
160+
"To support reproducibility, once a Hyrax action starts, the configuration is locked.\n",
161+
"There are _very few_ exceptions to this rule, and Hyrax clearly logs when the\n",
162+
"config does change.\n",
163+
"This ensures that when the config state is saved, it accurately represents the settings\n",
164+
"used by Hyrax.\n",
165+
"\n",
166+
"The state of the configuration will always be saved alongside the output of a Hyrax\n",
167+
"action in a time stamped directory like ``YYYYmmdd-HHMMSS-train-RAND``.\n",
168+
"\n",
169+
"This is another way that Hyrax supports low-code experimentation with machine learning."
170+
]
171+
}
172+
],
173+
"metadata": {
174+
"kernelspec": {
175+
"display_name": "hyrax",
176+
"language": "python",
177+
"name": "python3"
178+
},
179+
"language_info": {
180+
"codemirror_mode": {
181+
"name": "ipython",
182+
"version": 3
183+
},
184+
"file_extension": ".py",
185+
"mimetype": "text/x-python",
186+
"name": "python",
187+
"nbconvert_exporter": "python",
188+
"pygments_lexer": "ipython3",
189+
"version": "3.12.9"
190+
}
191+
},
192+
"nbformat": 4,
193+
"nbformat_minor": 5
194+
}

‎docs/notebooks/user_config.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
[general]
2+
log_level = "error"
3+
4+
[train]
5+
weights_filename = "model_weights.pth"

‎docs/reference_and_faq.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ Reference
33

44
.. toctree::
55
:maxdepth: 1
6-
6+
7+
Hyrax default configuration <hyrax_default_config>
78
OLD Builtin Datasets <autoapi/hyrax/data_sets/index>
89
OLD Custom Datasets and Models <external_libraries>
910
OLD Verbs <verbs>

0 commit comments

Comments
 (0)