Skip to content

Commit 4c36226

Browse files
alambalamb
authored
Docs: Add Examples to Config Options page (#17039)
1 parent cc78edc commit 4c36226

File tree

4 files changed

+92
-22
lines changed

4 files changed

+92
-22
lines changed

‎datafusion/common/src/config.rs‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1008,11 +1008,23 @@ impl ConfigOptions {
10081008
e.0.set(key, value)
10091009
}
10101010

1011-
/// Create new ConfigOptions struct, taking values from
1012-
/// environment variables where possible.
1011+
/// Create new [`ConfigOptions`], taking values from environment variables
1012+
/// where possible.
10131013
///
1014-
/// For example, setting `DATAFUSION_EXECUTION_BATCH_SIZE` will
1015-
/// control `datafusion.execution.batch_size`.
1014+
/// For example, to configure `datafusion.execution.batch_size`
1015+
/// ([`ExecutionOptions::batch_size`]) you would set the
1016+
/// `DATAFUSION_EXECUTION_BATCH_SIZE` environment variable.
1017+
///
1018+
/// The name of the environment variable is the option's key, transformed to
1019+
/// uppercase and with periods replaced with underscores.
1020+
///
1021+
/// Values are parsed according to the [same rules used in casts from
1022+
/// Utf8](https://docs.rs/arrow/latest/arrow/compute/kernels/cast/fn.cast.html).
1023+
///
1024+
/// If the value in the environment variable cannot be cast to the type of
1025+
/// the configuration option, the default value will be used instead and a
1026+
/// warning emitted. Environment variables are read when this method is
1027+
/// called, and are not re-read later.
10161028
pub fn from_env() -> Result<Self> {
10171029
struct Visitor(Vec<String>);
10181030

‎datafusion/execution/src/config.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,9 @@ impl SessionConfig {
117117
}
118118

119119
/// Create an execution config with config options read from the environment
120+
///
121+
/// See [`ConfigOptions::from_env`] for details on how environment variables
122+
/// are mapped to config options.
120123
pub fn from_env() -> Result<Self> {
121124
Ok(ConfigOptions::from_env()?.into())
122125
}

‎dev/update_config_docs.sh‎

Lines changed: 36 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -50,22 +50,49 @@ cat <<'EOF' > "$TARGET_FILE"
5050
-->
5151
5252
<!---
53-
This file was generated by the dev/update_config_docs.sh script.
53+
NOTE: This file was generated by the dev/update_config_docs.sh script.
5454
Do not edit it manually as changes will be overwritten.
5555
Instead, edit dev/update_config_docs.sh or the docstrings in datafusion/core/src/config.rs.
5656
-->
5757
5858
# Configuration Settings
5959
60-
The following configuration options can be passed to `SessionConfig` to control various aspects of query execution.
60+
DataFusion configurations control various aspects of DataFusion planning and execution
6161
62-
For applications which do not expose `SessionConfig`, like `datafusion-cli`, these options may also be set via environment variables.
63-
To construct a session with options from the environment, use `SessionConfig::from_env`.
64-
The name of the environment variable is the option's key, transformed to uppercase and with periods replaced with underscores.
65-
For example, to configure `datafusion.execution.batch_size` you would set the `DATAFUSION_EXECUTION_BATCH_SIZE` environment variable.
66-
Values are parsed according to the [same rules used in casts from Utf8](https://docs.rs/arrow/latest/arrow/compute/kernels/cast/fn.cast.html).
67-
If the value in the environment variable cannot be cast to the type of the configuration option, the default value will be used instead and a warning emitted.
68-
Environment variables are read during `SessionConfig` initialisation so they must be set beforehand and will not affect running sessions.
62+
## Setting Configuration Options
63+
64+
### Programmatically
65+
You can set the options programmatically via the [`ConfigOptions`] object. For
66+
example, to configure the `datafusion.execution.target_partitions` using the API:
67+
68+
```rust
69+
use datafusion::common::config::ConfigOptions;
70+
let mut config = ConfigOptions::new();
71+
config.execution.target_partitions = 1;
72+
```
73+
74+
### Via Environment Variables
75+
76+
You can also set configuration options via environment variables using
77+
[`ConfigOptions::from_env`], for example
78+
79+
```shell
80+
DATAFUSION_EXECUTION_TARGET_PARTITIONS=1 ./your_program
81+
```
82+
83+
### Via SQL
84+
85+
You can also set configuration options via SQL using the `SET` command. For
86+
example, to configure `datafusion.execution.target_partitions`:
87+
88+
```sql
89+
SET datafusion.execution.target_partitions = '1';
90+
```
91+
92+
[`ConfigOptions`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.ConfigOptions.html
93+
[`ConfigOptions::from_env`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.ConfigOptions.html#method.from_env
94+
95+
The following configuration settings are available:
6996
7097
EOF
7198

‎docs/source/user-guide/configs.md‎

Lines changed: 37 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,22 +18,50 @@
1818
-->
1919

2020
<!---
21-
This file was generated by the dev/update_config_docs.sh script.
21+
NOTE: This file was generated by the dev/update_config_docs.sh script.
2222
Do not edit it manually as changes will be overwritten.
2323
Instead, edit dev/update_config_docs.sh or the docstrings in datafusion/core/src/config.rs.
2424
-->
2525

2626
# Configuration Settings
2727

28-
The following configuration options can be passed to `SessionConfig` to control various aspects of query execution.
28+
DataFusion configurations control various aspects of DataFusion planning and execution
2929

30-
For applications which do not expose `SessionConfig`, like `datafusion-cli`, these options may also be set via environment variables.
31-
To construct a session with options from the environment, use `SessionConfig::from_env`.
32-
The name of the environment variable is the option's key, transformed to uppercase and with periods replaced with underscores.
33-
For example, to configure `datafusion.execution.batch_size` you would set the `DATAFUSION_EXECUTION_BATCH_SIZE` environment variable.
34-
Values are parsed according to the [same rules used in casts from Utf8](https://docs.rs/arrow/latest/arrow/compute/kernels/cast/fn.cast.html).
35-
If the value in the environment variable cannot be cast to the type of the configuration option, the default value will be used instead and a warning emitted.
36-
Environment variables are read during `SessionConfig` initialisation so they must be set beforehand and will not affect running sessions.
30+
## Setting Configuration Options
31+
32+
### Programmatically
33+
34+
You can set the options programmatically via the [`ConfigOptions`] object. For
35+
example, to configure the `datafusion.execution.target_partitions` using the API:
36+
37+
```rust
38+
use datafusion::common::config::ConfigOptions;
39+
let mut config = ConfigOptions::new();
40+
config.execution.target_partitions = 1;
41+
```
42+
43+
### Via Environment Variables
44+
45+
You can also set configuration options via environment variables using
46+
[`ConfigOptions::from_env`], for example
47+
48+
```shell
49+
DATAFUSION_EXECUTION_TARGET_PARTITIONS=1 ./your_program
50+
```
51+
52+
### Via SQL
53+
54+
You can also set configuration options via SQL using the `SET` command. For
55+
example, to configure `datafusion.execution.target_partitions`:
56+
57+
```sql
58+
SET datafusion.execution.target_partitions = '1';
59+
```
60+
61+
[`configoptions`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.ConfigOptions.html
62+
[`configoptions::from_env`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.ConfigOptions.html#method.from_env
63+
64+
The following configuration settings are available:
3765

3866
| key | default | description |
3967
| ----------------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

0 commit comments

Comments
 (0)