Skip to content

Commit ca5679c

Browse files
authored
Clarify single node cluster syntax (#527)
* clarify single node cluster syntax * markdown typo * update name of single node
1 parent 578c664 commit ca5679c

File tree

1 file changed

+35
-4
lines changed

1 file changed

+35
-4
lines changed

docs/resources/cluster.md

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,15 +77,46 @@ resource "databricks_cluster" "shared_autoscaling" {
7777

7878
When you [create a Databricks cluster](https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling), you can either provide a `num_workers` for the fixed-size cluster or provide `min_workers` and/or `max_workers` for the cluster within the `autoscale` group. When you give a fixed-sized cluster, Databricks ensures that your cluster has a specified number of workers. When you provide a range for the number of workers, Databricks chooses the appropriate number of workers required to run your job - also known as "autoscaling." With autoscaling, Databricks dynamically reallocates workers to account for the characteristics of your job. Certain parts of your pipeline may be more computationally demanding than others, and Databricks automatically adds additional workers during these phases of your job (and removes them when they’re no longer needed).
7979

80-
When using a [Single Node cluster](https://docs.databricks.com/clusters/single-node.html), `num_workers` needs to be `0`. It could be set to `0` explicitly, or just left out, as it defaults to `0`. When `num_workers` is `0`, provider checks for presence of the required Spark configurations:
80+
`autoscale` optional configuration block supports the following:
81+
82+
* `min_workers` - (Optional) The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
83+
* `max_workers` - (Optional) The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.
84+
85+
When using a [Single Node cluster](https://docs.databricks.com/clusters/single-node.html), `num_workers` needs to be `0`. It can be set to `0` explicitly, or simply not specified, as it defaults to `0`. When `num_workers` is `0`, provider checks for presence of the required Spark configurations:
8186
* `spark.master` must has prefix `local`, like `local[*]`
8287
* `spark.databricks.cluster.profile` must have value `singleNode`
8388

89+
and also `custom_tag` entry:
90+
* `"ResourceClass" = "SingleNode"`
8491

85-
`autoscale` optional configuration block supports the following:
92+
The following example demonstrates how to create an single node cluster:
8693

87-
* `min_workers` - (Optional) The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
88-
* `max_workers` - (Optional) The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.
94+
```hcl
95+
data "databricks_node_type" "smallest" {
96+
local_disk = true
97+
}
98+
99+
data "databricks_spark_version" "latest_lts" {
100+
long_term_support = true
101+
}
102+
103+
resource "databricks_cluster" "single_node" {
104+
cluster_name = "Single Node"
105+
spark_version = data.databricks_spark_version.latest_lts.id
106+
node_type_id = data.databricks_node_type.smallest.id
107+
autotermination_minutes = 20
108+
109+
spark_conf = {
110+
# Single-node
111+
"spark.databricks.cluster.profile" : "singleNode"
112+
"spark.master" : "local[*]"
113+
}
114+
115+
custom_tags = {
116+
"ResourceClass" = "SingleNode"
117+
}
118+
}
119+
```
89120

90121
### library Configuration Block
91122

0 commit comments

Comments
 (0)