Skip to content

Commit ac46eb1

Browse files
committed
#23 Add documentation about unstruct() to README.
1 parent 968f4ad commit ac46eb1

File tree

1 file changed

+33
-0
lines changed

1 file changed

+33
-0
lines changed

README.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,39 @@ scala> df.nestedMapColumn(inputColumnName = "my_array.a", outputColumnName = "c"
219219
+---+---------------------------------------+
220220
```
221221

222+
## Other transformations
223+
224+
### Unstruct
225+
226+
Syntax: `df.nestedUnstruct("NestedStructColumnName")`.
227+
228+
Flattens one level of nesting when a struct is nested in another struct. For example,
229+
230+
```scala
231+
scala> df.printSchema
232+
root
233+
|-- id: long (nullable = true)
234+
|-- my_array: array (nullable = true)
235+
| |-- element: struct (containsNull = true)
236+
| | |-- a: long (nullable = true)
237+
| | |-- b: string (nullable = true)
238+
| | |-- c: struct (containsNull = true)
239+
| | | |--nestedField1: string (nullable = true)
240+
| | | |--nestedField2: long (nullable = true)
241+
242+
scala> df.nestedUnstruct("my_array.c").printSchema
243+
root
244+
|-- id: long (nullable = true)
245+
|-- my_array: array (nullable = true)
246+
| |-- element: struct (containsNull = true)
247+
| | |-- a: long (nullable = true)
248+
| | |-- b: string (nullable = true)
249+
| | |-- nestedField1: string (nullable = true)
250+
| | |-- nestedField2: long (nullable = true)
251+
```
252+
253+
Note that the output schema doesn't have the `c` struct. All fields of `c` are now part of the parent struct.
254+
222255
## Changelog
223256
- #### 0.2.1 released 21 January 2020.
224257
- [#10](https://github.com/AbsaOSS/spark-hats/issues/10) Fixed error column aggregation when the input array is `null`.

0 commit comments

Comments
 (0)