Skip to content

Commit b4870c2

Browse files
committed
group streaming related changes
1 parent 2859489 commit b4870c2

File tree

1 file changed

+113
-113
lines changed

1 file changed

+113
-113
lines changed

blog/2025-10-14-nushell_v0_108_0.md

Lines changed: 113 additions & 113 deletions
Original file line numberDiff line numberDiff line change
@@ -388,17 +388,6 @@ Error: nu::shell::cant_convert
388388
00000010 00000001
389389
```
390390

391-
### `for` loops no longer collect their source ([#16593])
392-
Previously, `for` loops only worked with values (ranges, lists, etc), so using them with streaming commands would collect the whole stream before iteration could start. Now it also works with streams too!
393-
394-
This makes it possible to use it with slow or unbounded streams:
395-
396-
```nushell
397-
for event in (watch . --glob=**/*.rs) {
398-
cargo test
399-
}
400-
```
401-
402391
### Guess no more! More completions in built-in commands ([#16383])
403392

404393
Thanks to some recent improvements behind the scenes, nushell built-in commands that expect a specific set of values now suggests these values.
@@ -533,6 +522,117 @@ def fish-completer [spans: list<string>] {
533522

534523
</details>
535524

525+
### `for` loops no longer collect their source ([#16593])
526+
Previously, `for` loops only worked with values (ranges, lists, etc), so using them with streaming commands would collect the whole stream before iteration could start. Now it also works with streams too!
527+
528+
This makes it possible to use it with slow or unbounded streams:
529+
530+
```nushell
531+
for event in (watch . --glob=**/*.rs) {
532+
cargo test
533+
}
534+
```
535+
536+
### Streams of Streams ([#16735])
537+
538+
When you run `each` over a list (or stream), it can only return a single item for each input. When we need to create multiple values from a single input, we can just return a list in the closure and run the output through `flatten`.
539+
540+
```nushell
541+
each { .. } | flatten
542+
```
543+
544+
When the closure itself generates a _stream_ however, that pattern can be inadequate, as the stream has to be collected into a list, just to be flattened afterwards. This might increase memory usage, and result in unnecessary waits.
545+
546+
To address this, `each` has a new flag: `--flatten`
547+
548+
`each --flatten` does not wait to collect closure output to ensure returning a single output item for each input, instead the stream from the closure is directly passed through. Here's a demonstration of differences:
549+
550+
<table>
551+
<tbody>
552+
<tr>
553+
<td>
554+
555+
```nushell:title="each | flatten"
556+
def slow-source [range: range] {
557+
$range | each {|e| sleep 0.1sec; $e }
558+
}
559+
560+
0..5..<25
561+
| each {|e| slow-source ($e)..<($e + 5) }
562+
| flatten
563+
| each {|e| print $e; $e}
564+
| ignore
565+
```
566+
567+
</td>
568+
<td>
569+
570+
```nushell:title="each --flatten"
571+
#
572+
def slow-source [range: range] {
573+
$range | each {|e| sleep 0.1sec; $e }
574+
}
575+
576+
0..5..<25
577+
| each --flatten {|e| slow-source ($e)..<($e + 5) }
578+
| each {|e| print $e; $e}
579+
| ignore
580+
```
581+
582+
</td>
583+
</tr>
584+
<tr>
585+
<td width="50%">
586+
587+
![each then flatten](https://gist.githubusercontent.com/Bahex/50beec0aaf82671d55f23feda65dca27/raw/690cdbe49d761d8a8f0886451868cd16fe275184/pr_16735_old.nu.cast.svg)
588+
589+
</td>
590+
<td width="50%">
591+
592+
![each --flatten](https://gist.githubusercontent.com/Bahex/50beec0aaf82671d55f23feda65dca27/raw/690cdbe49d761d8a8f0886451868cd16fe275184/pr_16735_new.nu.cast.svg)
593+
594+
</td>
595+
</tr>
596+
</tbody>
597+
</table>
598+
599+
### Arbitrary Pipeline Metadata ([#16821])
600+
601+
In addition to existing metadata fields (`content_type`, `source`, `span`), it's now possible to attach arbitrary metadata fields to pipeline data using `metadata set --merge`:
602+
603+
```nushell
604+
"data" | metadata set --merge {custom_key: "value"} | metadata | get custom_key
605+
```
606+
607+
Combined with `metadata access`, this makes it possible to carry extra data with streams, without impacting the stream data itself.
608+
609+
### HTTP Response Metadata ([#16821])
610+
611+
All `http` commands now attach response data (previously only accessible with `http * --full`) as metadata to their output streams. This can be accessed under the `http_response` field within the pipeline metadata:
612+
613+
- `status` - HTTP status code
614+
- `headers` - Response headers as `[{name, value}, ...]`
615+
- `urls` - Redirect history
616+
617+
Accessing this metadata after the response completes may not offer much benefit.
618+
```nushell
619+
http get https://api.example.com | metadata | get http_response.status
620+
# => 200
621+
```
622+
623+
Where it shines is accessing it _alongside_ the streaming response body using `metadata access`:
624+
```nushell
625+
http get --allow-errors https://api.example.com/events.jsonl
626+
| metadata access {|meta|
627+
if $meta.http_response.status != 200 {
628+
error make {msg: "failed"}
629+
} else { }
630+
}
631+
| lines
632+
| each { from json }
633+
| where event_type == "error"
634+
```
635+
536636
### Add `$nu.is-lsp` to help with printing in lsp mode ([#16635])
537637
When nushell is launched with the `--lsp` flag, nushell will set `$nu.is-lsp` to true so that users can programmatically know when nushell is in LSP mode.
538638

@@ -708,69 +808,6 @@ The `str length` command now has a `--chars` flag to allow you to count characte
708808
5
709809
```
710810

711-
### Streams of Streams ([#16735])
712-
713-
When you run `each` over a list (or stream), it can only return a single item for each input. When we need to create multiple values from a single input, we can just return a list in the closure and run the output through `flatten`.
714-
715-
```nushell
716-
each { .. } | flatten
717-
```
718-
719-
When the closure itself generates a _stream_ however, that pattern can be inadequate, as the stream has to be collected into a list, just to be flattened afterwards. This might increase memory usage, and result in unnecessary waits.
720-
721-
To address this, `each` has a new flag: `--flatten`
722-
723-
`each --flatten` does not wait to collect closure output to ensure returning a single output item for each input, instead the stream from the closure is directly passed through. Here's a demonstration of differences:
724-
725-
<table>
726-
<tbody>
727-
<tr>
728-
<td>
729-
730-
```nushell:title="each | flatten"
731-
def slow-source [range: range] {
732-
$range | each {|e| sleep 0.1sec; $e }
733-
}
734-
735-
0..5..<25
736-
| each {|e| slow-source ($e)..<($e + 5) }
737-
| flatten
738-
| each {|e| print $e; $e}
739-
| ignore
740-
```
741-
742-
</td>
743-
<td>
744-
745-
```nushell:title="each --flatten"
746-
#
747-
def slow-source [range: range] {
748-
$range | each {|e| sleep 0.1sec; $e }
749-
}
750-
751-
0..5..<25
752-
| each --flatten {|e| slow-source ($e)..<($e + 5) }
753-
| each {|e| print $e; $e}
754-
| ignore
755-
```
756-
757-
</td>
758-
</tr>
759-
<tr>
760-
<td width="50%">
761-
762-
![each then flatten](https://gist.githubusercontent.com/Bahex/50beec0aaf82671d55f23feda65dca27/raw/690cdbe49d761d8a8f0886451868cd16fe275184/pr_16735_old.nu.cast.svg)
763-
764-
</td>
765-
<td width="50%">
766-
767-
![each --flatten](https://gist.githubusercontent.com/Bahex/50beec0aaf82671d55f23feda65dca27/raw/690cdbe49d761d8a8f0886451868cd16fe275184/pr_16735_new.nu.cast.svg)
768-
769-
</td>
770-
</tr>
771-
</tbody>
772-
</table>
773-
774811
### The `to md` command now always returns valid Markdown tables ([#16681])
775812
Previously, a command like this:
776813
```shell
@@ -797,45 +834,6 @@ Now, such special characters are automatically escaped, ensuring the output alwa
797834
### `compact` your records ([#16810])
798835
`compact` can now be used to remove `null` or `--empty` items from records too.
799836

800-
### `compact --empty` now recognizes 0 byte binary values as empty ([#16810])
801-
802-
### Arbitrary Pipeline Metadata ([#16821])
803-
804-
In addition to existing metadata fields (`content_type`, `source`, `span`), it's now possible to attach arbitrary metadata fields to pipeline data using `metadata set --merge`:
805-
806-
```nushell
807-
"data" | metadata set --merge {custom_key: "value"} | metadata | get custom_key
808-
```
809-
810-
Combined with `metadata access`, this makes it possible to carry extra data with streams, without impacting the stream data itself.
811-
812-
### HTTP Response Metadata ([#16821])
813-
814-
All `http` commands now attach response data (previously only accessible with `http * --full`) as metadata to their output streams. This can be accessed under the `http_response` field within the pipeline metadata:
815-
816-
- `status` - HTTP status code
817-
- `headers` - Response headers as `[{name, value}, ...]`
818-
- `urls` - Redirect history
819-
820-
Accessing this metadata after the response completes may not offer much benefit.
821-
```nushell
822-
http get https://api.example.com | metadata | get http_response.status
823-
# => 200
824-
```
825-
826-
Where it shines is accessing it _alongside_ the streaming response body using `metadata access`:
827-
```nushell
828-
http get --allow-errors https://api.example.com/events.jsonl
829-
| metadata access {|meta|
830-
if $meta.http_response.status != 200 {
831-
error make {msg: "failed"}
832-
} else { }
833-
}
834-
| lines
835-
| each { from json }
836-
| where event_type == "error"
837-
```
838-
839837
### Breaking changes
840838

841839
### `collect` removes `source` metadata ([#16821])
@@ -1042,6 +1040,8 @@ This was due to a miscompilation bug, which is now fixed.
10421040

10431041
### Other fixes
10441042

1043+
* `compact --empty` now recognizes 0 byte binary values as empty ([#16810])
1044+
10451045
* Fixed associative equality comparison for floats. ([#16549])
10461046

10471047
* The `**` (power) operator now parses as right-associative instead of left-associative, matching standard mathematical behavior. ([#16552])

0 commit comments

Comments
 (0)