from — source data from pools, files, or URIs
from <pool>[@<commitish>]
from <pattern>
file <path> [format <format>]
get <uri> [format <format>]
from (
pool <pool>[@<commitish>] [ => <leg> ]
pool <pattern>
file <path> [format <format>] [ => <leg> ]
get <uri> [format <format>] [ => <leg> ]
pass
...
)
The from operator identifies one or more data sources and transmits
their data to its output. A data source can be
- the name of a data pool in a Zed lake, with optional commitish;
- the names of multiple data pools, expressed as a regular expression or glob pattern;
- a path to a file;
- an HTTP, HTTPS, or S3 URI; or
- the
passoperator, to treat the upstream data path as a source.
:::tip Note File paths and URIs may be followed by an optional format specifier. :::
Sourcing data from pools is only possible when querying a lake, such as
via the zed command or
Zed lake API. Sourcing data from files is only possible
with the zq command.
When a single pool name is specified without @-referencing a commit or ID, or
when using a pool pattern, the tip of the main branch of each pool is
accessed.
In the first four forms, a single source is connected to a single output. In the fifth form, multiple sources are accessed in parallel and may be joined, combined, or merged.
A data path can be split with the fork operator as in
from PoolOne | fork (
=> op1 | op2 | ...
=> op1 | op2 | ...
) | merge ts | ...
Or multiple pools can be accessed and, for example, joined:
from (
pool PoolOne => op1 | op2 | ...
pool PoolTwo => op1 | op2 | ...
) | join on key=key | ...
Similarly, data can be routed to different paths with replication
using the switch operator:
from ... | switch color (
case "red" => op1 | op2 | ...
case "blue" => op1 | op2 | ...
default => op1 | op2 | ...
) | ...
Examples below below assume the existence of the Zed lake created and populated by the following commands:
export ZED_LAKE=example
zed -q init
zed -q create -orderby flip:desc coinflips
echo '{flip:1,result:"heads"} {flip:2,result:"tails"}' | zed load -q -use coinflips -
zed branch -q -use coinflips trial
echo '{flip:3,result:"heads"}' | zed load -q -use coinflips@trial -
zed -q create numbers
echo '{number:1,word:"one"} {number:2,word:"two"} {number:3,word:"three"}' | zed load -q -use numbers -
zed query -f text 'from :branches | yield pool.name + "@" + branch.name | sort'
The lake then contains the two pools:
coinflips@main
coinflips@trial
numbers@main
The following file hello.zson is also used.
{greeting:"hello world!"}
Source structured data from a local file
zq -z 'file hello.zson | yield greeting'
=>
"hello world!"
Source data from a local file, but in line format
zq -z 'file hello.zson format line'
=>
"{greeting:\"hello world!\"}"
Source structured data from a URI
zq -z 'get https://raw.githubusercontent.com/brimdata/zui-insiders/main/package.json | yield productName'
=>
"Zui - Insiders"
Source data from the main branch of a pool
zed -lake example query -z 'from coinflips'
=>
{flip:2,result:"tails"}
{flip:1,result:"heads"}
Source data from a specific branch of a pool
zed -lake example query -z 'from coinflips@trial'
=>
{flip:3,result:"heads"}
{flip:2,result:"tails"}
{flip:1,result:"heads"}
Count the number of values in the main branch of all pools
zed -lake example query -f text 'from * | count()'
=>
5
Join the data from multiple pools
zed -lake example query -z '
from coinflips | sort flip
| join (
from numbers | sort number
) on flip=number word'
=>
{flip:1,result:"heads",word:"one"}
{flip:2,result:"tails",word:"two"}
Use pass to combine our join output with data from yet another source
zed -lake example query -z '
from coinflips | sort flip
| join (
from numbers | sort number
) on flip=number word
| from (
pass
pool coinflips@trial => c:=count() | yield "There were ${int64(c)} flips"
) | sort this'
=>
"There were 3 flips"
{flip:1,result:"heads",word:"one"}
{flip:2,result:"tails",word:"two"}