Skip to content

[LIQ] Unify from, select, and group by clauses to use field lists, implement multi-source cross-join#1909

Merged
zefhemel merged 4 commits intosilverbulletmd:mainfrom
mjf:liq-improve-parser-select-from-group-by
Mar 25, 2026
Merged

[LIQ] Unify from, select, and group by clauses to use field lists, implement multi-source cross-join#1909
zefhemel merged 4 commits intosilverbulletmd:mainfrom
mjf:liq-improve-parser-select-from-group-by

Conversation

@mjf
Copy link
Copy Markdown
Contributor

@mjf mjf commented Mar 25, 2026

Unify from, select, and group by clauses to use field lists

Query clauses from, select, and group by now accept the same field
syntax as Lua table constructors (name = expr, bare expr, or [expr] = expr), giving them a consistent grammar and enabling named bindings
everywhere. #feature

What changed:

  • New FieldList rule in lua.grammar. All three clauses now parse
    their arguments through it instead of using expression lists or the
    special Name "=" exp pattern.

  • AST types LuaFromClause, LuaSelectClause, and LuaGroupByClause
    now carry fields: LuaTableField[] instead of a single expression or
    an expression array. The parser, static analysis helpers and the
    evaluator are all updated to work with this new shape.

Named bindings in select

select name, age now derives column names from bare variable or
property access expressions so the result table has string keys (name,
age) rather than integer indices. Explicit naming still works.

Named bindings in group by

group by fields can carry explicit aliases that propagate into the
post-grouping scope. For example:

group by
  n = name
select {
  label = n
}

The alias n is now visible in select, having, and order by.
Previously only the raw expression form (name) was bound and an
explicit alias was silently lost. #bugfix

A new LuaGroupByEntry type pairs each group by expression with its
optional alias. The environment builder binds every
declared alias to the corresponding key value, regardless of whether the
group key is a scalar or a multi-key table.

Named bindings in from

from p = page continues to work as before. The clause now also accepts
the full field-list syntax which lays the groundwork for multi-source
from (cross join) in a future change. #todo #feature

Complete LIQ syntax now

Complete LIQ syntax (in Postgres documentation style) could be now
roughly expressed as follows: #todo #documentation

query[[ clause [...] ]]

where clause is one of:

    FROM field [, ...]
    WHERE condition
    GROUP BY field [, ...]
    HAVING condition
    SELECT field [, ...]
    ORDER BY sort_key [, ...]
    LIMIT count [, offset]
    OFFSET start

where field is one of:

    expression
    name = expression
    [expression] = expression

where sort_key is:

    expression [ ASC | DESC ]
               [ NULLS { FIRST | LAST } ]
               [ USING name ]
               [ USING FUNCTION function_body ]

where expression is any valid Lua expression, including:

    aggregate_call
    aggregate_call FILTER ( WHERE condition )

Signed-off-by: Matouš Jan Fialka mjf@mjf.cz

zefhemel and others added 4 commits March 25, 2026 10:16
Query clauses `from`, `select`, and `group by` now accept the same field
syntax as Lua table constructors (`name = expr`, bare expr, or `[expr]
= expr`), giving them a consistent grammar and enabling named bindings
everywhere. #feature

What changed:

* New `FieldList` rule in `lua.grammar`. All three clauses now parse
  their arguments through it instead of using expression lists or the
  special `Name "=" exp` pattern.

* AST types `LuaFromClause`, `LuaSelectClause`, and `LuaGroupByClause`
  now carry fields: `LuaTableField[]` instead of a single expression or
  an expression array. The parser, static analysis helpers and the
  evaluator are all updated to work with this new shape.

Named bindings in `select`
--------------------------
`select name, age` now derives column names from bare variable or
property access expressions so the result table has string keys (`name`,
`age`) rather than integer indices. Explicit naming still works.

Named bindings in `group by`
----------------------------
`group b`y fields can carry explicit aliases that propagate into the
post-grouping scope. For example:

```sql
group by
  n = name
select {
  label = n
}
```

The alias `n` is now visible in `select`, `having`, and `order by`.
Previously only the raw expression form (`name`) was bound and an
explicit alias was silently lost. #bugfix

A new `LuaGroupByEntry` type pairs each `group by` expression with its
optional alias. The environment builder binds every
declared alias to the corresponding key value, regardless of whether the
group key is a scalar or a multi-key table.

Named bindings in `from`
------------------------
`from p = page` continues to work as before. The clause now also accepts
the full field-list syntax which lays the groundwork for multi-source
from (cross join) in a future change. #todo #feature

Complete LIQ syntax now
-----------------------
Complete LIQ syntax (in Postgres documentation style) could be now
roughly expressed as follows: #todo #documentation

```text
query[[ clause [...] ]]

where clause is one of:

    FROM field [, ...]
    WHERE condition
    GROUP BY field [, ...]
    HAVING condition
    SELECT field [, ...]
    ORDER BY sort_key [, ...]
    LIMIT count [, offset]
    OFFSET start

where field is one of:

    expression
    name = expression
    [expression] = expression

where sort_key is:

    expression [ ASC | DESC ]
               [ NULLS { FIRST | LAST } ]
               [ USING name ]
               [ USING FUNCTION function_body ]

where expression is any valid Lua expression, including:

    aggregate_call
    aggregate_call FILTER ( WHERE condition )
```

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
Examples of what works now:

```
${query [[
  from
    x = {   1,   2 },
    y = {  10,  20 },
    z = { 100, 200 }
  select {
    sum = x + y + z
  }
]]}
```

Or (more complicated three-way cross-join) to illustrate:

```
${query [[
  from
    s = {
      { id = 1, name = 'Eva',  },
      { id = 2, name = 'Adam', },
      { id = 3, name = 'John', },
      { id = 4, name = 'Zef',  },
    },
    c = {
      { id = 101, title = 'Mathematics',      },
      { id = 102, title = 'Arts',             },
      { id = 103, title = 'Physics',          },
      { id = 104, title = 'Computer Science', },
      { id = 105, title = 'Literature',       },
    },
    e = {
      { sid = 1, cid = 101, },
      { sid = 1, cid = 103, },
      { sid = 2, cid = 101, },
      { sid = 2, cid = 102, },
      { sid = 2, cid = 103, },
      { sid = 3, cid = 102, },
      { sid = 3, cid = 105, },
      { sid = 4, cid = 101, },
      { sid = 4, cid = 103, },
      { sid = 4, cid = 104, },
    }
  where
    s.id  == e.sid and
    e.cid == c.id
  order by
    s.name
  group by
    s.name
  having
    s.name:match('^A') or
    s.name:match('f$')
  select
    student = s.name:upper(),
    courses = string_agg(c.title, ', '
      order by c.title desc
    )
]]}
```

Signed-off-by: Matouš Jan Fialka <mjf@mjf.cz>
@mjf mjf changed the title [LIQ] Unify from, select, and group by clauses to use field lists [LIQ] Unify from, select, and group by clauses to use field lists, implement multi-source cross-join Mar 25, 2026
@mjf
Copy link
Copy Markdown
Contributor Author

mjf commented Mar 25, 2026

Few artifical examples of what works now with the multi-source cross-join:

${query [[
  from
    x = {   1,   2 },
    y = {  10,  20 },
    z = { 100, 200 }
  select {
    sum = x + y + z
  }
]]}
sum
111
211
121
221
112
212
122
222

Or more candy...

${query [[
  from
    x = {   1,   2,   3, },
    y = {  10,  20,  30, },
    z = { 100, 200, 300, }
  group by
    x, y, z
  having
    first(x) == 3  and
    y        == 30 and
    last(z)  == 300
  select
    x,
    y,
    z,
    sum = x + y + z
]]}
x y z sum
3 30 300 333

Or (more complicated three-way cross-join) to illustrate:

${query [[
  from
    s = {
      { id = 1, name = 'Eva',  },
      { id = 2, name = 'Adam', },
      { id = 3, name = 'John', },
      { id = 4, name = 'Zef',  },
    },
    c = {
      { id = 101, title = 'Mathematics',      },
      { id = 102, title = 'Arts',             },
      { id = 103, title = 'Physics',          },
      { id = 104, title = 'Computer Science', },
      { id = 105, title = 'Literature',       },
    },
    e = {
      { sid = 1, cid = 101, },
      { sid = 1, cid = 103, },
      { sid = 2, cid = 101, },
      { sid = 2, cid = 102, },
      { sid = 2, cid = 103, },
      { sid = 3, cid = 102, },
      { sid = 3, cid = 105, },
      { sid = 4, cid = 101, },
      { sid = 4, cid = 103, },
      { sid = 4, cid = 104, },
    }
  where
    s.id  == e.sid and
    e.cid == c.id
  order by
    s.name
  group by
    s.name
  having
    s.name:match('^A') or
    s.name:match('f$')
  select
    student = s.name:upper(),
    courses = string_agg(c.title, ', '
      order by c.title desc
    )
]]}
student courses
ADAM Physics, Mathematics, Arts
ZEF Physics, Mathematics, Computer Science

@zefhemel
Copy link
Copy Markdown
Collaborator

This is super cool man!

@zefhemel zefhemel merged commit 5996e51 into silverbulletmd:main Mar 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants