Skip to content

[design doc] EXPLAIN in Postgres syntax #31643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mgree merged 7 commits into from
Apr 9, 2025
Merged

[design doc] EXPLAIN in Postgres syntax #31643

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
932b3f1
design doc for postgres syntax
mgree Feb 26, 2025
3d53849
write down concrete proposal
mgree Mar 7, 2025
caae5c4
tiny tweaks, per conversation with @ggevay
mgree Mar 12, 2025
da316ff
address feedback
mgree Mar 19, 2025
f2f1ba8
add some explanation of naming principles from convo w/@frankmcsherry
mgree Mar 19, 2025
be4e83e
missed a few comments
mgree Mar 19, 2025
b03336b
finish incomplete sentence
mgree Mar 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
338 changes: 338 additions & 0 deletions doc/developer/design/20250226_postgres_style_explain.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,338 @@
# Postgres-style syntax for `EXPLAIN`

- Associated: https://github.com/MaterializeInc/database-issues/issues/8889

## The Problem

`EXPLAIN` is meant to help users understand how Materialize actually
runs their queries. In the name of streamlining the education process,
we should make our output as much like Postgres's as is practicable.

Changing `EXPLAIN` is tricky, though: we rely heavily on `EXPLAIN`'s
completionist output to test our optimizer and debug queries. We must
be careful to keep these tests while enabling the new behavior.

[https://github.com/MaterializeInc/materialize/pull/31185
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing ]. Also, maybe you wanted to make this into a sentence?


## Success Criteria

Our default `EXPLAIN` output should be concise and in a format
reminiscent of Postgres's. Ideally, `EXPLAIN` output should match the
output in `mz_lir_mapping`. The documentation should reflect this new
syntax.

## Out of Scope

We are not going to build new `EXPLAIN` infrastructure, diagrams,
etc. For example, we are not going to attempt to differentiate between
the different meanings of `ArrangeBy` in MIR.

We are not going to invent fundamentally new ways of explaining
how Materialize works.

We are not going to do a user study in advance of any changes. (But we
will listen attentively to feedback!)

## Solution Proposal

Postgres explain plans have the format:

```
Operator
Detail
-> Child Operator #1
Detail
...
-> Child Operator #2
Detail
...
```

We should aim to follow Postgres's norms: operator names spelled out
with spaces, and properties are clearly elucidated in human-readable
formats. When it is sensible, we have simply borrowed Postgres's
terminology, i.e., `Reduce` is renamed to `GroupAggregate`.

Postgres displays some parts of the query differently from us, namely:

- Column names:
+ When a column name is available, it just gives the name (no number).
+ When a column name is unavailable, it gives the number using `$2`.
- `Map` and `Project` do not appear

We will use LIR as the new default `EXPLAIN`/`EXPLAIN AS TEXT`
output. We will update `mz_lir_mapping` to use the new Postgres-style
syntax (fixing [a bug with `Let` and `LetRec`
rendering](https://github.com/MaterializeInc/database-issues/issues/8993)
in the process.

We will need three pieces of work, which should all land together:

- We must implement the new output and put in appropriate SLT tests
for it.
- We must update `mz_lir_mapping` to use the new vocabulary.
- We must update the documentation to explain the new output, ideally
using this output everywhere `EXPLAIN` is used.

### Concrete Mapping

| LIR node | `mz_lir_mapping` node | New, Postgres-style syntax |
| :---------- | :--------------------------------------- | :-------------------------------------- |
| `Constant` | `Constant` | `Constant` |
| `Get` | `Get::PassArrangements l0` | `Index Scan on l0 using ...` |
| `Get` | `Get::Arrangement l0 (val=...)` | `Index Lookup on l0 using ...` |
| `Get` | `Get::Collection l0` | `Read l0` |
Copy link
Contributor

@ggevay ggevay Mar 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think GetPlan's correspondence to reality is a bit more complicated. I don't have the time today to delve into this, but here are my raw notes on what each of the variants are, and some possible refactorings:

GetPlan is unncessarily hard to understand:

  • GetPlan::PassArrangements:
    • no MFP
    • ! BUT: there might or might not be an input arrangement that we are passing (so it's not always an Index Scan)
  • GetPlan::Arrangement:
    • there is an input arrangement
    • no output arrangement
    • there is an MFP
    • we might be seeking a key (but that code path is deprecated, so it occurs very rarely; still occurs in aggregates.slt and is_null_propagation.slt) (so it's usually not an Index Lookup)
  • GetPlan::Collection:
    • no input arrangement (and no output arrangement)
    • currently, there is always an MFP (but this will change when we change PassArrangements)

possible refactoring (which would be useful even independently from EXPLAIN, because the above state of affairs is IMHO really counter-intuitive):

  • GetPlan::PassArrangements -- add an if to only produce it if there is an arrangement
  • split off a new lookup variant from GetPlan::Arrangement
  • rename GetPlan variants
  • Get::keys -- make it clear that these are output arrangements

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's talk about this (maybe with @frankmcsherry?) and find the right way to factor this out so that things are actually clear.

Copy link
Contributor

@frankmcsherry frankmcsherry Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm for sure available to discuss. I think we have a few moments of inherent complexity (from the pov of explain) that is more incidental from the pov of plans. Probably the right thing at the moment is to land a thing that speaks unambiguously about the important details. I'm all for figuring out the complexity here, boiling it away, but .. we'll have to iterate and perhaps the right thing at the moment is eat the fact that the truth is gross, for now.

| `Mfp` | `MapFilterProject` | `Map/Filter/Project` |
| `FlatMap` | `FlatMap` | `Flat Map` |
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Postgres have Flat Map? If not, we could consider Table Function.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly:

michaelgreenberg=# EXPLAIN SELECT DISTINCT l_discount, generate_series(1, 5) from lineitem;
                               QUERY PLAN                               
------------------------------------------------------------------------
 HashAggregate  (cost=23.50..36.00 rows=1000 width=22)
   Group Key: l_discount, generate_series(1, 5)
   ->  ProjectSet  (cost=0.00..18.50 rows=1000 width=22)
         ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(4 rows)

I'm fine with Table Function or ProjectSet.

Copy link
Contributor

@frankmcsherry frankmcsherry Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I vote Table Function!

| `Join` | `Join::Differential` | `Differential Join` |
| `Join` | `Join::Delta` | `Delta Join` |
| `Reduce` | `Reduce::Distinct` | `Distinct GroupAggregate` |
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just Distinct?

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just following Postgres, which always calls reduces a GroupAggregate:

EXPLAIN SELECT DISTINCT l_discount FROM lineitem;
                            QUERY PLAN                            
------------------------------------------------------------------
 HashAggregate  (cost=12.50..14.50 rows=200 width=18)
   Group Key: l_discount
   ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(3 rows)

Copy link
Contributor

@ggevay ggevay Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to show it as Distinct. I'd say this is a clear improvement over Postgres not showing it as Distinct. There is no need to follow Postgres just for familiarity here, because we don't really need to rely on familiarity, as it should be pretty clear for users what Distinct is.

| `Reduce` | `Reduce::Accumulable` | `Accumulable GroupAggregate` |
| `Reduce` | `Reduce::Hierarchical (monotonic)` | `Monotonic Hierarchical GroupAggregate` |
| `Reduce` | `Reduce::Hierarchical (buckets: ...)` | `Bucketed Hierarchical GroupAggregate` |
| `Reduce` | `Reduce::Basic` | `Non-incremental GroupAggregate` |
| `Reduce` | `Reduce::Collation` | `Collated GroupAggregate` (details?) |
| `TopK` | `TopK::MonotonicTop1` | `Monotonic Top1` |
| `TopK` | `TopK::MonotonicTopK` | `Monotonic TopK` |
| `TopK` | `TopK::Basic` | `Non-monotonic TopK` |
| `Negate` | `Negate` | `Negate Diffs` |
| `Threshold` | `Threshold` | `Threshold Diffs` |
| `Union` | `Union` | `Union` |
| `Union` | `Union (consolidates output)` | `Consolidating Union` |
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering whether users need to know about whether a Union consolidates. Seems like a somewhat esoteric implementation-detail, so maybe not. Also, I can't really think of a situation where user would need to make some decision based on knowing whether a Union consolidates or not.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to skip it! My understanding was that we sometimes cared about this in accounting for memory footprint, but maybe that's an us/VERBOSE TEXT thing?

Copy link
Contributor

@ggevay ggevay Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's more for us, enough in VERBOSE TEXT. (But even I haven't looked at this in a long time. Currently, the heuristic for whether a Union consolidates is very simple: if there is at least 1 negated input, then we consolidate. If we were to make the heuristic more complex then we'd need to more often look at it. There was an abandoned attempt on this here: #30360 )

| `ArrangeBy` | `Arrange` | `Arrange` |
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are no keys, but raw is true, then maybe think up another name for this case.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good idea! Feels related to the Get refactoring.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I propose we write Stream/Arrange when raw=true, on the principle that we should indicate when we could be streaming rather than using an arrangement.

Copy link
Contributor

@ggevay ggevay Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If there are no keys, but there is raw, then it could be something like Stream or Unarrange.
  • If it's just keys but no raw, then Arrange is fine.

But I'm not sure what to do with the weird cases:

  • Both keys and raw: how to indicate that there are two things being produced? Or maybe it's enough to just say Arrange, because the raw part won't be important performance-wise in this case?
  • Neither keys nor raw: hopefully this won't occur, because it doesn't make a whole lot of sense :D

| `Let` | `e0 With l1 = e1 ...` | `e1 With l1 = e1 ...` |
| `LetRec` | `e0 With Mutually Recursive l1 = e1 ...` | `e0 With Mutually Recursve l1 = e1 ...` |

In the new Postgres-style syntax, extra information will appear on the
next line: for joins, it will be the join pipelines; for
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LIR has MFPs inside pipelines. We'll need to decide what to do with these. (MIR doesn't have these, so the current MIR EXPLAIN is often lying by saying that the MFP is executed after the join, when in fact the lowering pushes the MFP inside the pipelines.)

Maybe we can just leave off all the joins paths by default, unless WITH (JOIN IMPLEMENTATIONS) is specified. But the Filters (and also maps and projects if we generally decide to show them) will have to appear somewhere.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're going to want to see join order though. Postgres always shows join order (implicitly, via AST structure) along with the join conditions at each step of the join (Hash Cond, Index Cond).

Copy link
Contributor

@ggevay ggevay Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not sure how to show the MFPs that are pushed inside join paths. The current MIR EXPLAIN doesn't show these, which sometimes creates confusion, while the current EXPLAIN PHYSICAL PLAN shows these in an extremely verbose way.

Postgres always shows join order (implicitly, via AST structure

This would get quite verbose for Delta joins, because we'd have a big tree for each join path. In other words, the verboseness of our printing would be quadratic with the number of join inputs.

Copy link
Contributor

@ggevay ggevay Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could just show joins similarly to how current MIR EXPLAIN shows them, and think about this later.

`Map/Filter/Project` it will be the expressions used in the maps and
filters.

For `Delta Join` in particular, we will want to push information
further down in the listing; see [TPC-H query 3](#tpc-h-query-3) below
for an example.

## Minimal Viable Prototype

These examples are adapted from existing MIR explain plans, so they
are not completely faithful to the language above (e.g., `Map` and
`Filter` are separate, when they will be combined in
`Map/Filter/Project`).

Arity is included in the Postgres style (cf. "width="), though we will
hopefully not need it when we have good column names.
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it could be off by default when we have good column names.


### TPC-H query 1

The query:

```sql
SELECT
l_returnflag,
l_linestatus,
sum(l_quantity) AS sum_qty,
sum(l_extendedprice) AS sum_base_price,
sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price,
sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge,
avg(l_quantity) AS avg_qty,
avg(l_extendedprice) AS avg_price,
avg(l_discount) AS avg_disc,
count(*) AS count_order
FROM
lineitem
WHERE
l_shipdate <= DATE '1998-12-01' - INTERVAL '60' day
GROUP BY
l_returnflag,
l_linestatus
ORDER BY
l_returnflag,
l_linestatus;
```

Postgres `EXPLAIN`:

```
GroupAggregate (cost=14.53..18.89 rows=67 width=248)
Group Key: l_returnflag, l_linestatus
-> Sort (cost=14.53..14.70 rows=67 width=88)
Sort Key: l_returnflag, l_linestatus
-> Seq Scan on lineitem (cost=0.00..12.50 rows=67 width=88)
Filter: (l_shipdate <= '1998-10-02 00:00:00'::timestamp without time zone)
(6 rows)
```

Materialize `EXPLAIN`:

```
Finish order_by=[#0{l_returnflag} asc nulls_last, #1{l_linestatus} asc nulls_last] output=[#0..=#9]
Project (#0{l_returnflag}..=#5{sum}, #9..=#11, #6{count}) // { arity: 10 }
Map (bigint_to_numeric(case when (#6{count} = 0) then null else #6{count} end), (#2{sum_l_quantity} / #8), (#3{sum_l_extendedprice} / #8), (#7{sum_l_discount} / #8)) // { arity: 12 }
Reduce group_by=[#4{l_returnflag}, #5{l_linestatus}] aggregates=[sum(#0{l_quantity}), sum(#1{l_extendedprice}), sum((#1{l_extendedprice} * (1 - #2{l_discount}))), sum(((#1{l_extendedprice} * (1 - #2{l_discount})) * (1 + #3{l_tax}))), count(*), sum(#2{l_discount})] // { arity: 8 }
Project (#4{l_quantity}..=#9{l_linestatus}) // { arity: 6 }
Filter (date_to_timestamp(#10{l_shipdate}) <= 1998-10-02 00:00:00) // { arity: 16 }
ReadIndex on=lineitem pk_lineitem_orderkey_linenumber=[*** full scan ***] // { arity: 16 }

Used Indexes:
- materialize.public.pk_lineitem_orderkey_linenumber (*** full scan ***)

Target cluster: quickstart
```

New Materialize `EXPLAIN`:

```
Finish
Order by: l_returnflag, l_linestatus
-> Project (columns=10)
Columns: l_returnflag..=sum, #9..=#11, count
-> Map (columns=12)
(bigint_to_numeric(case when (count = 0) then null else count end), (sum_l_quantity / #8), (sum_l_extendedprice / #8), (sum_l_discount / #8))
-> Accumulable GroupAggregate (columns=8)
Group Key: l_returnflag, l_linestatus
Aggregates: sum(l_quantity), sum(l_extendedprice), sum((l_extendedprice * (1 - l_discount))), sum(((l_extendedprice * (1 - l_discount)) * (1 + l_tax))), count(*), sum(l_discount)
-> Project (columns=6)
Columns: l_quantity..=l_linestatus
-> Filter (columns=16)
Predicates: date_to_timestamp(l_shipdate) <= 1998-10-02 00:00:00
-> Index Scan using pk_lineitem_orderkey_linenumber on lineitem (columns=16)

Used Indexes:
- materialize.public.pk_lineitem_orderkey_linenumber (*** full scan ***)
```

### TPC-H Query 3

The query:

```sql
SELECT
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate,
o_shippriority
FROM
customer,
orders,
lineitem
WHERE
c_mktsegment = 'BUILDING'
AND c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate < DATE '1995-03-15'
AND l_shipdate > DATE '1995-03-15'
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
ORDER BY
revenue DESC,
o_orderdate;
```

Postgres `EXPLAIN`:

```
Sort (cost=20.78..20.79 rows=1 width=44)
Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric - lineitem.l_discount)))) DESC, orders.o_orderdate
-> GroupAggregate (cost=20.74..20.77 rows=1 width=44)
Group Key: lineitem.l_orderkey, orders.o_orderdate, orders.o_shippriority
-> Sort (cost=20.74..20.74 rows=1 width=48)
Sort Key: lineitem.l_orderkey, orders.o_orderdate, orders.o_shippriority
-> Nested Loop (cost=0.29..20.73 rows=1 width=48)
-> Nested Loop (cost=0.14..19.93 rows=1 width=12)
-> Seq Scan on customer (cost=0.00..11.75 rows=1 width=4)
Filter: (c_mktsegment = 'BUILDING'::bpchar)
-> Index Scan using fk_orders_custkey on orders (cost=0.14..8.16 rows=1 width=16)
Index Cond: (o_custkey = customer.c_custkey)
Filter: (o_orderdate < '1995-03-15'::date)
-> Index Scan using fk_lineitem_orderkey on lineitem (cost=0.14..0.79 rows=1 width=40)
Index Cond: (l_orderkey = orders.o_orderkey)
Filter: (l_shipdate > '1995-03-15'::date)
(16 rows)
```

Materialize `EXPLAIN`:

```
Finish order_by=[#1{sum} desc nulls_first, #2{o_orderdate} asc nulls_last] output=[#0..=#3]
Project (#0{o_orderkey}, #3{sum}, #1{o_orderdate}, #2{o_shippriority}) (columns=4)
Reduce group_by=[#0{o_orderkey}..=#2{o_shippriority}] aggregates=[sum((#3{l_extendedprice} * (1 - #4{l_discount})))] (columns=4)
Project (#8{o_orderkey}, #12{o_orderdate}, #15{o_shippriority}, #22{l_extendedprice}, #23{l_discount}) (columns=5)
Filter (#6{c_mktsegment} = "BUILDING") AND (#12{o_orderdate} < 1995-03-15) AND (#27{l_shipdate} > 1995-03-15) (columns=33)
Join on=(#0{c_custkey} = #9{o_custkey} AND #8{o_orderkey} = #17{l_orderkey}) type=delta (columns=33)
implementation
%0:customer » %1:orders[#1]KAif » %2:lineitem[#0]KAif
%1:orders » %0:customer[#0]KAef » %2:lineitem[#0]KAif
%2:lineitem » %1:orders[#0]KAif » %0:customer[#0]KAef
ArrangeBy keys=[[#0{c_custkey}]] (columns=8)
ReadIndex on=customer pk_customer_custkey=[delta join 1st input (full scan)] (columns=8)
ArrangeBy keys=[[#0{o_orderkey}], [#1{o_custkey}]] (columns=9)
ReadIndex on=orders pk_orders_orderkey=[delta join lookup] fk_orders_custkey=[delta join lookup] (columns=9)
ArrangeBy keys=[[#0{l_orderkey}]] (columns=16)
ReadIndex on=lineitem fk_lineitem_orderkey=[delta join lookup] (columns=16)

Used Indexes:
- materialize.public.pk_customer_custkey (delta join 1st input (full scan))
- materialize.public.pk_orders_orderkey (delta join lookup)
- materialize.public.fk_orders_custkey (delta join lookup)
- materialize.public.fk_lineitem_orderkey (delta join lookup)

Target cluster: quickstart
```

New Materialize `EXPLAIN`:

```
Finish
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what Postgres prints for LIMIT/OFFSET, but we might try to match that too if it makes sense.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They seem to show LIMIT but not OFFSET:

michaelgreenberg=# EXPLAIN SELECT DISTINCT l_discount FROM lineitem LIMIT 10 OFFSET 15;
                               QUERY PLAN                               
------------------------------------------------------------------------
 Limit  (cost=12.65..12.75 rows=10 width=18)
   ->  HashAggregate  (cost=12.50..14.50 rows=200 width=18)
         Group Key: l_discount
         ->  Seq Scan on lineitem  (cost=0.00..12.00 rows=200 width=18)
(4 rows)

Order by: sum desc nulls_first, o_orderdate
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could leave off nulls_first/nulls_last if they are at their default setting. (Which I think is nulls_first for desc, and nulls_last for asc.

-> Project (columns=4)
Columns: o_orderkey, sum, o_orderdate, o_shippriority
-> Reduce (columns=4)
Group key: o_orderkey..=#2o_shippriority
Aggregates: sum((l_extendedprice * (1 - l_discount)))
-> Project (columns=5)
Columns: o_orderkey, o_orderdate, o_shippriority, l_extendedprice, l_discount
-> Filter (columns=33)
Predicates: (c_mktsegment = "BUILDING") AND (o_orderdate < 1995-03-15) AND (l_shipdate > 1995-03-15)
-> Delta Join (columns=33)
Conditions: c_custkey = o_custkey AND o_orderkey = l_orderkey
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned elsewhere, we'll have a problem with self-joins: we'll be seeing conditions like x = x, where one x is from one join input, and the other x is from another join input. Even including the table name won't resolve it if it's a self-join. And we have a lot of self-joins due to outer join lowering / subquery lowering creating them.

A way out could be to write something like %1.x = %2.x, where %1, %2, ... are the join inputs.

How does Postgres resolve this btw.?

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They use the relevant aliases:

michaelgreenberg=# explain select l1.l_orderkey, l2.l_orderkey from lineitem l1, lineitem l2 where l1.l_partkey = l2.l_partkey and l1.l_orderkey <> l2.l_orderkey and l1.l_shipdate < l2.l_shipdate;
                                      QUERY PLAN                                       
---------------------------------------------------------------------------------------
 Hash Join  (cost=14.50..35.00 rows=66 width=8)
   Hash Cond: (l1.l_partkey = l2.l_partkey)
   Join Filter: ((l1.l_orderkey <> l2.l_orderkey) AND (l1.l_shipdate < l2.l_shipdate))
   ->  Seq Scan on lineitem l1  (cost=0.00..12.00 rows=200 width=12)
   ->  Hash  (cost=12.00..12.00 rows=200 width=12)
         ->  Seq Scan on lineitem l2  (cost=0.00..12.00 rows=200 width=12)
(6 rows)

I don't think it's possible to have a self-join without such aliases, but I'm not 100% on that.

michaelgreenberg=# explain select * from lineitem join lineitem on (l_partkey);
ERROR:  table name "lineitem" specified more than once

Copy link
Contributor

@ggevay ggevay Mar 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Discussed in the Optimizer sync meeting: Our lowering for outer joins / subqueries also creates self-joins. The solution will probably be to make the lowering think up synthetic aliases.)

Copy link
Contributor Author

@mgree mgree Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in #31878 should make it so we have table aliases in joins:

materialize/src/sql/src/plan/query.rs

Lines 6625 to 6633 in 8e29cdc

pub fn intern_scope_item(&mut self, item: &ScopeItem) -> Arc<str> {
if let Some(table_name) = &item.table_name {
// In order to avoid clutter, we're just going to use the table name (not database or schema)
self.intern(format!("{}.{}", table_name.item, item.column_name))
} else {
self.intern(item.column_name.as_str())
}
}
}

Pipelines:
%0:customer » %1:orders[#1]KAif » %2:lineitem[#0]KAif
%1:orders » %0:customer[#0]KAef » %2:lineitem[#0]KAif
%2:lineitem » %1:orders[#0]KAif » %0:customer[#0]KAef
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably leave these off by default, unless WITH (JOIN IMPLEMENTATIONS) is specified.

-> Arranged (columns=8)
Keys: [c_custkey]
-> Index Scan using pk_customer_custkey on customer (columns=8)
Delta join first input (full scan): pk_customer_custkey
-> Arrange (columns=9)
Keys: [o_orderkey], [o_custkey]
-> Index Scan using pk_orders_orderkey, fk_orders_custkey on orders (columns=9)
Delta join lookup: pk_orders_orderkey, fk_orders_custkey
-> Arrange (columns=16)
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These Arranges won't actually be here in this case, because they are not present in LIR, due to reusing existing indexes. (Which is a very good thing -- it was confusing as hell in the MIR EXPLAIN that ArrangeBy might not might not do a thing)

But we might want to still somehow show the keys that the join picked for each input.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Tried to update this, not 100% on it.

Keys: [l_orderkey]
-> Index Scan using fk_lineitem_orderkey on lineitem (columns=16)
Delta join lookup: fk_lineitem_orderkey

Used Indexes:
- materialize.public.pk_customer_custkey (delta join 1st input (full scan))
- materialize.public.pk_orders_orderkey (delta join lookup)
- materialize.public.fk_orders_custkey (delta join lookup)
- materialize.public.fk_lineitem_orderkey (delta join lookup)

Target cluster: quickstart
```

## Alternatives

Should we more radically reduce the AST?

Should we abandon static `EXPLAIN` and encourage `mz_lir_mapping` use?
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's viable, because there are a lot of minor things to take care of when implementing plan printing, and it would be hard to do all those in SQL.

Copy link
Contributor Author

@mgree mgree Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough!


## Open questions

Should we show `Project`?

Should we show _all_ expressions for `Map` and `Filter`?
Copy link
Contributor

@ggevay ggevay Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(https://github.com/MaterializeInc/database-issues/issues/8889#issuecomment-2612639970 discusses this in the penultimate paragraph. I'm not sure, but leaning towards yes.)


How much of this data should `mz_lir_mapping` show?