Discuss TODO items

[pyongjoo]

@nikhil96sher Please suggest in comments what we need to work on. Once we finish our discussion, I will turn them into individual Issues. In our joint meeting, we can discuss who can work on what.

1. Block direct commits to master/main. (done)
2. [rust] Create GitHub Actions for running cargo test for every PR. (done)
3. [rust] Add ArrayData module for handling csv-like data.
4. [rust] Add an implementation for join (which builds and retains a hash table inside).
   • this node keeps a small table (in hash table) in memory and pipeline the data from a large table.
5. [rust] Add an implementation for merging two tables (sorted on a key)
   • this node is to efficiently join lineitem and orders, assuming they are already sorted on their keys (i.e., orderkey).

[nikhil96sher]

Here are some more implementation tasks I can think of right now:

1. [rust] Implement Operation node: GROUP-BY Aggregate Operation with different aggregates.
   • Both with and without any GROUP-BY column
   • sum, count, avg,
   • min, max,
   • count distinct.
2. [rust] Implement Basic Operation nodes
   • SELECT columns,
   • ORDERBY columns,
   • LIMIT k
3. [rust] Implement Operation Node: WHERE
   • To support simple and complex predicate combinations.
   • Supported using DNF or CNF form of predicates.
4. [rust] Implement Operation Node: EXPRESSION
   • To support derived expressions like (l_quantity)*(1-l_discount) OR CASE statements.
5. [rust] Converting SQL TPC-H Queries into a Query Node structure.
6. [rust] Query Performance Evaluation
   • Evaluate the performance of query broken down by different operations.
   • Evaluate the performance of query broken down by different phases: Reading Data, Evaluating Query.
7. [rust] Integrate with a basic Exact Query processing mechanism
   • Ex: Postgres or equivalent of pd.sql_query in Rust
   • To validate the generated results, compute relative errors (in stages) as well as to compare runtime performance.
   • Will be used in tests to validate the correctness of merge operations.
8. [rust] Query Computation Visualization
   • Add query visualization methods to visualize the way the query is being built/processed.

[pyongjoo]

FYI, I just set that main can be updated ONLY VIA pull requests. However, no approvals are needed (yet) for PRs to get merged.

[pyongjoo]

@nikhil96sher At the moment, I think we can program directly in rust those individual operations without implementing automatic translation from (sql text) -> (rust code). But, to make our programming easier/faster, we can introduce some commonly used structs (like selecting columns based on indexes) and re-use them for different hand-written queries.