feat: flow tql cte

[discord9]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

• Added support for using a simple TQL CTE in CREATE FLOW, e.g. WITH ... AS (TQL EVAL ...) SELECT * FROM ....
• Extended parser paths for flow creation to recognize and validate this form, while keeping the accepted shape intentionally strict.
• Updated flow query-type detection so these CTE-based flow queries are treated as TQL/batching mode.
• Fixed source-table extraction for hybrid CTE queries so flow metadata records real underlying source tables (not CTE aliases).
• Added/updated unit and SQLness tests for parsing, source-table extraction, and end-to-end flow behavior in standalone/distributed cases.

This PR enables a practical new flow authoring pattern: wrapping TQL EVAL in a CTE (for cleaner structure and column renaming) without loosening flow query constraints. The goal is to make CREATE FLOW more expressive while preserving predictable semantics, correct mode selection, and accurate source-table lineage.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[gemini-code-assist]

Summary of Changes

Hello @discord9, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the CREATE FLOW functionality by enabling the use of simple TQL Common Table Expressions (CTEs). This allows for more complex and organized TQL queries to be defined and utilized within data flows, streamlining certain use cases. The changes involve updates to the SQL parsing logic, TQL detection mechanisms, and the introduction of specific validation rules to ensure proper usage of this new feature.

Highlights

• TQL CTE Support in CREATE FLOW: Introduced support for the simplest form of TQL Common Table Expressions (CTEs) within CREATE FLOW statements, allowing users to define and use TQL queries as CTEs.
• Refactored TQL Detection Logic: Replaced manual SQL statement parsing for TQL detection with a new, centralized utility function is_tql across flow and operator modules, improving consistency and maintainability.
• Strict Validation for TQL CTEs: Implemented strict validation to ensure that only simple TQL CTEs are allowed within CREATE FLOW statements, specifically those where the main query selects directly from a single TQL CTE.
• Enhanced TQL Parsing with now() Requirement: Modified the TQL parser to optionally enforce the presence of now() expressions within TQL EVAL statements when used in specific contexts, such as CREATE FLOW.
• New Test Cases: Added comprehensive test cases to validate the parsing and execution of CREATE FLOW statements with TQL CTEs in both distributed and standalone environments.

Changelog

• src/flow/src/batching_mode/task.rs
  • Removed sql::parser and sql::statements::statement::Statement imports.
  • Added sql::parsers::utils::is_tql import.
  • Refactored determine_query_type to use the new is_tql utility function for TQL detection, simplifying the logic.
• src/operator/src/statement/ddl.rs
  • Added sql::parsers::utils::is_tql import.
  • Integrated is_tql utility into determine_flow_type to correctly identify TQL statements, including TQL CTEs, for flow creation.
• src/sql/src/parsers/create_parser.rs
  • Added InvalidFlowQuerySnafu to error imports.
  • Renamed parse_sql_or_tql to parse_flow_sql_or_tql.
  • Modified parse_flow_sql_or_tql to detect and handle WITH clauses, specifically for TQL CTEs.
  • Introduced validation for WITH clauses to ensure only the simplest TQL CTE queries are accepted in CREATE FLOW statements.
  • Added new test cases for CREATE FLOW with TQL CTEs, including scenarios for valid TQL CTEs, unsupported SQL CTEs, and TQL CTEs requiring now() expressions.
• src/sql/src/parsers/utils.rs
  • Added crate::parsers::with_tql_parser::CteContent and crate::statements::query::Query imports.
  • Updated is_tql function to also recognize simple TQL CTE queries as TQL statements.
  • Implemented is_simple_tql_cte_query function to strictly validate the structure of TQL CTEs, ensuring they follow the simplest form (single TQL CTE, main query selects from it).
• src/sql/src/parsers/with_tql_parser.rs
  • Removed ParseOptions import from crate::parser.
  • Refactored parse_with_tql to call a new parse_with_tql_with_now function.
  • Modified parse_with_tql_with_now, parse_hybrid_cte, parse_cte_content, and parse_tql_content_in_cte to accept a require_now_expr boolean parameter, allowing conditional enforcement of now() in TQL EVAL statements.
  • Simplified TQL parsing within CTEs by directly using ParserContext::parse_tql instead of ParserContext::create_with_dialect.
• tests/cases/distributed/flow-tql/flow_tql_cte.result
  • Added new test result file for distributed TQL CTE flow.
• tests/cases/distributed/flow-tql/flow_tql_cte.sql
  • Added new test case file for distributed TQL CTE flow.
• tests/cases/standalone/flow-tql/flow_tql_cte.result
  • Added new test result file for standalone TQL CTE flow.
• tests/cases/standalone/flow-tql/flow_tql_cte.sql
  • Added new test case file for standalone TQL CTE flow.

Activity

• No human activity (comments, reviews) has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the simplest form of TQL CTE in flows. The changes involve updating the SQL parser to recognize and validate TQL CTEs within CREATE FLOW statements, and adjusting query type detection to handle these new constructs. The implementation is mostly clean and includes new tests. I've identified a performance issue due to double parsing and a weak assertion in one of the new tests. Please see my comments for details.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cde92ba023

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".