Added ANTLR parse tree persistent caching for procedures by manisha-deshpande · Pull Request #4547 · babelfish-for-postgresql/babelfish_extensions

manisha-deshpande · 2026-02-06T03:09:44Z

Description

[TBD]

Issues Resolved

BABEL-6037

Test Scenarios Covered

[TBD]

Use case based -
Boundary conditions -
Arbitrary inputs -
Negative test cases -
Minor version upgrade tests -
Major version upgrade tests -
Performance tests -
Tooling impact -
Client tests -

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

manisha-deshpande · 2026-02-06T19:30:12Z

contrib/babelfishpg_tsql/sql/ownership.sql

 	create_date SYS.DATETIME NOT NULL,
 	modify_date SYS.DATETIME NOT NULL,
 	definition sys.NTEXT DEFAULT NULL,
+	antlr_parse_tree JSONB DEFAULT NULL, -- JSONB serialized ANTLR parse tree for caching


babelfish_function_ext regression test fails.
Column can be queried from postgres side, not babelfish side due to unsupported datatype.
Should the column type be TEXT instead? Would that affect storage space?

--- /home/runner/work/babelfish_extensions/babelfish_extensions/test/JDBC/./expected/babelfish_function_ext-vu-cleanup.out 2026-02-06 18:16:42.761144533 +0000 +++ /home/runner/work/babelfish_extensions/babelfish_extensions/test/JDBC/./output/babelfish_function_ext-vu-cleanup.out 2026-02-06 18:42:00.808212640 +0000 @@ -52,7 +52,7 @@ -- babelfish_function_ext entry should have been removed after dropping all these functions/procedure SELECT * FROM sys.babelfish_function_ext WHERE funcname LIKE 'babel_2877_vu_prepare%'; GO -~~START~~ -varchar#!#varchar#!#nvarchar#!#text#!#text#!#bigint#!#bigint#!#datetime#!#datetime#!#ntext -~~END~~ +~~ERROR (Code: 33557097)~~ + +~~ERROR (Message: data type jsonb is not supported yet)~~

You can ignore this error for now. We can later add a TDS sender function for JSONB (which just sends it as JSON)

Should the column type be TEXT instead? Would that affect storage space?

Yes, JSONB will allow fast lookups compared to JSON/TEXT which will required deserialization of its own. (Which will become a problem for bigger procedures).

robverschoor · 2026-03-12T15:48:56Z

contrib/babelfishpg_tsql/src/pltsql_serialize_jsonb.h

+ *
+ * This header provides the interface for serializing and deserializing
+ * ANTLR PLtsql parse trees to/from JSONB format. The serialized data is
+ * stored in the cross-session cache (babelfish_func_ext) to enable faster


it's a catalog rather than a cache - even though the catalog will be cached

robverschoor · 2026-03-12T16:51:17Z

contrib/babelfishpg_tsql/src/pltsql_deserialize.c

+
+			/* Read the value for this key */
+			tok = JsonbIteratorNext(&ctx.it, &v, false);
+


why overwrite tok ?

robverschoor

addded some comments

tanscorpio7 · 2026-03-19T14:52:26Z

contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--5.5.0--5.6.0.sql

+DROP TABLE IF EXISTS sys.babelfish_function_ext;
+
+CREATE TABLE sys.babelfish_function_ext (


Should be ALTER TABLE ADD COLUMN

tanscorpio7 · 2026-03-19T14:57:41Z

contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--5.5.0--5.6.0.sql

+ antlr_parse_tree_text TEXT DEFAULT NULL, -- Native PG nodeToString() serialized parse tree
+ antlr_parse_tree_datums TEXT DEFAULT NULL, -- Native PG nodeToString() serialized datums array
+ antlr_parse_tree_modify_date SYS.DATETIME DEFAULT NULL,
+ antlr_parse_tree_bbf_version TEXT DEFAULT NULL,


Two colums should be sufficent, no ?

pg_node_tree - the tree

bbf_version_for_tree

Also why will storing it in binary have performnace improvments ? Wouldn't it have the reverse effect

I've let it remain as two separate columns for now as the split by delimiter function call on large concatenated strings (for large procedures) might be expensive (unnecessary overhead)

tanscorpio7 · 2026-03-19T14:58:50Z

contrib/babelfishpg_tsql/src/guc.c


 bool		pltsql_enable_create_alter_view_from_pg = false;
 bool		pltsql_enable_alter_owner_from_pg = false;
+bool		pltsql_enable_procedure_parse_cache = false;


Lets use the general term routine applies to both procedures and functions.

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

…ble and type Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

…E, TRY Statements Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

…d retrieval Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Add cross-session ANTLR parse tree caching for T-SQL stored procedures. Serialized parse trees and datums are stored in babelfish_function_ext using nodeToString/stringToNode. On procedure execution, cached results are restored to skip ANTLR re-parsing. Cache reads validate the stored bbf_version and modify_date before deserializing, skipping stale entries from different Babelfish versions or procedures modified with the GUC disabled. Changes: - Add antlr_parse_tree_text, antlr_parse_tree_datums, antlr_parse_tree_modify_date, and antlr_parse_tree_bbf_version columns to sys.babelfish_function_ext - Store serialized parse tree and version in pltsql_store_func_default_positions - Restore and validate cached parse tree in new function pltsql_restore_func_parse_result invoked prior to ANTLR parse Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Add bbf_version validation, exec-time cache repopulation, and rename/alter/dependency invalidation logic and tests Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Move PLtsql outfuncs/readfuncs code generation entirely to the extension, eliminating the need for PLtsql-specific headers in the engine's gen_node_support.pl input files. Key changes: - gen_pltsql_node_support.pl now generates pltsql_nodetags.h with extension-owned T_PLtsql_* NodeTag values (offset from 1000 to avoid collision with engine's NodeTag enum). Includes ABI stability check that fails the build if node types are added without updating $last_nodetag/$last_nodetag_no. - Wrapper files pltsql_outfuncs.c and pltsql_readfuncs.c mirror the engine's pattern: #include the generated static functions and switch fragments, expose public pltsql_outNode() and pltsql_parseNodeString() dispatch functions. - pltsql_serialize_macros.h provides WRITE_*/READ_* macros replicated from engine internals (not exposed in any PG header). - pl_handler.c registers outNode_hook and parseNodeString_hook in _PG_init() so the engine's outNode()/parseNodeString() delegate to extension code for PLtsql node types. - pltsql.h includes generated pltsql_nodetags.h for T_PLtsql_* defines. - Makefile updated: compiles wrapper .o files (not gen .o directly), with proper dependency rules for generated files. Files changed: src/pltsql_serialize/gen_pltsql_node_support.pl - nodetags generation + ABI check src/pltsql_serialize/pltsql_outfuncs.c - new wrapper src/pltsql_serialize/pltsql_readfuncs.c - new wrapper src/pltsql_serialize/pltsql_serialize_macros.h - shared macros src/pltsql_serialize/pltsql_node_stubs.c - custom read/write nodes src/pltsql.h - include pltsql_nodetags.h src/pl_handler.c - register hooks Makefile - build rules Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

manisha-deshpande requested a review from tanscorpio7 February 6, 2026 03:10

manisha-deshpande force-pushed the jira-babel-6037 branch from 0e2dd88 to 751d3e8 Compare February 6, 2026 18:16

manisha-deshpande commented Feb 6, 2026

View reviewed changes

manisha-deshpande requested a review from forestkeeper February 9, 2026 19:24

manisha-deshpande force-pushed the jira-babel-6037 branch from 751d3e8 to 6bb6abf Compare February 23, 2026 23:26

robverschoor reviewed Mar 12, 2026

View reviewed changes

manisha-deshpande force-pushed the jira-babel-6037 branch from 6316c43 to ffb844e Compare March 19, 2026 10:32

tanscorpio7 requested changes Mar 19, 2026

View reviewed changes

manisha-deshpande added 11 commits March 24, 2026 20:39

Added ANTLR parse tree persistent caching for procedures

15b2e87

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Extended serialization logic for PLtsql datums, namespace item, varia…

c866f46

…ble and type Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Refactored and optimized PLtsql serialization deserialization code

56a38ac

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Extended serialization logic for PLtsql WHILE, PRINT, EXIT, GOTO, CAS…

86734ce

…E, TRY Statements Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Added basic test cases

b17d732

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Introduced guc for enabling and disabling antlr parse tree caching an…

865c1ee

…d retrieval Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Added serialization support for PLTSQL_STMT_DECL_TABLE

8c4bb14

Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Handle additional caching scenarios

9c9ea44

Add bbf_version validation, exec-time cache repopulation, and rename/alter/dependency invalidation logic and tests Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

Addressed PR comments and fixed regression test failures

1a30c79

Task: BABEL-6037 Signed-off-by: Manisha Deshpande <mmdeshp@amazon.com>

manisha-deshpande force-pushed the jira-babel-6037 branch from 84d3a71 to f7a3051 Compare March 24, 2026 20:54

empty commit to rebuild

ae3ac48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added ANTLR parse tree persistent caching for procedures#4547

Added ANTLR parse tree persistent caching for procedures#4547
manisha-deshpande wants to merge 12 commits intobabelfish-for-postgresql:BABEL_5_X_DEVfrom
amazon-aurora:jira-babel-6037

manisha-deshpande commented Feb 6, 2026

Uh oh!

manisha-deshpande Feb 6, 2026

Uh oh!

tanscorpio7 Feb 8, 2026

Uh oh!

robverschoor Mar 12, 2026

Uh oh!

robverschoor Mar 12, 2026

Uh oh!

robverschoor left a comment

Uh oh!

tanscorpio7 Mar 19, 2026

Uh oh!

tanscorpio7 Mar 19, 2026

Uh oh!

manisha-deshpande Mar 23, 2026

Uh oh!

tanscorpio7 Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		/* Read the value for this key */
		tok = JsonbIteratorNext(&ctx.it, &v, false);

		DROP TABLE IF EXISTS sys.babelfish_function_ext;

		CREATE TABLE sys.babelfish_function_ext (

Conversation

manisha-deshpande commented Feb 6, 2026

Description

Issues Resolved

Test Scenarios Covered

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robverschoor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants