docs: US-034 - Update PRD and progress log with lexer/getOutFunctions fixes

Author: DQinYuan

prd.json

@@ -565,7 +565,7 @@
       ],
       "priority": 34,
       "passes": false,
-      "notes": "PROGRESS (2026-02-25): Continuing progress on US-034.\n\n## Fixed in this iteration:\n1. Fixed testVisitTernaryNode in InstructionGeneratorTest - updated to expect 6 instructions (TracePeekInstruction was added)\n2. Created OutVariableDetector for scope-aware out variable detection\n3. Updated Express4Runner.getOutVarNames to use OutVariableDetector\n4. Fixed function parameter handling in out variable detection (parameters create new scopes)\n5. Added class reference filtering (Math, java, javax, etc. not treated as out variables)\n\n## Key Changes:\n- OutVariableDetector uses scope stack to properly track variable declarations\n- Function and lambda parameters create isolated scopes that don't affect outer scope\n- Class references like Math.abs(1) and java.lang.Math are filtered from out variables\n- IfNode handling: variables in else branch that aren't declared before if are out vars\n\n## Files Changed:\n- src/main/java/com/alibaba/qlexpress4/parser/visitor/OutVariableDetector.java (new file, ~460 lines)\n- src/main/java/com/alibaba/qlexpress4/Express4Runner.java (updated getOutVarNames)\n- src/test/java/com/alibaba/qlexpress4/parser/visitor/InstructionGeneratorTest.java (updated test expectations)\n\n## Test Results:\n- 644 tests run, 18 failures + 12 errors = 30 total issues (down from 33)\n- getOutVarNamesTest now passes (all 5 assertions)\n- testVisitTernaryNode tests now pass\n\n## Remaining Work (30 issues):\n- 18 Express4RunnerTest failures (interpolation, number, import, trace, etc.)\n- 12 Express4RunnerTest errors (extension functions, import, field access, etc.)\n- 6 OperatorLimitTest failures (operator count logic)\n- 1 ClearDfaCacheTest error (missing index)\n- 1 TestSuiteRunner error (assert type casting)\n- 1 CustomItemsDocTest error (function not found)\n- 1 Pf4jClassSupplierTest error (field access)\n"
+      "notes": "PROGRESS (2026-02-25): Continuing progress on US-034.\n\n## Fixed in this iteration:\n1. Fixed lexer string escape sequences - single-quote strings now process \\n, \\t, \\r, \\b, \\f, \\\" using escapeChar() helper\n2. Fixed double-quoted string interpolation - DISABLE mode now processes \\${ as $ (not preserved)\n3. Fixed braceDepth tracking - only applies when interpolation is ENABLED\n4. Created FunctionDefinitionExtractor for extracting function definitions\n5. Enhanced FunctionExtractor to track top-level function calls (calls inside function definitions are excluded)\n6. Updated getOutFunctions() to use top-level function call tracking and local function definitions\n\n## Key Changes:\n- readSingleQuoteString() now uses escapeChar() for all escape sequences\n- readDoubleQuoteString() checks interpolationMode for \\${ handling and braceDepth tracking\n- FunctionExtractor.Context tracks functionDepth and topLevelFunctionDefinitions\n- FunctionExtractor.extractWithContext() returns Context with both calls and definitions\n- getOutFunctions() filters: 1) top-level calls only, 2) not in userDefineFunction, 3) not in localDefinitions\n\n## Files Changed:\n- src/main/java/com/alibaba/qlexpress4/parser/lexer/QLexpressLexer.java (string escape fixes)\n- src/main/java/com/alibaba/qlexpress4/parser/visitor/FunctionExtractor.java (top-level tracking)\n- src/main/java/com/alibaba/qlexpress4/parser/visitor/FunctionDefinitionExtractor.java (new file)\n- src/main/java/com/alibaba/qlexpress4/Express4Runner.java (updated getOutFunctions)\n\n## Test Results:\n- 644 tests run, 628 pass (97.5% pass rate), 16 failures + 12 errors = 28 total issues (down from 30)\n- getOutFunctionsTest now passes (all 6 assertions)\n- interpolationTest now passes\n- 114 lexer tests all pass (string escapes fixed)\n\n## Remaining Work (28 issues):\n- 16 Express4RunnerTest failures (numberTest, stringEscapeTest, checkSyntaxTest, expressionTraceTest, getOutVarAttrsTest, addOperatorTest, etc.)\n- 12 Express4RunnerTest errors (extensionFunctionTest, scripTimeoutTest, docImportJavaTest, etc.)\n- 6 OperatorLimitTest failures (operator count logic)\n- 1 ClearDfaCacheTest error (missing index)\n- 1 TestSuiteRunner error (assert type casting)\n- 1 CustomItemsDocTest error (function not found)\n- 1 Pf4jClassSupplierTest error (field access)\n"
     }
   ]
 }

progress.txt

@@ -1591,3 +1591,48 @@ The new hand-written parser reports more accurate error positions. These differe
 - TracePeekInstruction was added for trace position tracking, increasing instruction count by 1
 
 ---
+
+## [2026-02-25] - US-034 (Lexer String Escapes and getOutFunctions Fix)
+- Fixed lexer string escape sequences for single-quote strings
+  - readSingleQuoteString() now uses escapeChar() helper for all escape sequences (\n, \t, \r, \b, \f, \")
+  - Previously only \\ and \' were handled, causing test failures
+- Fixed double-quoted string interpolation handling
+  - When InterpolationMode is DISABLE, \${ is processed as $ (not preserved as \${)
+  - braceDepth tracking only applies when interpolation is ENABLED
+- Created FunctionDefinitionExtractor class for extracting function definitions from AST
+- Enhanced FunctionExtractor to track top-level function calls
+  - Function calls inside function definitions are marked with isTopLevel=false
+  - Added functionDepth tracking (0 = top-level, not inside any function)
+  - Added topLevelFunctionDefinitions Set to track functions defined at top level
+- Updated Express4Runner.getOutFunctions() to:
+  - Use FunctionExtractor.extractWithContext() to get both calls and definitions
+  - Filter calls by: 1) isTopLevel() == true, 2) not in userDefineFunction, 3) not in localDefinitions
+- Added FunctionDefinitionExtractor.java (new file, ~285 lines)
+
+**Files changed:**
+- src/main/java/com/alibaba/qlexpress4/parser/lexer/QLexpressLexer.java (string escape fixes, braceDepth logic)
+- src/main/java/com/alibaba/qlexpress4/parser/visitor/FunctionExtractor.java (top-level tracking)
+- src/main/java/com/alibaba/qlexpress4/parser/visitor/FunctionDefinitionExtractor.java (new file)
+- src/main/java/com/alibaba/qlexpress4/Express4Runner.java (updated getOutFunctions)
+- prd.json (updated progress notes)
+
+**Test results:** 644 tests run, 628 pass (97.5% pass rate)
+- Reduced from 30 issues to 28 issues
+- All 114 lexer tests pass
+- getOutFunctionsTest passes (all 6 assertions)
+- interpolationTest passes
+
+**Learnings for future iterations:**
+- String escape sequences in QLExpress should be processed at the lexer level
+  - Single-quote strings: \n, \t, \r, \b, \f, \", \\, \' all need to be converted to actual characters
+  - Double-quote strings: same escapes, plus \${ handling for interpolation
+- Interpolation mode affects both ${...} processing AND escape sequence handling
+  - DISABLE mode: ${ is literal text, \${ is processed as $
+  - ENABLED modes: ${ starts interpolation, \${ is preserved (escaped interpolation)
+- For getOutFunctions(), the key insight is that function calls INSIDE function definitions should NOT be considered "out functions"
+  - Only top-level function calls (outside any function definition) should be checked
+  - Local function definitions at the top level should also be excluded
+- The test expectations were complex: nested function definitions don't make their names available at outer scope
+  - Example: `function add() { function sub() {...} }` - `sub` is NOT available at top level
+
+---